Saturday 8 February 2020

Using exitQgis() in PyQGIS?



After infinitely many attempts to fix this, I can't prevent python to crash (segmentation fault) when scripting in QGIS. I am using the qgis2 build from osgeo4mac, installed with homebrew on El Capitan. I am interested in running something like:


from qgis.core import *


app = QgsApplication([], False)
app.initQgis()

# DO STUFF HERE

app.exitQgis()

The above script will work in the sense that all the STUFF I want it to do will be properly executed, and I believe that app.exitQgis() will also remove the data provider and layer registries from memory. However, python will crash unexpectedly on quit because of a SIGSEGV error.


Upon reading this answer from spacedman, I managed to avoid the crash with the following structure:



from qgis.core import *

def main(app):
# DO STUFF HERE
return

app = QgsApplication([], False)
app.initQgis()
main(app)
app.exitQgis()


Although I have no idea why, the above runs smoothly. I however encounter the SIGSEGV error again when I try to run something more involved, using the Processing toolbox:


from qgis.core import *
from processing.core.Processing import Processing

def main(app):
# DO STUFF HERE
Processing.runAlgorithm(...) # DO MORE INVOLVED STUFF
return


app = QgsApplication([], False)
app.initQgis()
Processing.initialize()
main(app)
app.exitQgis()

Again, the above script will properly execute everything, but python will crash on quit.


I am out of solutions and I was wondering what would happen if I simply omit the app.exitQgis() line from the above script. This would do all what needs to be done, avoid the crash, but would not "clean up".


Is this a terrible idea?


Will this use up all my RAM if I run it multiple times?



Update:


The segfault seems to be triggered by objects loaded in memory. For example this will crash:


from qgis.core import *
app = QgsApplication([], False)
app.initQgis()
layer = QgsVectorLayer('shapefile.shp', 'all', 'ogr')
app.exitQgis()

But this will not:


from qgis.core import *

app = QgsApplication([], False)
app.initQgis()
layer = QgsVectorLayer('shapefile.shp', 'all', 'ogr')
del layer # CLEANING UP HERE
app.exitQgis()

which is why a function structure also prevents the crash: loaded layers are local variables within the function environment. Regarding the use of the Processing toolbox, the number of objects in memory does indeed increase after the use of Processing.runAlgorithm(...). Using the garbage collector module, import gc, I find that gc.get_count() increases right after an algorithm from Processing is used.




some details on the crash. the python ouput produced by the app.exitQgis() line:


src/core/auth/qgsauthmethodregistry.cpp: 154: (~QgsAuthMethodRegistry)[936ms] cleanup: Basic

src/core/auth/qgsauthmethodregistry.cpp: 154: (~QgsAuthMethodRegistry) [0ms] cleanup: Identity-Cert
src/core/auth/qgsauthmethodregistry.cpp: 154: (~QgsAuthMethodRegistry) [0ms] cleanup: PKI-PKCS#12
src/core/auth/qgsauthmethodregistry.cpp: 154: (~QgsAuthMethodRegistry) [0ms] cleanup: PKI-Paths
src/core/qgsproviderregistry.cpp: 234: (clean) [3ms] cleanup:DB2
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:WFS
src/core/qgsproviderregistry.cpp: 234: (clean) [1ms] cleanup:arcgisfeatureserver
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:arcgismapserver
src/core/qgsproviderregistry.cpp: 234: (clean) [1ms] cleanup:delimitedtext
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:gdal
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:gpx

src/core/qgsproviderregistry.cpp: 234: (clean) [1ms] cleanup:memory
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:mssql
src/core/qgsproviderregistry.cpp: 234: (clean) [1ms] cleanup:ogr
src/providers/ogr/qgsogrconnpool.cpp: 41: (~QgsOgrConnPool) [0ms] Entering.
src/providers/ogr/qgsogrconnpool.cpp: 41: (~QgsOgrConnPool) [0ms] Leaving.
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:ows
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:postgres
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:spatialite
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:virtual
src/core/qgsproviderregistry.cpp: 234: (clean) [0ms] cleanup:wcs

src/core/qgsproviderregistry.cpp: 234: (clean) [1ms] cleanup:wms
src/providers/ogr/qgsogrconnpool.cpp: 36: (QgsOgrConnPool) [78ms] Entering.
src/providers/ogr/qgsogrconnpool.cpp: 36: (QgsOgrConnPool) [0ms] Leaving.
[Finished in 7.8s with exit code -11]

the OSX crash report:


Date/Time:             2017-08-03 13:06:51.003 -0400
OS Version: Mac OS X 10.11.3 (15D21)
Report Version: 11
Anonymous UUID: 08F4C3E0-65A6-5136-C3BB-0D146F412EAC



Time Awake Since Boot: 47000 seconds

System Integrity Protection: enabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: EXC_I386_GPFLT

Exception Note: EXC_CORPSE_NOTIFY

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libgdal.20.dylib 0x00000001178771f4 OGR_DS_GetDriver + 14
1 libogrprovider.so 0x0000000126408000 QgsOgrProviderUtils::OGRDestroyWrapper(void*) + 40
2 libogrprovider.so 0x00000001263ee5e4 QgsOgrProvider::close() + 66
3 libogrprovider.so 0x00000001263ee348 QgsOgrProvider::~QgsOgrProvider() + 164
4 libogrprovider.so 0x00000001263ee694 QgsOgrProvider::~QgsOgrProvider() + 14
5 org.qgis.qgis2_core 0x0000000114ebdfc6 QgsVectorLayer::~QgsVectorLayer() + 58
6 _core.so 0x0000000114077a30 sipQgsVectorLayer::~sipQgsVectorLayer() + 14

7 _core.so 0x0000000114079c41 release_QgsVectorLayer(void*, int) + 66
8 sip.so 0x0000000108ac48c1 forgetObject + 114
9 sip.so 0x0000000108ac46b1 sipWrapper_dealloc + 14
10 org.python.python 0x00000001070dc257 subtype_dealloc + 667
11 org.python.python 0x00000001070bbee2 dict_dealloc + 129
12 org.python.python 0x00000001070bd13d insertdict_by_entry + 249
13 org.python.python 0x00000001070bba5a insertdict + 51
14 org.python.python 0x00000001070bb221 dict_set_item_by_hash_or_entry + 40
15 org.python.python 0x00000001070bfd78 _PyModule_Clear + 177
16 org.python.python 0x00000001071218ce PyImport_Cleanup + 958

17 org.python.python 0x000000010712c2ec Py_Finalize + 297
18 org.python.python 0x000000010713e153 Py_Main + 2455
19 libdyld.dylib 0x0000000107a595ad start + 1


No comments:

Post a Comment

arcpy - Changing output name when exporting data driven pages to JPG?

Is there a way to save the output JPG, changing the output file name to the page name, instead of page number? I mean changing the script fo...