As a followup to my previous question (Converting SimpleGeo Places GeoJSON file to shapefile), I am now struggling with what I perceive might be an ogr2ogr "memory" limitation, but I cannot be sure. I have successfully converted the SimpleGeo dump files into proper geojson files (verified that are directly readable in QGIS).
Now I want to convert this data to SHP or other formats (SQLite, PostGIS). I am running the following command with success on smaller files (where "places_fr.gejson" has a file size of ~118MB):
ogr2ogr -f "ESRI Shapefile" places_fr.shp places_fr.geojson
When I attempt to run this same exact command with a geojson file that is 340MB, ogr2ogr crashes in one of two ways:
- "lh_table_new: calloc failed" (first attempt)
- Windows throws an "ogr2ogr.exe has stopped working. Please close the program." (subsequent attempts)
I am running the ogr command via FWTools in a Windows 7 environment dual core with 8GB of installed RAM (4GB available, 12GB virtual memory available). Is there a memory limit to ogr2ogr converting these datasets? I have also tried converting into SQLite with the same results, so I don't think it is an SHP conversion issue. Thanks.
Answer
Forewarning: given the amout of memory you are reporting on your system I suppose you are using a 64 bit build of Windows. If not this solution does not apply.
The memory limit for all 32 bit applications on Windows (regardless of the Windows version, so it is true also for 64 bit Windows) is 2GB. You might be hitting that limit, to confirm run ogr2ogr and check memory usage with task manager.
If the problem is indeed due to a memory issue you can do two things:
build yourself a 64 bit version of GDAL (I could not find readily avail binaries)
patch ogr2ogr.exe with this tool: http://ntcore.com/4gb_patch.php. After patching (make a backup copy of course) the binary should be available to address a grandtotal 4GB
No comments:
Post a Comment