Saturday, 21 May 2016

database - What is the best hack for importing large datasets into PostGIS?


I have to import large Shapefiles (> 1 million records) into PostGIS, and I have been wondering about the best way to do it.


enter image description here



In my question I used the word "hack", instead of tool, on purpose because I think this is not so much a matter of which tool, but which set of steps, or configuration settings to use. So far, I have tried the SPIT plugin (QGIS), the shp2pgsql Postgis tool and the GDAL ogr2ogr tool. You can view my full review on this post. So far, I find all of them really unresponsive, when dealing with a large dataset. I was wondering if someone experienced a similar issue, and if you could share something about the approach.



Answer



I made a test for you:



  • PostgreSQL 9.3

  • PostGIS 2.1

  • Windows 7

  • i7 3770@3.4 GHz processor

  • GDAL 2.0-dev 64-bit

  • shapefile of 1.14 million polygons, file size 748 MB



Ogr2ogr command:



ogr2ogr -f PostgreSQL PG:"dbname='databasename' host='addr' port='5432' user='x' password='y'" test.shp --config PG_USE_COPY YES -nlt MULTIPOLYGON



Total time:1 minute 30 sec


No comments:

Post a Comment

arcpy - Changing output name when exporting data driven pages to JPG?

Is there a way to save the output JPG, changing the output file name to the page name, instead of page number? I mean changing the script fo...