Friday 5 July 2019

gdal - Performance of google map tile creation processes


I know that question is rather vague but please bear w/me. I'm trying to get an idea of what sort of product performance - specifically timing - people have seen for various methodologies they have used to create google/bing map tiles. There are a slew of methods for how to do this (e.g. gdal2tiles, FME, maptiler, etc). An initial attempt at simply taking a large PNG and creating tiles using imagemagick, on a pretty decent linux server, yielded some pretty long processing times and so I wanted to see what other people are using in production. New tiles would need to be generated at least daily and so turnaround time on this is pretty critical.


The only real requirement is that it can run on a linux server. Obviously, free is better but I don't want to restrict myself to that. The input can be raw gridded/raster data or a large image. The output needs to be image tiles capable of being used as-is in google or bing maps.


Just for the sake of comparison, I'll say that the timings should be for google map's zoom level 7.


I appreciate everyone's help and again I want to apologize for how vague this question probably seems.


UPDATE: As far as the inputs, I currently have multiple (raw) data sources in various formats: netCDF, GRIB, GRIB2. In addition to the raw data itself, I also have the ability to generate really large images of that data which could then be sliced/tiled.


Ideally, I would just be chopping the image up but I am willing to try whatever will get me the fastest results.



Answer




Here are some of my results for the following raster file:


JPEG 14456x14490 14456x14490+0+0 DirectClass 62mb

$ time gdal2tiles [...]


Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.
Generating Overview Tiles:
0...10...20...30...40...50...60...70...80...90...100 - done.

real 5m7.675s

user 5m5.070s
sys 0m2.060s

$ time [pngnq && pngcrush for every tile, 4500 in total]


real    9m32.827s
user 18m34.190s
sys 0m27.230s

Yup, that's in minutes - I optimized for output size, not speed. The machine is a virtual Intel Xeon 2x3GHz, 4G of memory. (And clearly, gdal2tiles could make use of some parallelization.)


No comments:

Post a Comment

arcpy - Changing output name when exporting data driven pages to JPG?

Is there a way to save the output JPG, changing the output file name to the page name, instead of page number? I mean changing the script fo...