cloud gis - Is there an architecture for distributed geoprocessing?

Suppose I have 50 computers on my LAN. Each computer has a geodatabase for all parcel polygons in a particular state in the US.

I'd like to write a geoprocessing task that finds all parcels valued over x $/acre that are within y feet of another parcel that is valued at less than z $/acre.

I'd like to formulate and run this query without knowing or caring that the data is distributed across 50 computers. Keep in mind the boundary conditions: I also want the query to return cases where expensive parcels in one state are near inexpensive parcels in another.

Is there an architecture that supports this sort of distributed geoprocessing?

The architecture can be described abstractly, or as an implementation specific to Azure or Amazon Web Services. Or, preferably, as a typical office where computers sit idle at night with plentiful ArcGIS desktop licenses.

Answer

1. store all your parcels in one central database


2. formulate a grid over the USA made of squares N feet on a side, where N is such that the number of parcels that fit within N will not blow out the memory on one of your nodes


3. create a table in your database with one row per grid square, an id column a geometry column and a status column


4. each node runs a small program that
   
   
   
   
   1. find the next unprocessed square
   
   
   2. marks it as in-process
   
   
   3. pulls all the parcels ST_DWithin(square,parcel,maxfeet)
   
   
   4. does the actual query
   
   
   5. writes back the query answer to a solution table in the central database
   
   
   6. marks the square as complete
   
   
   7. return to 1
   
   
   
   

The obvious failure case is as your radius-of-interest in the parcel query grows large enough that large portions of your dataset are potential candidates to match each parcel.