I am trying to run an Intersect process in arcgis 10 sp 3 with 2 file sets (aspect and slope) from up to a 1m DEM across an area of 65,000sq km. The aspect has 9,930,384 records and the slope has 31,435,462 records (approx 12GB total in 2 file geo-databases).
I have run repair geometry about 3 times and now the datasets do not report any errors (each time took over 30h).
Now I get
Executing (Intersect): Intersect "D:\SCRATCH\Projects\106\data\7_asp_Merge.gdb\asp_HghstRez_M_rep #" D:\SCRATCH\Projects\106\data\working\working.gdb\AsSl_Int ALL # INPUT Start Time: Sun Oct 23 02:19:10 2011 Reading Features...
Processing Tiles...
ERROR 999999: Error executing function.
Invalid Topology [Too many lineseg endpoints.]
Failed to execute (Intersect).
Failed at Sun Oct 23 04:09:12 2011 (Elapsed Time: 1 hours 50 minutes 2 seconds)
Is this really a topology issue or a file size issue?
I have tried to use the ArcINFO SPLIT tool but it fails even with over 1TB of free space on the drive and on smaller file set causes jagged edges. I can’t use DICE as the areas to intersect between the asp and slope must be exactly the same. I understand that on large datasets ESRI cracks (automatically tiles) the datasets –can this be introducing issues? Is there any more info I can provide to problem solve.
The spec of the machines is more than ESRI minimum –we have 16GB RAM, Intel Xeon, Windows 7, 64-bit, 2 x One TB disks and more than 1.2TB free on the drives. All files used in the process are on the local drives.
just found this explanation (July 2nd, 2012) that gives a lot of helpful hints on resolving the issues.
http://blogs.esri.com/esri/arcgis/2010/07/23/dicing-godzillas-features-with-too-many-vertices/
Answer
Very few contiguous cells in a detailed DEM will have identical values of both slope and aspect. Therefore, if the input features represent contiguous areas of common slope and common aspect, we should expect the result of this intersection procedure to have, on average, almost one feature per cell.
There were originally 65,000 * 1000^2 = 6.5 E10 cells in the DEM. To represent each of these requires at least four ordered pairs of either 4-byte integer or 8-byte floating coordinates, or 32-64 bytes. That's a 1.3 E12 - 2.6 E12 byte (1.3 - 2.5 TB) requirement. We haven't even begun to account for file overhead (a feature is stored as more than just its coordinates), indexes, or the attribute values, which themselves could need 0.6 TB (if stored in double precision) or more (if stored as text), plus storage for identifiers. Oh, yes--ArcGIS likes to keep two copies of each intersection around, thereby doubling everything. You might need 7-8 TB just to store the output.
Even if you had the storage needed, (a) you might use twice this (or more) if ArcGIS is caching intermediate files and (b) it's doubtful that the operation would complete in any reasonable time, anyway.
The solution is to perform grid operations using grid data structures, not vector data structures. If vector output absolutely is needed, perform the vectorization after all grid operations are complete.
No comments:
Post a Comment