I wonder if anyone else has run into this problem; and I'm baffled as to its reason and solution.
Whenever I do an export of features that have large-number attributes (over 14 digits) to a shapefile, they seem to lose precision. ArcGIS presents them in a default format of e-notation if they are large, but it seems to lose the last few digits of the number, for example:
728400000032249 becomes 728400000032000
Any ideas?
Answer
From the ESRI forum ArcGIS Desktop - Data Management (General) forum
Precision for shapefiles
A quote & credit to whuber (posted in 2007)
Double precision floats maintain 52 bits of precision, regardless of the sign of the number. That translates to 52 * log(2) = 15.65 decimal places. Some exceptional tiny values--less than about 10^-308 in size--have less precision; these are said to be "denormalized." Otherwise, the precision is the same for all doubles; there is no "typically" about it.
Your "bunch of extra zeros" is correct when it is understood in binary. In base 10, it is rare for a conversion from singles to doubles to result in additional zeros.
Shapefiles have to use better than single precision, which has only 23 bits of precision (about 6.92 significant decimal digits). Typical earth coordinate systems, like UTM, can have coordinates extending into the many millions of meters: that would limit shapefiles to about one meter precision when using such systems. One meter is not precise enough for large scale work. Thus, ESRI had no real choice in the matter when they designed shapefiles to use floating point coordinates rather than scaled integer coordinates.
source: http://forums.esri.com/Thread.asp?c=93&f=986&t=234634
Solution - avoid the shapefile format (stick to geodatabase) for better precision or accept the 1 meter precision.
No comments:
Post a Comment