dem - What is a good metric for mapping accuracy

I have recently received a dataset that supposedly mapped the altitude a given region (a DEM). I would like to determine the accuracy of the new map by comparing it with a previously generated DEM. I am not sure what kind of metric I can use to ensure that the performance is good enough. (Good enough means that the horizontal deviation of a feature can not be off too far and that the vertical deviation can not be off too far)

One way to attack this would be to descritize the space into small chunks and then look at the distribution of the vertical residuals. If that block has vertical residuals that are less than the vertical deflection then I can assign that block a TRUE value and if it does not I could assign the block a FALSE value. I can then simply calculate the mean of the TRUE/FALSE values across all the blocks to get an overall value such as 67% or something which would tell me how good my mapping performance was.

There are huge problems with this method:

1. It does not seem to factor in the horizontal deflection requirement


2. It seems as though it would violate the iid (identical independently distributed) assumption normally required of the mean calculation. This is the same as stating that the individual boxes would be correlated.


3. There does not seem to be a defensible method to choose how big the descritization box size should be.

An alternative could be that I simply place special markers with known 3D locations and then simply measure their vertical and horizontal displacement. This seems like it could share some of the same problems as the earlier method. (how many markers should I place? are the measurements correlated if they are too close?)

Before running off and doing something silly, I wanted to see if the community has a golden metric or method that has good statistical properties. Something that is continuously measured across the DEM, takes into account both the vertical deflection and the horizontal deflection, and I can look at the distribution and make a defensible argument? Please let me know if this question is unclear and thanks for any suggestions.

P.S. Could somebody could comment on the need of the two individual mapping accuracy levels (one for vertical deflection and one for horizontal deflection)? It seems to me that a horizontal mis-mapping would be reflected in a vertical deflection. Is it standard to have two separate requirements for mapping?

Answer

The the accuracy refers to the absence of bias. So you can estimate your accuracy as the mean of the difference between your two datasets. It is often difficult, and sometimes impossible, to distinguish horizontal from vertical bias. However, it is possible if you are in a rugged terrain. Alternatively, you can directly work in 3D for your displacement, but this will be problematic in case of flat surfaces.

For the horizontal displacement, you should check if moving the surrounding of one location maximize the correlation or minimize the RMSE (local template matching). Then you compute the mean displacement and you obtain the horizontal accuracy. The size of your block will affect your processing time and the resolution of your measurement, but you will get better estimates of the correlation with larger block, so it is a question of compromise. It is true that a systematic sample could lead to spatial autocorrelation (especially if your terrain has some dominant orientation), but you can avoid this by using simple random sampling for your points.

The vertical accuracy is easy if you assume no horizontal bias. Note that the effect of horizontal bias could be compensated in terms of vertical bias: if you have a shift to the East, the height of East facing sides will be overestimated, there is no effect on the vertical shift on horizontal surfaces, and the height West facing slopes are underestimated.

Note that the bias can be removed if you know which dataset is correct. Then you can also estimate the precision based on the standard deviation.