georeferencing - Generally accepted root mean square (RMS) error for rectifying topographic maps

This is a generic inquiry on the common practice/standard for determining the acceptable values of RMS when georeferencing topographic maps. Is there an absolute value?

Some literature suggests that it should be "less than or equal to 1/2 of the side of a cell which make up the total resolution of the image."

Answer

No, there is no absolute value for RMS, because it depends on the quality of the map being georeferenced, the quality of the target (base) map, and the purpose of the georeferencing. In particular, any advice that relates RMS to cellsize is misinformed, because cellsize reflects precision in the digital representation of an image whereas the RMS error reflects average accuracy (assuming the basemap is perfectly accurate). Although distinguishing precision and accuracy may seem like aimless pedantry, confusing them is a basic mistake with practical consequences.

All this is rather vague, so let's look at a specific example. Recently I received a series of screenshots of maps showing soil sample locations. To obtain coordinates, I planned to georeference these screenshots to an orthophoto base map and then digitize the points with heads-up digitization. Among the considerations were:

1. The orthophoto base map has 0.3 m cellsize.


2. The screenshots have approximately 2 m cellsize.


3. The soil sample locations were not surveyed; they were located "by eye" on the map when the sampler was in the field. The client estimated the accuracy was about 3 m, but 10 m is more likely.


4. The screenshots have few sharp details: they are primarily contour lines, with occasional fencelines (which are not clearly visible in the orthophoto). Thus establishing many good links would be time-consuming and error-prone.


5. There was likely some local distortion in the screenshots, meaning that high accuracy (low RMS) can be achieved only with complex transformations.


6. It was important to digitize the soil sample locations so that relative distances were fairly accurate for nearby points, but absolute accuracy was unnecessary, because one outcome of the study will be to obtain many more soil samples that refine and make more precise this preliminary survey.

To obtain an RMS of half the larger cellsize would require a high-order polynomial transformation or warping across a grid of points, calling for establishing a network of around 50 - 100 good links between the images: one to several hours of careful work, most likely, given the difficulty of even finding visible links. To obtain an RMS of half the smaller cellsize would require an order of magnitude more effort: days of work. However, for the purposes of the study an RMS of 5 m would be more than sufficient. This was achieved with 7 links and an affine transformation, just a few minutes' work. Note that this RMS is several times greater than the larger of the two cellsizes in the images.

This example illustrates how blindly following bad rules of thumb can be costly. Pay attention first to your data quality objectives; everything else follows from them.