Understanding coordinate management in ArcGIS

---

The spatial reference includes the coordinate system information, XY/Z/M tolerance and  resolution values, and XY/Z/M domain extents.

The resolution and domain extent values determine how the coordinates of a geometry are stored. Coordinates must align with a grid mesh that is defined by the domain extent and resolution values. Resolution values are in the same units as the associated coordinate system. For example, if a spatial reference is using a projected coordinate system with units of meters, the x,y resolution value is defined in meters. For many applications, a resolution value of 0.0001 meter (1/10 mm) is appropriate. You should use a resolution value that is at least 10 times smaller than the accuracy of the data. If a coordinate system is unknown, or for Ms, you need to set appropriate resolution values according to the data without knowing the unit of measure.

The following equation is used to generate integers for feature data sources that persist coordinate values as integers:

In this equation, the left side is the integer value to be persisted. Round is a function that rounds its double-precision, floating-point operand to the nearest 53-bit integer. Snapping a coordinate to a grid means applying the equation (1) previously shown, followed by applying its inverse, to generate a double-precision coordinate rounded to the specified resolution and corresponding to an integer grid point. Personal geodatabases, for example, persist coordinates as double-precision values, but those values have been snapped. Coordinates are not snapped when working with a shapefile.

If you want to compare coordinates, you have several options. If your coordinates have not been "seen" by ArcGIS in any way, then the method of comparison is up to you. Coordinates that have been processed by a topological operator (for example, Simplify or Intersect) or have been stored into a personal, file, or enterprise geodatabase, however, have been snapped, as previously described. In that case, you have the following options for coordinate comparison:

IRelationalOperator.Equals is a tolerance-based comparison operator. It uses the x,y tolerance value from the spatial reference of the geometries being compared to determine if two x,y coordinate values are equal. This method does not compare Zs, Ms, or point ID vertex attributes.

IClone.IsEqual is a tolerance-based comparison operator but also compares vertex attributes using the z- and m-tolerances when comparing z- and m-attributes. There is no tolerance for the point ID attribute.

IGeometry.SnapToSpatialReference moves the coordinates of a geometry to their closest grid mesh points (see the previous discussion of snapping). ISpatialReference2.ApplyPrecision is a low-level method that accomplishes the same thing but operates on arrays of point structures (WKSPoints) rather than on geometry objects. ISpatialReference2GEN.ApplyPrecision is available for those languages that cannot pass C-style arrays to ArcObjects methods. It is implemented on the SpatialReferenceEnvironment object and delegates to ISpatialReference2.ApplyPrecision. It is important to understand that snapping is a stable operation. The floating-point representation of a coordinate might change the first time it has been snapped, but not thereafter.

Use SnapToSpatialReference or ApplyPrecision if you want to compare double-precision quantities exactly (that is, if you want to avoid using the tolerance property). In general, it is difficult to predict if an unsnapped double-precision value will compare exactly with a snapped value, even if they appear to be equal when printed as a sequence of base 10 digits. A good reference for gaining a precise understanding of the behavior of double-precision quantities is What Every Computer Scientist Should Know About Floating-Point Arithmetic, by David Goldberg, published in the March 1991 issue of Computing Surveys. Copyright 1991, Association for Computing Machinery, Inc.

One recommendation is to keep the minimum domain x-, y-, and z-values as close as possible to the extent of the data. This minimizes the exponent shifting that occurs during the floating-point subtraction operation in the preceding equation (1), and also allows better compression of the integer versions of the coordinates, as described in the following text.

In addition to the integerization of coordinate values, ArcSDE and file geodatabases compress the resulting integer coordinates by removing leading zeros among other things. Coordinates with large numeric values after integerization do not compress as well as those with smaller numeric values; therefore, coordinates clustered near the upper right corner of the domain extent do not compress as well as those near the lower left or minimum domain extent.

If you have worked with geometries before ArcGIS 9.2, the resolution values are the inverse of the precision values. The precision values are also known as the xyUnits, zUnits, and mUnits or the scale factor. At 9.2, the terms resolution and precision are often treated as synonyms. There might be parts of the application programming interface (API) or documentation that still refer to precision or scale factor when they should use the term resolution instead. The word precision is also used to describe a spatial reference with an enhanced grid, as discussed next.

The coordinate values of a geometry must fall within the appropriate x,y-, z-, and m-domain extents. The largest legal map coordinate value is the upper right of the domain extent. This coordinate value must also be represented as an integer according to the preceding equation (1). The domain extents and resolution values are connected. Together they form a grid mesh with the resolution defining the mesh separation or cell size. Internally, the resolution and domain extents are used to define an integer grid.

At version 9.1 and prior releases, each integer coordinate was allotted 31 bits. Starting with version 9.2, 53 bits are used. Thus, for a given resolution, the domain extent can be much larger or alternatively for a given domain extent, the resolution can be much smaller than is possible for data using a basic-precision spatial reference. A data source that stores 53-bit coordinates is always associated with a high-precision spatial reference. Existing datasets that have not had their spatial references upgraded are always associated with low-precision spatial references.

As an example, if the minimum domain value is 0 and the resolution is 1, the maximum domain value for a high-precision data source is 9007199254740990 or 2⁵³-2. If the resolution is 0.0001, the maximum domain value is 900719925474.0990. These values are for high-precision spatial references. Before version 9.2, spatial references were what is now called basic precision. A basic-precision spatial reference uses maximum domain values of 2147483647 or 2³¹-1. If the same resolution value of 0.0001 and a minimum domain extent of 0 are used, the largest value a geometry could have is 214748.3647. That is too small for many projected coordinate systems such as universal transverse Mercator (UTM) and State Plane; therefore, when you work with basic-precision spatial references, you must carefully balance the trade-off between domain extent and the resolution or precision values.

If possible, always work with high-precision spatial references. By default, a new spatial reference is high precision. If you are editing or creating data for a geodatabase of any type that has not had its spatial reference upgraded, you have to continue to work with basic-precision spatial references. New Component Object Model (COM) interfaces are available on the various spatial reference objects to determine whether one is basic or high precision. A new interface is provided on the SpatialReferenceEnvironment object to convert between high- and basic-precision spatial references using certain assumptions.

In addition to the resolution and domain extent properties of a spatial reference’s coordinate grid, the x,y-tolerance property is also important. The x,y-tolerance property is applied to x,y coordinates during relational and topological operations. The x,y-tolerance property of a spatial reference describes how far a point is allowed to move in the x,y plane whenever it is necessary to determine if two points are the same. The minimum allowable tolerance value is twice the resolution. The z-tolerance property is used when validating topologies in a geodatabase. Different tolerance values can produce different answers for relational and topological operations. For example, two geometries might be classified as disjoint (no points in common) with the minimum tolerance, but a larger tolerance might cause them to be classified as touching.

To achieve precise and predictable results using the geometry library, it is essential that the spatial reference of geometries in a workflow is well defined. When performing a spatial operation using two or more geometries, for example, an intersection, the coordinate systems of the two geometries must be equal. If the coordinate systems of the geometries are different or undefined, the operation can produce unexpected results. A spatial reference should have explicitly defined tolerance values. In two-at-a-time geometry operations, such as the methods found on the IRelationalOperator and ITopologicalOperator interfaces, the tolerance value of the left operand geometry is used. If the tolerance property of the spatial reference is undefined or there is no spatial reference associated with a geometry, a default grid guaranteed to contain the operands, and the minimum allowable tolerance based on the grid are used.

The spatial reference is not defined when creating an instance of a geometry. It is the developer's responsibility to define a spatial reference that makes sense for the geometry and for the operation. If an existing geometry is coming from a feature class, then usually the spatial reference is well defined and the geometry can be used directly or projected without problems.