Choosing the resolution and domain of a low-precision spatial reference

Note:

This topic only applies to creating new datasets with low-precision spatial references in ArcGIS version 9.2 or higher. Before reading this topic, you should understand the differences between low- and high-precision spatial references. For information on the differences, see Migrating to high precision.

The ability to store coordinates in high-precision spatial references was a new aspect introduced at ArcGIS 9.2. Compared to low precision, high-precision storage allows you to store coordinates closer together while expanding the x,y,z,m domains (the area you can store) of the dataset. To take advantage of these benefits, converting data to high precision is recommended in most cases. However, data with low-precision spatial references continues to work in ArcGIS as it always has. If you need to deliver data to clients who have not upgraded or you are satisfied with your existing situation, you might decide to continue using low-precision spatial references for now.

In ArcGIS 9.2 and higher, you can continue to create low-precision datasets by using a pre-9.2 geodatabase that has not been through the upgrade process. If you create a new feature class or feature dataset with this geodatabase or import data into it, the new data will have a low-precision spatial reference. If you have upgraded a pre-9.2 geodatabase, existing feature datasets are still low precision until you upgrade the spatial references. Because of this, you can create a new low-precision feature class within an existing low-precision feature dataset. You cannot create low-precision stand-alone feature classes or feature datasets within any new geodatabase you create with ArcGIS version 9.2 or higher, or within any geodatabase you have upgraded to version 9.2 or higher.

Note:

If you're uncertain about the version of a geodatabase, right-click the geodatabase in the Catalog tree and click Properties. The Upgrade Status section shows the version.

When specifying a low-precision spatial reference, you must set the x,y resolution and domain. A large domain is only possible with a large resolution value. Consequently, values reflecting a small resolution demand smaller domains that may be too small for many projected coordinate systems such as universal transverse Mercator (UTM) or state plane. Therefore, when working with low-precision spatial references, you must carefully balance the trade-off between domain and resolution value.

This topic shows you how to set the x,y resolution and domain of a low-precision spatial reference. The first section discusses the fundamentals of resolution. The second section discusses different approaches for calculating appropriate resolution values for your data.

About x,y resolution and domain

A low-precision geodatabase stores coordinates as positive 4-byte integers that have a maximum value of 2,147,483,647. It may seem that you are limited to storing 1-foot or 1-meter precision with an integer, but that is not the case; you decide what your 4-byte integer units represent. If you need to store meter precision, you have 2.14 billion meters to work with (approximately 53 times the circumference of the earth). Or you could decide to store centimeters, in which case you would have 2.14 billion centimeters to work with (about one-half the circumference of the earth). Resolution values represent the minimum allowed separation between two coordinate values. Resolution values are used to convert decimal values to the integers stored in the geodatabase. The geodatabase converts them on the fly, allowing you to work with decimal numbers only, even if you are using the lowest-level ArcObjects API.

The following table shows examples of the minimum separation between coordinates, the coordinate system units, and the equivalent resolution values.

Minimum separation between coordinates	Resolution	Coordinate system units
1 centimeter	0.01	Meters
1 millimeter	0.001	Meters
2 centimeters	0.02	Meters
1 inch	0.083333	Feet

Examples of the minimum separation between coordinates, the coordinate system units, and the equivalent resolution values

The geodatabase actually does a little more to convert decimal values to integers. The decimal-based coordinate values are also shifted during the conversion. You only need to be concerned about this shift if you are manually calculating your resolution and domain.

The relationship between the resolution and the x,y domain is directly proportional. Because you have 2.14 billion integers, there is an outer edge for the domain. As your resolution values get smaller, the extent of your domain also gets smaller. When you attempt to add features outside the domain, you will get the following error: "The coordinates or measures are out of bounds." It is important that you do not make your resolution values so small that you will not be able to add features for your entire study area. However, with approximately 2.14 billion integers to work with, in most cases you can avoid this problem by simply setting the resolution values appropriately. For example, you can store the entire world with 1-meter resolution but only one-half of the world with 1-centimeter resolution. Using a decimal degree-based geographic coordinate system like NAD83, you could use 1.9-centimeter resolution for the entire world in a single feature class.

How to set the x,y resolution and domain

Before you specify the x,y resolution and domain, there are three things to consider:

• Will the resolution maintain the accuracy of your data collection?
• Will the domain cover the entire extent of your study area?
• For ArcSDE geodatabases, is the resolution large enough to minimize storage and maximize performance?

You don't always need to worry about all these issues. Many times, you can let the default settings generated by the software deal with these issues for you. Below are three different approaches. Choose the one that is most appropriate for your application.

A. Take the defaults when importing data.

B. Specify the domain and accept the resolution default.

C. Specify the resolution and domain.

Approach A: Take the defaults when importing data

This is the easiest of the approaches because you simply take the default resolution and domain generated for you when you import data. Use this approach if you

• Have at least one vector dataset or a group of tiled datasets that covers the entire extent of your study area
• Want the smallest resolution possible within your study area

If you have a dataset that covers the entire study area, import the dataset first and accept the default values for the resolution and domain. The defaults will create a domain that encompasses all the features with a little room to grow. If you have tiled datasets that together cover the entire study area, calculate a domain that encompasses all the datasets using the Create Spatial Reference tool. Then create an empty feature class with this domain and load the tiled data into it.

Using this method, the resolution value will be minimized within the default domain. Because the resulting resolution could be small, this would not be the best approach if you are trying to get the optimum performance out of an ArcSDE geodatabase. However, this approach will ensure that all your data will fit inside the domain and you are using the smallest resolution possible for your data.

As you create or import subsequent datasets to the geodatabase, use the spatial reference calculated from this original feature class. You can do this by importing the spatial reference from this feature class whenever you create new feature classes or feature datasets. You can also set your geoprocessing settings to use the spatial reference from this feature class by following these steps:

1. In ArcCatalog or ArcMap, click the Geoprocessing menu and click Environments.
2. Expand Output Coordinates.
3. For Output Coordinates, click As Specified Below.
4. Next to the following text box, click the folder icon.
5. On the XY Coordinate System tab, click Import.
6. Navigate to and choose the first feature class that you imported into the geodatabase.
7. Click Add.
8. Click OK on all the open dialog boxes.

Once you've finished these steps, all subsequent geoprocessing operations, including importing new data, performed by the current user on this machine will use this spatial reference.

Approach B: Specify the domain and accept the resolution default

This approach helps you determine the domain for your study area, then minimizes the resolution within that study area. Use this approach if you

• Do not have a single vector dataset that covers the extent of your study area, but you can define your study area on a map
• Want the smallest resolution possible within your study area

The result of this approach will be exactly the same as approach A; therefore, it has the same strengths and weaknesses. Before you can begin, you must know the coordinate system that you plan to use. For information on choosing a coordinate system, see the "Map projections" topic in the ArcGIS Desktop Help. If you plan to use the state plane or UTM coordinate systems, you can find data defining the zone locations at <ArcGIS installation location>\ArcGIS\Reference Systems in the usstpln83 and utm shapefiles.

First, determine the domain for your study area:

1. Start ArcMap and add reference data for the world or your area of interest. Look for reference data in the following locations:
   • ESRI Data & Maps CD-ROM (included with ArcGIS)
   • <ArcGIS installation location>\ArcGIS\Metadata\Data
   • Geography Network
2. Set the coordinate system of the data frame to the one that you want to use for the new dataset.

   a. Open the data frame properties.

   b. Click the Coordinate System tab.

   c. Open the Predefined folder and navigate to the coordinate system that you plan to use.

   d. Click OK.

3. Zoom in to the part of the world that you plan to use as a study area.
4. Use the New Rectangle tool on the Draw toolbar to draw a rectangle defining your new study area on the map.
5. Right-click that new rectangle and click Properties.
6. Click the Size and Position tab.
7. Under Position for Anchor Point, click the lower left check box.
8. Copy and paste the coordinates in the X and Y text boxes into a text file. Delete the unit measure at the end of the coordinates. These coordinates are the lower left corner of your study area.
9. Under Position for Anchor Point, click the upper right check box.
10. Copy and paste the coordinates in the X and Y text boxes into a text file. Be sure to delete the unit measure at the end of the coordinates. These coordinates are the upper right corner of your study area.

Now apply this domain when creating a new feature class:

1. In the Catalog tree, navigate to your version 9.2 or higher geodatabase, right-click, point to New, then click Feature Class.
2. Type an appropriate name, such as StudyArea.
3. Choose the feature type and whether z- or m-values are supported.
4. Select or import your coordinate system.
5. Click Next.
6. Copy and paste your coordinates out of the text file into the appropriate text boxes. Notice that the resolution adjusts as you change the domain.
7. Click Next.
8. Add any fields to the feature class.
9. Click Finish on the New Feature Class wizard.

Now you can import the spatial reference from the StudyArea feature class for all other data that you create in that study area. You can also set your geoprocessing environment so all new data created from geoprocessing operations uses this spatial reference. See approach A for how to set the geoprocessing environment to use a spatial reference from a feature class.

Approach C: Specify the resolution and domain

With this approach, you calculate the resolution and domain manually. Use this approach if you want to maximize the performance of low-precision data in an ArcSDE geodatabase.

Step 1: Calculate the resolution.

First, you must calculate an appropriate resolution. Set your resolution values to be 10 times smaller than the best accuracy of your data collection. This will ensure that the precision of your data collection is maintained in the geodatabase regardless of how you manipulate the data with ArcGIS (geoprocessing, topology cluster tolerance, geometry operations, and so on). Consider the following examples:

Data collection method	Coordinate system units	Equipment precision	Recommended resolution
Digitize 1:250,000 map	Feet	+/-416 feet	1 foot
Professional GPS	Meters	+/-0.5 meters	0.05 meters
Survey with theodolite	Meters	+/- 5 millimeters	0.0005 meters

Set your resolution values to be 10 times smaller than the best accuracy of your data collection.

Calculating resolution based on data that uses a geographic coordinate system (GCS) is slightly more difficult because angular units (degrees) are not consistent everywhere on the planet. As the latitude changes, each degree of longitude represents a different length on the ground. If you want to calculate the resolution by using a linear unit with data in a GCS, you will have to perform some calculations. If you calculate an appropriate resolution when your angular units are at their largest, you will maintain even more precision in areas where angular units are smaller. For example, if you are maintaining 1-meter precision where 1 degree equals 100 miles on the ground, your geodatabase will maintain 1-centimeter precision where 1 degree equals 1 mile on the ground. In a geographic coordinate system, angular units are largest at the equator. Resolution will be the inverse of the linear length in 1 degree at the equator. As mentioned above, the resolution value should be divided by 10 to account for any ArcGIS processing operations. You can use the following equation:

Resolution = 360 / GCS equatorial circumference / 10

For example, GCS_WGS_1984 has a circumference of 40075016.7 meters. Thus

Precision = 360 * 40075016.7 / 10 = 8.9831528 x 10-7

Another option is to multiply the semimajor axis of the GCS by the number of radians per angular unit, which is the equivalent of

Resolution = 1 / (Semimajor axis * 2π/ 360 * 10)

You can find this technical information about your GCS by opening its property dialog box in the Catalog tree. If you don't see the Coordinate Systems folder in the ArcCatalog tree, you can make coordinate systems visible from the General tab of the ArcCatalog Options dialog box in ArcCatalog.

Step 2: Check the resolution against your study area.

To validate that your resolution will work given your study area, divide the greater of the width or height (range) of your study area by the resolution. If the result is less than 2,147,483,647, your data can fit inside a domain with your chosen resolution.

Even though your data can fit inside a domain, your coordinates may fall outside the coordinate system boundary. Consider the following fictitious dataset with map units of meters:

A range of 800,000 (the width) divided by a resolution of 0.0001 equals 800,000,000, which is less than 2.14 billion; therefore, the data will fit. However, the upper right corner of the study area will be 1,000,000,000x, 4,060,000,000y (that is, [1,000,000x] / 0.0001 and [4,060,000y] / 0.0001). Notice that the y-value is outside the 0 to 2.14 billion range by about 1.9 billion units. To store these coordinates inside the geodatabase, you must shift the domain to surround the data.

Step 3: Calculate an appropriate minimum x,y.

Before you can shift the domain to surround your data, you must identify the center of your domain in map units. The goal is to place your data in the center of the domain so your data can expand in all directions if necessary. All the calculations for shifting the coordinate system are in coordinate system units.

First, find the center of the domain in integer space:

2,147,483,647 / 2 = 1,073,741,823

Next, convert the center to coordinate system units by multiplying by the resolution. This example uses a resolution of 0.001:

1,073,741,823 * 0.001 = 1,073,741.823

Now that you have found the center of the domain in coordinate system units, you need to calculate a new minimum x and y of your domain. The formula for calculating the minimum x and y of your domain is as follows:

Min X = ([DataMinX + DataMaxX] / 2)—Domain center in coordinate system units 
Min Y = ([DataMinY + DataMaxY] / 2)—Domain center in coordinate system units

This equation finds the minimum coordinates of your domain to locate the center of your data at the center of the domain. Remember, all these calculations are in coordinate system units. Examine this equation for the x dimension given the example data.

First, find the center of your data:

(DataMinX + DataMaxX) / 2 
(200,000 + 1,000,000) / 2 = 600,000

Next, find the difference between the center of your data and the center of geodatabase space:

Min X = 600,000 - 1,073,741.824 = -473,741.824

Because this is a negative number, the domain will shift to the left. Remember, the shift is applied to the domain, not the data. The shift is calculated for both dimensions, so you would need to repeat this process for the y-coordinate. If you are trying to maximize performance in an ArcSDE geodatabase, do not center the domain. Instead, set the minimum domain values as close to the data as possible.

Step 4: Create the dataset.

Once you have calculated the resolution and minimum x- and y-values, you are ready to create a feature dataset or stand-alone feature class. The first time you create one of these with the New Feature Class or New Feature Dataset wizard, uncheck the Accept default resolution and domain extent check box and enter the resolution and minimum x- and y-values you calculated. The maximum x- and y-values will be calculated automatically. For all subsequent data that you import or create, you can simply import this spatial reference. You can also set your geoprocessing environment so all new data created from geoprocessing operations uses this spatial reference. See approach A for how to set the geoprocessing environment to use a spatial reference from a feature class.

Specifying the z- and m-resolution and domain

The z- and m-domains are easier to calculate than the x,y domain. Examine your data and enter the lowest number for the minimum value and the resolution to support its accuracy. You can calculate z- and m-resolution the same way you calculated precision for the x,y coordinates. Just like x,y coordinates, you have 2,147,483,647 integers with which to work. Generally, it is not necessary to center the z- and m-domains about the data, as you can set an absolute minimum based on your data.

When calculating the minimum for a z-domain, you could use the lowest point on the earth (-11,033 meters—Mariana trench). Generally, measure coordinates are positive numbers, so a minimum value of 0 may be appropriate. You can also set the minimum m-domain to have a slight negative offset to account for negative values that could be produced by the extrapolation of measures during operations such as Calibrate. You could then correct these negative values later instead of rejecting them during the extrapolation.