Transferring large raster datasets

There are several things to consider when transferring raster data, such as how large the data is, whether you can make a local copy or it needs to be a remote copy, and what distribution media is available. Raster data, such as raster datasets or raster catalogs, can be extremely large, with datasets reaching sizes of several terabytes. As a result, special considerations need to be made to move data that, as a unit, exceeds the capacity of commonly used transport media.

The main difference in methods used when transferring data is whether the rasters are stored in file geodatabases or ArcSDE geodatabases.

Transferring raster data in file geodatabases

If data is duplicated on-site, you can copy and paste the data with Catalog. The benefits of copy and paste are that you do not need extra space to store the export files and it is fairly simple to do. If the raster dataset is stored in a file geodatabase, you can copy the entire file geodatabase folder to the transport media. However, if the file geodatabase contains other types of data that you do not want shared, you will need to create a new file geodatabase on the transport media and specifically copy only the raster data into it.

Exporting raster data from ArcSDE

If the raster dataset is stored in an ArcSDE geodatabase, you can create a file geodatabase on the transport media and copy the raster dataset to it. The benefit of using the file geodatabase to transfer data is it is ready to use. You can simply connect the transport media and use the data. You can also use the ArcSDE sdeexport command line tool to move your data. Because the file size of the raster object could be too large for your transport media, you may need to break up the data into volumes. You can use the following to create volumes:

sdeexport –X option creates volumes

Some DBMS vendors provide a detachable database file capability. Oracle uses transportable table spaces—for example, at Oracle 10g, the data files are inter-OS capable—, whereas SQL Server uses detached files. Using detachable database file capabilities is much faster than loading an exported file.

The sdeexport –X option should be used if the export file size exceeds your transfer media's available storage space. The –X option specifies the maximum size of the export file volume. At ArcSDE 9.0, the maximum volume size is 4 GB. At ArcGIS 9.1, the volume size increased to the OS file limits. To determine the size of the export file that will be created, use the sderaster list operation with the –storage option.

The sdeexport command requires a large amount of sort space to export raster data; therefore, you will need to create temporary space that is approximately 4 percent of the size of the final export file. If sderaster –o list –storage returns a total of 100 GB for the export file, you will need to have 4 GB of temporary space available.

The following is an sdeexport example:

$ sdeexport –o create –t earth –f earth –X 4G
ArcSDE for <dbms> Mon Jul 19 16:06:36 PDT 2004
SDEX File Export         Administration Utility
---------------------------------------------------------
Exporting ArcSDE object to “earth" in SDEX 9.0 export format ...
Exporting table “earth".
  Spatial column "FOOTPRINT"
  Raster column "RASTER"
    1 features converted.
    1 features exported.

$ ls –l
total 5
-rw-rw-r--   1 sde    nuucp  4294967296 Aug  5 17:01 earth.000
-rw-rw-r--   1 sde    nuucp  4294967296 Aug  5 17:01 earth.001
-rw-rw-r--   1 sde    nuucp        9281019 Aug  5 17:01 earth.002

The example uses the following parameters:

• -o is the operation
• -t is the input table
• -f is the output file, or prefix for the volume
• -x is the maximum size for each volume that is created

Importing raster data into ArcSDE

If you want to import data into an ArcSDE geodatabase, you can use the ArcSDE sdeimport command line tool. The following is an sdeimport example:

$ sdeimport –o create –t earth_at_15m –f earth –k earth_at_15m

ArcSDE 9.0 for <dbms>  Mon Jul 19 16:06:36 PDT 2004
SDEX File Import         Administration Utility
----------------------------------------------------------
Importing SDEX from earth.000 ...
  Importing spatial column "FOOTPRINT"
  Importing raster column "RASTER"
    1 record read.
    1 record stored.

The example uses the following parameters:

• -o is the operation
• -t is the output table
• -f is the input file
• -k is the configuration keyword

Further considerations

You can use the DBMS export format as an alternative to moving larger raster objects from one location to another. Unless you move the entire ArcSDE instance, you will need to manually update the ArcSDE metadata tables when you import the data. The easiest way is to create an empty raster object, drop the tables of that object, and replace them with the imported ones.

SQL Server allows you to detach the database so the files of its file group may be copied to another location.

Using Oracle transportable tablespaces, you can export the metadata of a list of tablespaces to an export file. To do this, the data file associated with the tablespaces and the export files is copied to the new location and the export file is imported. The tables must be in read-only mode when they are exported and copied. This also requires the use of parameter files.