Creating a raster format

Using RAW as a raster format

Many raster formats can be simulated using the RAW raster format. Prior to creating a new raster format a developer should determine if the format cannot be specified using RAW. If so, it may be advantageous to create a raster type that creates a raster process definition including a fully defined RAW raster format. The RAW reader is very optimized. The following is a definition of the nodes within the Format Info section for .RPDef generator that are used when \raster\formatname is set as RAW.

Using a GDAL format driver

GDAL format drivers are supported by ArcGIS and so can be supported also by Image Server. Custom format drivers that conform to the GDAL implementation can be written and used. For more information on implementing a GDAL format driver, see http://www.gdal.org/gdal_drivertut.html.

To configure the GDAL format driver with ArcGIS, the GDAL format driver should be named with a prefix "gdal_" and with the extension ".so" or ".dll", such as gdal_fm.dll.”

More information on using these drivers within ArcGIS can found in: Supported raster dataset file formats

Using GDAL works well, but is not as efficient as writing a custom Image Server format reader. This is due to the additional levels of abstraction. One of the main optimizations that can be performed when using a custom Image Server format reader instead of GDAL is to handle Band Interleaved by Pixel (BIP) structured data directly. GDAL outputs the imagery as Band Sequential (BSQ) which may result in the data being converted from BIP to BSQ and then back to BIP used by Image Server internally. Additionally GDAL is generally used in applications that open and then keep an image open so will read the full header information. Within Image Server the files are opened and only the pixel data needs to be read. As with all Image Server raster formats it is important to ensure that the GDAL driver is thread safe

Creating a Custom Image Server raster format

Custom Image Server Format readers are efficient for reading pixel data in ArcGIS Image Server. As such readers only read pixels and need not read raster properties or metadata they can be created as being highly optimized. The MrSID and TIFF12 format readers used by Image Server use such Custom Image Server raster Formats.

A raster format reader must be thread safe and be capable of being run simultaneously as multiple processes on the same machine. If there is any shared data between processes or threads a suitable locking mechanism must be used.

Within Image Server raster formats caching is generally not performed. As Image Server needs to read such larger datasets, generally in a near random fashion the overheads in memory usage and possible fragmentation from caching at the format reader have found not to be generally beneficial.

FormatCore.dll implements a "C" based plugin system. A 'plugin' consists of a shared / dynamic link library using C exports and an XML based format description file. This design is loosely based on the TIFF file format - the Developer Kit interfaces support multiple layouts ( "component planes" - BIP or BSQ ) as well as tiled and untiled data. Depending on the format of the imagery the plug ins can support these structures directly in the interface else convert the data. Some image formats support ‘slices’ to handle for example temporal data. These Slice’s can also be supported if required, but generally the slice is assumed to be 0. The RPDef may define a specified slice and this will be passed to the format reader.

Each Raster format requires a corresponding .format file. This is an XML file stored in the "Program Files\ArcGIS\ImageServer10.0\Raster Formats\plugins" directory. The schema is defined by (link to FormatCore Plugin.xsd) which defines the name, module and dependencies. The name is linked to the format name defined in the RPDef. The module is the name of the DLL.

Following is the general flow as to how a Custom Image Server format is used:

On start up of the first Image Server component (Service Provider or Service Editor) the Raster Formats\Pluging directory is searched for all .format files and read. For a mapping between the define format name and the DLL is created and initialize function called. This initialize is only called on start up.

For each access to an image, the CreateReader function is called. This function should create an instance of a class defining a ‘virtual reader’ that can handle one image at a time and return a pointer that is used as the handle.

The GetImageCount function may be called with the name of the file. This need not open the image, but should return the number of sub-images in the file. This can be used to check the existence of the file.

A call to OpenImage is made with the specified filename and sub-image ID. This should open the file for reading.

GetImageInformation is called which should return the structure of the image. Note that this information is also available in the .RPDef image info structure. It is therefore not absolutely necessary to read the header of the image from file. In some cases (eg TIF reader) the headers are actually read from the file to make the reading more resilient to changes in the file. For some image format reading the header information can be time consuming and this should be optimized or read instead during the creation of the RPDef.

If the structure returned indicates that a color map exists, the ReadImageColorMap function is called which should create the ImageCololMap structure.

The SelectImageROI function is then called, passing the region of interest (ROI) to be read. This defines the extents of the image that will be read. This enables the Format Reader to initialize or optimize itself to a specific area of the file. Subsequent ReadImage functions will make calls only within this region. The requests will form a sliding window that will move down (and potentially up) the ROI. The request will always be for the full width (Columns) for the ROI. The ROIs define the extent of the file to be read which may be large.

Depending on if the structure is ScanLine or Tiled, multiple ReadImageScanlines or ReadImageTile will be called. For ReadImageScanlines the start and end rows are defined. The size of such scan line requests is generally kept to within 4 MB by default. For ReadImageTile the Tile ID as x,y will be defined. For both these function, the term Plane is used to define a possible group of bands. A BIP structured image has one plane while a BSQ image has 3 planes.

Revised SelectImageROI with associated ReadImage functions may be called depending on how the image is to be read. The readers must be capable of reading part of a file that are earlier in the file. For some compressed file formats this can be challenging and costly. Image Server attempts to minimize the backward reading.

On completion of reading the image for the specified client request CloseImage will be called at which point the reader should be destroyed.

Only shutdown of the last ImageServer component the shutdown function will be called.

Creating a raster type

For the raster to be read by Image Server the definition of the format to read must be included in the Raster Process Definition associated with each image. These RPDef files can be created in different ways and including the georeferencing and metadata for the raster. Within Service Editor images are added using a Raster Type, which calls an associated Raster Process Definition Generator to create RPDef files based on user defined input. As the adding of a raster is only performed once the implementation need not be so optimized. For more information on Raster Types.

How to install a custom raster format

Shut down all instances of Service Provider and Service Editor running.

Copy .dll and .format into the "Program Files\ArcGIS\ImageServer10.0\Raster Formats\plugins" directory.