Troubleshooting an ArcSDE service

Most problems associated with starting an ArcSDE service occur because of a problem with the system environment. Often, a critical step was missed during the installation or configuration of the software.

The troubleshooting tips in this topic reference several different administration commands. See the Administration Command Reference for details on how the commands are used.

The giomgr exit codes are documented in the ArcSDE Developer Help. The ArcSDE Developer Help is available on the ArcSDE component installation DVD as well as on the Geodatabase Resource Center site.

Identifying problems

• The Windows Event Viewer

  For installations on a Windows operating system, you can use the Windows Event Viewer to look up possible problems. The Windows Event Viewer provides diagnostic information that may also help explain ArcSDE startup problems.To use the Windows Event Viewer

  1. Open the Control Panel.
  2. From the Control Panel menu, double-click Administrative Tools.
  3. From the Administrative Tools menu, click Component Services.
  4. From the Component Services menu, expand the Event Viewer folder and select the Application option.
  5. Look for a red stop sign icon in the Type column and the corresponding name of the ArcSDE service in the Source column. Double-click the ArcSDE service entry to bring up the Event Detail window.
  6. The Event Detail window includes a description of the problem.
  Although the Event Viewer will often include a description of the problem, you can also check the %SDEHOME%\etc\sde_<service>.log file. This file will contain further information relating to Windows startup problems. This file and other error log files are described below.

• Error log files

  ArcSDE and all supported database management systems (DBMSs) track their activities by writing messages of events to ASCII log files. The log files may be examined to trace errors that have occurred. ArcSDE writes to two log files: giomgr_<service>.log and sde_<service>.log. (If you are using a direct connection, the application writes error messages to sdedc_<dbms>.log instead of the application server's sde_<service>.log file.)

  • The giomgr_<service>.log file

    The giomgr_<service>.log file is an ASCII file that contains an entry for all giomgr process activities. (<service> is the name of the ArcSDE service.) Each time a user connects or attempts to connect to the ArcSDE service, a message is logged. When the user disconnects, another message is logged. The giomgr_<service>.log file also captures the startup and shutdown procedures of the ArcSDE service. However, this file does not contain specific error messages; it just shows you the activity of the giomgr process.

  • The sde_<service>.log file

    Whenever a gsrvr process encounters a problem, the ArcSDE service records an entry in the sde_<service>.log. Sometimes, the messages are warnings; other times, they point to ArcSDE service errors that you must address. When examining the sde_<service>.log file, keep in mind the messages will be written to this file when errors occur in the ArcSDE service process. Sometimes an ArcSDE application will report a problem related to ArcSDE, but this event will not appear in the sde_<service>.log. That is because the error has occurred on the ArcSDE client side and not the server side.Beginning with 9.3, the sde_<service>.log file also stores information when a malformed connection request is sent. This could indicate someone is trying to hack into the server. An entry could look like the following:

    [Mon Mar 05 09:35:14 2007] [0] [GIOMGR] Invalid connection request
    length of -2147477504 from [10.47.6.5:32923] [Mon Mar 05 09:35:14 2007] [0] 
    [GIOMGR] Could Not Get XDR Request.

    Notice that the log entry lists the IP address that is making the connection request.In the message, "Could Not Get XDR Request" was returned because the giomgr was not able to accept or understand the connection request due to the length being invalid.The sde_<service>.log is truncated each time the ArcSDE service is stopped and restarted.

  • DBMS error log files

    Each DBMS has its own way of logging errors. Consult the relevant DBMS administration guide to determine how your DBMS logs errors.

Troubleshooting the ArcSDE service on UNIX servers

System path variable issues

If the system path variables are improperly set or not set at all, you can get the errors below. Review how to set the path variables for your ArcSDE product in the ArcSDE Installation Guide.

• If the PATH environment variable does not include the $SDEHOME/bin directory, the following error message is reported:

  sdemon: Command not found

• If the library path environment variable does not include the $SDEHOME/lib directory, the following error message is reported:

  ld.so.1: sdemon: fatal: libsde90.so: open failed: No such file or directory
  Killed

• If the library path environment does not include the necessary DBMS library directory, an error message similar to the following is reported:

  ld.so.1: /ultra1/ora10gexe/bin/giomgr: fatal:
  libclntsh.so\: open failed: No such file or directory
  Killed
  Could not start ArcSDE - Check Network, $SDEHOME disk, DBMS settings and dbinit.sde.

ArcSDE service already started

If the ArcSDE service license has not been installed, the application server will not start. You must install the license using the keymanager administration command. Contact ESRI Customer Support to obtain a valid license.

If the input/output (I/O) manager is already running, the following message appears:

SDE Already Running
ArcSDE server license has not been installed

Temporary file permission problems

If any ArcSDE temporary files exist and they are not owned by the ArcSDE administrator, the following error message is returned:

ERROR: Cannot Initialize Shared Memory (-79)
Delete /tmp/<service name> and /tmp<service
name>.lock if present.
Could not start ArcSDE - Check Network,
$SDEHOME disk, DBMS settings and dbinit.sde.

To fix this, delete the temporary files /tmp/<service name> and /tmp/<service name>.lock. For example, if the service name is ESRI_sde, you would delete the files /tmp/ESRI_sde and /tmp/ESRI_sde.lock. You may have to log in as the root user to delete these files.

Files have been deleted from /tmp

If, after the ArcSDE service has been started, the files stored under the /tmp directory are deleted, the ArcSDE service will fail when a user either connects or disconnects. The service relies on UNIX socket protocol files created under the /tmp directory. As a rule, you should not delete the files under the /tmp directory. However, if you absolutely must, you should shut down the ArcSDE service before doing so. See Stopping a local ArcSDE service on Linux or UNIX, Stopping a local ArcSDE service on Windows, or Stopping a remote ArcSDE service for instructions on how to do this.

Problems relating to the DBMS

• If the DBMS is not started, you will receive an error message similar to the following:

  init_DB DB_instance_open_as_dba: -51
  DBMS error code: 1034
  ORA-01034: ORACLE not available
  Could not start ArcSDE - Check Network, $SDEHOME disk, DBMS settings, and dbinit.sde.

  Make sure the database instance has been started and try again.

• If the ArcSDE administrator password is not correct, you will receive an error message similar to the following:

  init_DB DB_instance_open_as_dba: -93
  DBMS error code: 1017
  ORA-01017: invalid username/password; login
  denied
  Could not start ArcSDE – Check Network,
  $SDEHOME disk, DBMS settings, and dbinit.sde.

  Reissue the command using the correct ArcSDE administrator password. If you have forgotten the password, you will need to contact the database administrator to reset it.

• If the ArcSDE administrator does not exist, you will receive an error message similar to the following:

  init_DB DB_instance_open_as_dba: -93
  DBMS error code: 1017
  ORA-01017: invalid username/password; login
  denied
  Could not start ArcSDE – Check Network,
  $SDEHOME disk, DBMS settings, and dbinit.sde.

  ArcSDE with most supported DBMSs requires an ArcSDE administrator to be present in the database. If it isn't, you or the database administrator will need to create one.

The SE_OUT_OF_MUTEXES (-109) error on a Solaris server

The Solaris operating system uses files to implement the POSIX shared semaphores that ArcSDE uses. If these files get left behind after an operating system failure or power outage, they can sometimes cause problems. The location of these files is controlled by the Solaris operating system. You will find them under either the /tmp or the /var/tmp directories as follows:

/tmp/.SEMD/
SDE_9.0_<instance>_iomgr_shared_semaphore
/tmp/.SEML/
SDE_9.0_<instance>_iomgr_shared_semaphore

or

/var/tmp/.SEMD/
SDE_9.0_<instance>_iomgr_shared_semaphore
/var/tmp/.SEML/
SDE_9.0_<instance>_iomgr_shared_semaphore

Following an operating system failure, if you fail to start the ArcSDE service and receive a -109 error, it is probably because the two shared semaphore files exist. If you find either of these files in either the /tmp or /var/tmp locations, delete them and try to start the ArcSDE service again.

Troubleshooting the ArcSDE service on Windows servers

Listed below are errors often encountered when starting an ArcSDE service on Windows. This list includes the error numbers, where applicable, and their likely causes.

997 Error starting esri_sde service

This error can occur with an incorrect or incomplete installation or configuration of ArcSDE:

"ESRI_sde service failed during initialization. 
Please check event log or error log files. 
Error starting ESRI_sde service(997) 
Could not start ArcSDE — Check Network, $SDEHOME disk, DBMS settings"

Solution

• Confirm that the correct password is being entered for the ArcSDE administrative user. If an incorrect password is being used, a -93 error will be present in the sde.errlog.
• Completely remove the current ArcSDE service and create a new service.
  1. Delete the service using sdeservice –o delete.
  2. Check the registry for orphaned service keys. Look in the following directories and delete the [esri_sde] folder/key if it exists:

     HKEY_LOCAL_MACHINE > SOFTWARE > ESRI > ArcInfo > ArcSDE > ArcSDE for [DBMS] > [service_name]

     HKEY_LOCAL_MACHINE > SYSTEM > ControlSet001 > Services > [service_name]

     HKEY_LOCAL_MACHINE > SYSTEM > ControlSet002 > Services > [service_name]

     HKEY_LOCAL_MACHINE > SYSTEM > CurrentControlSet > Services > [service_name]

  3. Reboot the server machine to flush the registry of the changes.
  4. Type set at a command prompt and verify that %SDEHOME%\bin is the first entry in the PATH environment variable (PATH=%SDEHOME%\bin;...).
  5. Verify that the ArcSDE service name and port number in your %SDEHOME%\etc\services.sde file match the entry in the C:\WINNT or WINDOWS\system32\drivers\etc\services file.
  6. Navigate to the %SDEHOME%\bin directory and re-create the sdeservice using sdeservice –o create.
  7. Run sdeservice –o list and verify that the service name, SDEHOME path, and license server are correct.

     If there are two back slashes (\\) in the SDEHOME path or one backslash (\) at the end of the path, manually remove it .

  8. At the command prompt, restart the ArcSDE service using sdemon –o start or net start <service_name>.

1068 Dependency failure

The DBMS to which the ArcSDE service is trying to connect could not be found. The most likely causes of this problem are

• The DBMS service is not started.
• The DBMS server has been removed.
• The DBMS connection information, entered when the ArcSDE service was created, is incorrect.

Make sure the DBMS server exists and the service is started, and check your DBMS connection information to ensure it is correct. If the error persists, use the sdeservice command to delete the existing ArcSDE service and re-create it.

1069 Login failure

Generally, this error implies that the Windows user who started the ArcSDE service is neither a Windows administrator nor a Windows power user. An incorrect password is another possibility.

If the system administrator account is not being used to start the service, make sure the user account is a member of the administrator or power user group.

1072 Registry was busy

Something is happening in the registry regarding the ArcSDE service entry. Perhaps the sdeservice with the delete operation was run, or the service has been opened with the registry editor. Alternatively, there may have been a problem with the Object Linking and Embedding Database (OLE DB) provider. Consult the installation guide for the correct version for the OLE DB provider for your installation.

1075 Service dependency deleted

The ArcSDE service is unable to locate the DBMS service to which it should connect. Make sure the DBMS service exists and is started. If the problem persists, use the sdeservice command to delete and re-create the ArcSDE service.

2140 Internal Windows error

The ArcSDE service wasn't able to complete the startup process. Examine the sde error logfile—%SDEHOME%\etc\sde_<sde_instance>.log—for possible clues as to why the ArcSDE service will not start.

Possible causes and solutions

• There is no entry for the instance in the system's services file. Check the system's services file for a service entry corresponding to the ArcSDE instance that is attempting to start.
• The entries in the system services file and the ArcSDE services file do not match. Verify that the entry in the system services file is the same as the entry in the file %SDEHOME%\etc\services.
• The last line of the system services file may not be read if it is not terminated with an end-of-line character. If the entry is on the last line of the system's services file, verify that the line ends with a carriage return.
• The TCP/IP component may be corrupted. Reinstall the TCP/IP component. Refer to the Windows documentation or Microsoft support site for details about how to do this.
• The ArcSDE administrator's password was entered incorrectly. If it was, use sdeservice –o modify –r SDE_DBA_PASSWORD to correct it.
• You are trying to connect to the geodatabase with an ArcSDE service using a Windows-authenticated user, but the Windows-authenticated user does not have permission to access the folder that contains the gsrvr.exe on the ArcSDE server. A user who is a Windows administrator on the ArcSDE server must grant READ and EXECUTE permissions on the %SDEHOME%\<dbms>exe folder to the Windows login attempting to make the connection.

An error was encountered while running ArcSDE Post Installation. Operation Failed, Unable to start iomgr. DBMS error code: 2714

You could see this error on a SQL Server database when attempting to start the service using the Post Installation wizard after the service has been created and an attempt to start the service has been made.

The likely cause is that the name of the geodatabase has been changed. A scenario in which this might occur is when you back up a current database and restore it with a different name for testing purposes within the same instance of SQL Server.

It is not possible to rename an ArcSDE geodatabase once it has been created. The name of the original database is hard-coded in database objects. When the name of a database is changed using a stored procedure or a database is restored to a name different from the original, the ArcSDE service for this database will not start.

Solution

Rename the database to its original name. Follow the steps below:

1. Shut down the service receiving changes.
2. Issue the following query in SQL Server Management Studio:

   ALTER DATABASE <database_name> MODIFY NAME = <new_database_name>If this is not possible because an existing database shares the same name, a different path must be used. If the original goal was to create a duplicate database for testing, a new database with a new name must be created. The ArcSDE Post Installation wizard should be used to populate the repository, authorize the software, and create the service. The data can then be copied between the two databases in ArcCatalog.

gsrvr.exe - DLL Initialization Failed or gsrvr.exe - Application Error: The application failed to initialize properly

On Windows, the ArcSDE service is started as a noninteractive desktop. The maximum amount of heap memory allocated to noninteractive desktops is limited by a Windows initialization parameter called SharedSection. If you receive this error message, you may need to change the SharedSection parameter.

The CONNECTIONS parameter in the SERVER_CONFIG table also restricts the number of concurrent connections a single ArcSDE service allows and may need to be increased.

• Heap memory

  The maximum amount of heap memory available for all desktops, both interactive and noninteractive, is 48 megabytes. Since the amount of memory is finite, you should take care in adjusting the SharedSection parameter.If the ArcSDE service starts as a domain account, the gsrvr.exe allocates from the desktop a noninteractive desktop heap of 512 KB, created for the ArcSDE service. If the ArcSDE service starts as the LocalSystem account, the gsrvr.exe allocates from the shared desktop a noninteractive desktop heap of 512 KB, pertaining to all services running as LocalSystem. If the ArcSDE service starts as the LocalSystem account with Allow service to interact with the desktop enabled, gsrvr.exe allocates from the default desktop an interactive desktop heap of 3 MB. SolutionsThe solution depends on the ArcSDE platform.There is currently no method to determine how much memory is in use by a single desktop or how much is left within the global pool. Use oh.exe, a Windows resource kit tool, to monitor how many desktops are in use.

  • Change the SharedSection parameter in the registry.

    Estimate the number of gsrvrs needed. For clients like ArcView, ArcEditor, and ArcExplorer, a gsrvr is usually a single ArcSDE connection. For ArcIMS, the number of gsrvrs depends on the service type. Assuming two virtual server threads per spatial server machine CPU, follow this general formula:

    (2 image service threads * total CPUs) + (number of query server threads)

    For example: (2 * 8) + 8 = 24 gsrvrs This recommendation also assumes the use of an image and query service, and all AXLs connect as the same user and database to the ArcSDE server. If different databases or users are referenced in your AXLs, then the formula becomes:

    (#databases * #mapservice threads)+ (#dbs*#queryservices)

    If the number of gsrvrs exceeds 64—the default CONNECTIONS setting in the ArcSDE SERVER_CONFIG table—consider either changing some of these connections to direct connections or boost the third SharedSection parameter in the registry. Since a direct connection runs as a thread within the client process, no desktop is allocated to it on the ArcSDE server.WARNING: The instructions in this step include making changes to essential parts of your operating system. It is recommended that you back up your operating system and files, including the registry, before proceeding. Consult with a qualified computer systems professional, if necessary. ESRI cannot guarantee results from incorrect modifications while following these instructions. Therefore, use caution and proceed at your own risk.This parameter is altered using the Windows registry editor.To change the SharedSection, at the MS_DOS prompt, type regedit in the input line and click OK. Navigate to the following registry path and double-click the Windows registry key.

    \\HKEY_LOCAL_MACHINE\ SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems

    This string contains startup parameters for Windows. Within the string, you will find the SharedSection parameter.The default value is 1024,3072,512. The third argument is the maximum amount of heap memory allocated to noninteractive desktops. By default, it is set to 512 kilobytes or one megabyte. At this setting, an ArcSDE service will accept approximately 56 connections. Increasing the maximum heap size of noninteractive desktops to two megabytes allows the ArcSDE service to accept up to 270 connections. The SharedSection value would be 1024,3072,2048 when the noninteractive heap size is set to two megabytes.Increase the third value of the SharedSection parameter from 512 to 1024. This doubles the amount of gsrvrs the ArcSDE service can spawn.

    SharedSection=1024,3076,1024

    The server must be rebooted once SharedSection is changed.

  • SQL Server sites using Windows authentication for their ArcSDE connections

    Users connecting to ArcSDE services with Windows authentication may experience different connection behavior. When the ArcSDE service spawns a gsrvr from a Windows authenticated connection, that gsrvr does not allocate from the same desktop as the server process, the giomgr.exe, but receives its own desktop (noninteractive) of 512 KB. Multiple Windows authenticated connections from the same machine allocate from the same desktop, but connections from different machines allocate from new desktops. If operating exclusively in Windows authentication mode, more ArcSDE connections may be serviced by reducing the size of both the interactive and noninteractive desktop heap instead of boosting it as mentioned above. This connection behavior will change in future releases of ArcSDE for SQL Server.

  • Sites that have terminal services or Citrix deployed on their ArcSDE server

    Terminal services reduce the desktop global memory pool from 48 MB to 20 MB. This means that fewer total desktops are created. If operating in this environment, the third parameter of SharedSection can be increased, but use caution with the number of different non-LocalSystem services created. Remember, whenever a non-LocalSystem service starts, it allocates memory from the global memory pool to a new desktop. It is not recommended that SQL Server sites use terminal services with Windows authenticated ArcSDE connections. If this configuration must be run and difficulties are experienced supporting enough ArcSDE connections, allowing the ArcSDE service to interact with the desktop may solve the problem. In this case, do not change any of the SharedSection parameters.

• SERVER_CONFIG CONNECTIONS parameter

  If you repeatedly reach the maximum number of concurrent connections allowed by the CONNECTIONS initialization parameter set in the SERVER_CONFIG system table (SDE_SERVER_CONFIG in SQL Server and PostgreSQL databases), you need to increase the value of this parameter. The default value is 48. See The giomgr defs file and the SERVER_CONFIG system table for more information.

  Note:

  Beginning with ArcSDE 9, direct connections count toward the number of connections as regards the CONNECTIONS parameter. In other words, the number of both direct connections and ArcSDE service connections cannot exceed the value of the CONNECTIONS parameter.

  You can alter the CONNECTIONS initialization parameter in the SERVER_CONFIG table using the sdeconfig administration command.
  1. To see what the current CONNECTIONS parameter value is, use the sdeconfig –o list command at the command prompt. The syntax for this command is

     sdeconfig –o list [–p <SDE_admin_password>]

     This will return all the initialization parameters in the SERVER_CONFIG table and their values.
  2. Change the value of the CONNECTIONS parameter with the sdeconfig –o alter command.

     sdeconfig –o alter –v <Property_Name=Value,...> [–i <service]
     [–s <server_name>] [–D <database>] –u <SDE_admin_user>
     [–p <SDE_admin_password>] [–N] [–q]

     In this case, the Property_Name would be CONNECTIONS, and the Value would be the number to which you want to increase the number of allowed concurrent connections.For details on using the sdeconfig command, consult the Administration Command Reference provided with the ArcSDE component of ArcGIS Server Enterprise.