Performing diagnostics on Linux/Solaris

The diagnostics tool provides valuable information on the state of ArcGIS Server. It is a tool to

• Gather the state of ArcGIS Server software.
• Identify problems locally and remotely.
• Provide possible causes and suggest solutions.

You can run this tool to determine whether ArcGIS Server was successfully configured for your system. This tool can be run in ArcGIS Server Manager or through the command line on your Linux/Solaris machine.

Run Diagnostics tool in ArcGIS Server Manager

In ArcGIS Server Manager, click the GIS Server > Diagnostics link on the left side of the page.

The tool lists the SOM machine and all the SOC machines. Select the machine that you would like to diagnose and click Run Diagnostics. See Interpret ArcGIS Server Diagnostics test results for the result.

Note:

The server diagnostics tool assumes that ArcGIS Server Monitor is running on the hosts you want to run the diagnostics tool against. To start the Monitor, execute the following script on each of those hosts: <ArcGIS Server Installation directory>/scripts/startmonitor. See How the diagnostics tool works: the Monitor framework to learn more about the Monitor Framework.

Running the diagnostics tool from the command line on your Linux/Solaris system

You can also run this diagnostics tool from the command line. This allows you to run the tool on local or remote machines (for example, SOC-only installation machines) and run it on a scheduled basis.

The script is installed under <ArcGIS Server Installation Directory>/scripts/Server_diag_tool. You need to run this script as the ArcGIS Server Install owner, the user who owns the ArcGIS Server installation directory. You can run it in three modes.

1. Interactive mode (default)

   % ./Server_diag_tool

   Runs diagnostic checks on ArcGIS Server hosts specified in the ArcGIS Server configuration file. This command provides a command line interface where you can select a particular host or a group of hosts on which to run diagnostic checks.
2. Silent Mode

   % ./Server_diag_tool --silent

   Runs the diagnostic tool only on the local host from a terminal console.
3. Custom Mode

   % ./Server_diag_tool --remote <HOSTNAME-1> <HOSTNAME-2>...<HOSTNAME-N>

   Runs diagnostic checks on ArcGIS Server hosts specified in the command line.

Interpret ArcGIS Server Diagnostics test results

The following is the information and explanation about all the checks performed by the diagnostics tool. Each diagnostics check is grouped with a code formatted DGXXX according to the part of ArcGIS Server it checks.

• DG001 - check /etc/hosts: has entry for localhosttest

  The /etc/hosts file should contain an entry for the localhost. If the entry does not exist, this check will fail.

• DG002 - check /etc/hosts: localhost = 127.0.0.1

  This verifies if the loopback IP address for localhost is set to 127.0.0.1. An example is as follows:

  127.0.0.1 localhost

• DG003 - check /etc/hosts: has entry for <hostname>

  This verifies if the /etc/hosts file has an entry for the ArcGIS Server host name.

• DG004 - check /etc/hosts: <hostname> != 127.0.0.1

  This will fail if the IP address for the hostname is set to the loopback address. It is required that the hostname entry in the /etc/hosts has a valid IP address.

  Example:

  10.14.912.5 gogo.esri.com gogo

• DG005 - This verifies if the /etc/passwd file contains an entry for agsadmin and agsuser. These OS users are created by the ArcGIS Server setup at the time of install, if they didn't exist before the install. Without these users, ArcGIS Server won't function properly.

  If your setup was successful and these two users were deleted later on, you can create these two users manually to check if ArcGIS Server functions. If you still have problems, there is likely other damage to ArcGIS Server. A reinstall is recommended.

  Example:

  DG005 - check /etc/passwd: user agsadmin

  DG005 - check /etc/passwd: user agsuser

• DG006 - check <ArcGIS Server Installation directory>: owner

  This verifies that the directory <ArcGIS Server Installation directory> is owned as the ArcGIS Server install owner. If this check fails, it is recommended that you reinstall ArcGIS Server.

• DG007 - check <ArcGIS Server Installation directory>: permissions

  The installation directory should have at least 711 privilege (drwx--x--x) for the ArcGIS Server owner. If this check fails, grant at least 711 permission to the folder.

• DG008 - check <ArcGIS Server Installation directory>: has space

  This verifies if the disk space usage for the ArcGIS Server installation directory is not 100%. If the disk space is all used, ArcGIS Server won't function properly. More disk space is needed.

• DG009 - check <ArcGIS Server Installation directory>/servercore/tmp/mw: exists

  The <ArcGIS Server Installation directory>/servercore/tmp/mw directory is used periodically to store temporary files. This check validates the presence of this directory. If this check fails, usually the ArcGIS Server installation folder is in a bad state. It is recommended that you reinstall ArcGIS Server.

• DG010 - check <ArcGIS Server Installation directory>/servercore/tmp/mw: is writable

  This verifies if the user that owns the ArcGIS Server install has write permissions to the directory <ArcGIS Server installation location>/server10.0/servercore/tmp/mw.

• DG011 - check <ArcGIS Server Installation directory>/servercore/tmp/mw: has space.

  This verifies that the disk space usage for <ArcGIS Server Installation directory>/servercore/tmp/mw is not 100%.

• DG012 - check <file or directory>:owner

  This verifies if the file or directory is owned as the ArcGIS Server install owner. If this type of check fails, run ServerConfig to fix it.

  Examples:

  DG012 - check /etc/remotesa: owner

  DG012 - check /etc/remotesa/remotesa.config: owner

• DG013 - check <file or directory>: permissions

  This verifies if the <file or directory> has the correct permissions.

  This type of check includes:

  DG013 - check /etc/remotesa: permissions

  This verifies if /etc/remotesa folder has 700 (drwx------) permission.

  DG013 - check /etc/remotesa/remotesa.config: permissions

  This verifies if /etc/remotesa/remotesa.config file has 700 (-rwx------) permission.

• DG014 - check server directories

  This check will fail if something is wrong with the Server.dat file (for example, if Server.dat is missing) and the diagnostic tool is unable to perform the DG015 checks.

• DG015 - check <path to Server Directory>

  The diagnostics check obtains the list of all server directories from <ArcGIS Server Installation directory>/server/system/Server.dat and validates the existence of each directory. If the check fails, you need to modify it to the correct server directory path in ArcGIS Server Manager.

  Examples:

  DG015 - check <ArcGIS Server Installation directory>/server/serverdir/arcgisjobs

  DG015 - check <ArcGIS Server Installation directory>/server/serverdir/arcgiscache

  DG015 - check <ArcGIS Server Installation directory>/server/serverdir/arcgisindex

• DG016 - check Registry for ESRI\ArcServer

  This verifies that the Registry used by ArcGIS Server contains the necessary entries for the location and type of installation. These entries are created by the setup program. If this check fails, you need to run ServerConfig.

• DG017 - check Registry for ESRI\CoreRuntime

  This verifies that the Registry used by ArcGIS Server core contains the necessary entries for the location and type of installation. These entries are created by the setup program. If this check fails, you need to run ServerConfig.

• DG018 - check Core Services: is running

  This verifies if the Core Services are running. The Core Services form the interop layer for ArcGIS Server, and their role is critical to the functioning of ArcGIS Server. If this check fails, you need to run ServerConfig.

• DG019 - check Core Services: RPC Port

  This verifies if the Core Services are using a fixed port 135 for Remote Procedure Calls (RPC). If this check fails, run ServerConfig.

• DG020 - check Core Services: Enterprise mode

  This verifies if the core service is running in Enterprise mode. If this check fails, you need to run ServerConfig.

• DG021 - check Core Services: Administrator

  This verifies if the core service's administrator is the ArcGIS Server install owner. If this check fails, run ServerConfig.

• DG022 - check Security: Type

  This verifies if the security authentication type for the Core Services is set correctly.

• DG023 - check Security: Port

  This verifies if the security process is running on port 3774. If this check fails, run ServerConfig.

• DG024 - check Security: Domain

  This verifies if the domain name used by the Core Service security mechanism is same as the SOM machine name. If this check fails, run ServerConfig.

• DG025 - check Security: DomainServer

  This verifies if the DomainServer name used by the Core Service security mechanism is same as the Domain name. Both Domain name and Domain server specify the host machine where the Local GIS User authentication service (for ArcGIS Server) is running. If this check fails, you need to run ServerConfig.

• DG026 - check Security: DomainServerSecondary

  This verifies if the secondary DomainServer name used by the Core Service security mechanism is set. This is for setting up SOM failover. If the failover is not set, ArcGIS Server will still function properly; the diagnostics tool just gives a [WARNING] message.

• DG027 - check process:<process name>

  This verifies if the process started by ArcGIS Server is running. ArcGIS Server won't function properly if the process is not running. If you encounter this type of failure, stop ArcGIS Server, then make sure there are no ArcGIS Server processes running. Restart ArcGIS Server. If the error still occurs, try running ServerConfig.

  This type of check includes:

  DG027 - check process: mwrpcss

  This verifies if the process "mwrpcss" which represents the "RPC Server" is running. The ArcGIS Server framework is built on the DCOM technology that supports distribution of COM objects across different machines and communication between them. The mwrpcss process provides this support on Linux/Solaris.

  DG027 - check process: regss

  This verifies if the process "regss" which represents the "Registry Service" is running. The Registry Service is responsible for managing the registry data used by Core Services.

  DG027 - check process: watchdog

  This verifies if the process "watchdog" is running. Watchdog is part of the Core Services and is responsible for keeping the "mwrpcss" and "regss" processes alive. If the mwrpcss and/or regss processes terminate unexpectedly, the watchdog will start them again immediately.

  DG027 - check process: remotesa

  This verifies if the Identification process (remotesa) is running. Identity Server provides authentication services for ArcGIS Server.

  DG027 - check process: ns-slapd

  This verifies if the Directory Server process (ns-slapd) is running. Directory Server is a repository for storing ArcGIS Server user accounts.

  DG027 - check process: arcsom.exe

  This verifies if the Server Object Manager (SOM) is running.

  DG027 - check process: java -jar arcgis_manager

  This verifies if ArcGIS Server Manager is running as the ArcGIS Server install owner.

  DG027 - check process: java/manager/service/tomcat/managerappserver

  This verifies if the Tomcat server which hosts the deployed Web applications and Web services is running as the ArcGIS Server install owner.

  DG027 - check process: Xvfb

  This verifies if the Xvfb process is running.

  DG027 - check process: java/manager/service/derby

  This verifies if the Java process required by ArcGIS Server Manager security is running.

• DG028 - check LDAP server: is listening

  This is an additional check for Directory Server to verify if it is listening on the port (MWR_LDAPPORT) as specified in the /etc/remotesa/remotesa.config file. This port is 62000 by default. If this check fails, stop ArcGIS Server. Make sure the specified port is not occupied by any other process. Restart ArcGIS Server.

  .

• DG029 - check permissions: arcsom.exe

  This verifies if the Server Object Manager executable (<ArcGIS Server Installation directory>/bin/arcsom.exe) has been given Read, Write and Set-UserID (setuid) access rights.

  If this check fails, run the following as the install owner:

  % chmod 755 arcsom.exe

  % chmod a+s arcsom.exe

• DG030 - check permissions: arcsom.init

  This verifies if the Server Object Manager executable (<ArcGIS Server Installation directory>/bin/arcsom.init) has been given Read, Write and Set-UserID (setuid) access rights.

  If this check fails, run the following as the install owner:

  % chmod 755 arcsom.init

• DG031 - check permissions: arcsoc.exe

  This verifies if the Server Object Container executable (<ArcGIS Server Installation directory>/bin/arcsoc.exe) has been given Read, Write and Set-UserID (setuid) access rights.

  If this check fails, run the following as the install owner:

  % chmod 755 arcsoc.exe

  % chmod a+s arcsoc.exe

• DG032 - check permissions: arcsoc.init

  This verifies if the Server Object Container executable (<ArcGIS Server Installation directory>/bin/arcsoc.init) has been given Read, Write and Set-UserID (setuid) access rights.

  To fix it if this check fails, as the install owner, run

  % chmod 755 arcsoc.init

• DG033 - check User Mappings

  ArcGIS Server maps all its Local GIS users to either of the two OS accounts - "agsadmin" or "agsuser". This step verifies that the ArcGIS Server user map file (<ArcGIS Server Installation directory>/servercore/.Server/.arcgis_usermap) has at least three entries. By default, this file should have entries for the ArcGIS Server users: "admin", "agsadmin" and "agsuser". To fix it, run ServerConfig.

• DG034 - check User Mappings permissions

  This verifies if the ArcGIS Server install owner has at least Read and Write access to the user map file in this location: <ArcGIS Server Installation directory>/servercore/.Server/.arcgis_usermap.

  If this check fails, switch to the <ArcGIS Server Installation directory>/servercore/.Server directory, and as the install owner, run:

  % chmod 700 .arcgis_usermap

• DG035 - check SOC hosts

  This check will show when something is wrong with the Server.dat file (or if Server.dat is missing) and the diagnostic tool is unable to perform the DG036 checks.

• DG036 - check SOC host <SOC host name> : is reachable

  The diagnostics tool obtains the list of all Server Machines (SOCs) from <ArcGIS Server Installation directory>/server/system/Server.dat and verifies if each Server Machine is reachable.

• DG037 - check SOC host <SOC host name> : is running

  The diagnostics tool obtains the list of all Server Machines (SOCs) from <ArcGIS Server Installation directory>/server/system/Server.dat and verifies if ArcGIS Server on each of those machines is alive and listening on Port 135.

• DG038 - check if port <port number> is listening

  This verifies if the ports listed are being used by ArcGIS Server. If these ports are not available for ArcGIS Server, ArcGIS Server won't run properly.

  If you get any fails in the check, stop ArcGIS Server, make sure the specified port is not occupied by any other process, then restart ArcGIS server.

  This type of check includes:

  DG038 - check if port 8399 is listening

  This verifies if the Tomcat server which hosts the deployed Web applications and Web services is listening on Port 8399.

  DG038 - check if port 8309 is listening

  This verifies if the AJP/1.3 Connector is listening on Port 8309.

  DG038 - check if port 8099 is listening

  This verifies if the Tomcat server which hosts ArcGIS Server Manager is listening on Port 8099.

  DG038 - check if port 8109 is listening

  This verifies if the AJP/1.3 Connector is listening on Port 8109.

  DG038 - check if port 8621 is listening

  This verifies if ArcGIS Server security is listening on Port 8621.

• DG039 - check LDAP server: password storage scheme

  This verifies if the LDAP server has the correct storage scheme. If this type of check fails, run ServerConfig.

• DG040 - check LDAP server: password validity

  This verifies if the password to access the LDAP server is valid. If this check fails, run the <ArcGIS Server Installation directory>/servercore/tools/ldap/setldappwd.sh script to fix it.

• DG041 - check if server configuration file (Server.dat) exists

  This verifies if the Server.dat file exists. This file should be in the <ArcGIS Server installtion directory>/server/system directory. With a SOM-only install, this file will be generated after you add a SOC machine to the system.

• DG042 - check if security store config file(security.xml) exists

  This verifies if the security.xml file (for ArcGIS Server security configuration) exists. This file is installed at <ArcGIS Server Installation directory>/java/manager/service/lib/security.xml

• DG043 - check /etc/hosts: IP address <IP address> is correct

  This verifies if the IP address for the hostname entry in the /etc/hosts file is correct.

• DG044 - check locale: Server locale matches environment locale

  This verifies if the locale in which the server is running matches the local environment of the operating system.

• DG045 - check if the length of <path to the jobs directory> exceeds 30 characters

  This check gives a [WARNING] if the <path to the jobs directory> exceeds 30 characters. You can define this path in ArcGIS Server Manager by clicking GIS Server > Server Directories. The default path is <ArcGIS Server installation directory>/server/serverdir/arcgisjobs. If this path exceeds 30 characters, you might run into problems using geoprocessing tools with grids and coverages. For detailed information, check Creating tasks for the UNIX/Linux environment.

• DG046 - For Solaris platform only - the release of the Solaris version must be more recent than Solaris 10, Release 08/07, Update 5.

  The check verifies if the version of Solaris OS is of the 10 release 08/07 and has applied Update 5 or beyond. This update or later ones contain Xorgs version of Xvfb which ArcGIS Server requires. Without it, ArcGIS Server will not function. This check is only for the Solaris platform. On the Linux platform, it returns [N/A].

• DG047 - For Solaris platform only - Xvfb is installed under /usr/X11/bin

  The Xorgs version of Xvfb by default is installed under /user/X11/bin. ArcGIS Server checks this location for Xvfb. If it is not present under this location, ArcGIS Server will not function. This check is only for the Solaris platform. On the Linux platform, it will return [N/A].

• DG048 - For Solaris platform only - check the value of the max-descriptors

  The value of the maximum open File-Descriptors is set by the ulimit on Solaris. If this value is lower than 4096, this check will give a warning because it could affect ArcGIS Server performance when services use ArcSDE connections. For a detailed explaination, check the ESRI Knowledge Base article 37267. This check is only for the Solaris platform. On the Linux platform, it returns [N/A].

• DG049 - For SOC only install - check the running of the authentication service on the SOM machine.

  If the authentication service is not running on the SOM machine that the SOC is pointing to, the SOC machine will not get authenticated and is not usable.

  To fix this issue:

  1. Ensure the ArcGIS Server services are running on the SOM machine. As the install owner, run:
     % ps -e | grep arc

  2. Verify that the SOC machine can access the SOM machine over the network. On the SOC machine, try to remotely log in to the SOM machine by running:
     % telnet <SOM host> 3774

How the diagnostics tool works: the Monitor framework

The diagnostics tool is supported by the Monitor framework. The Monitor is a stand-alone program that runs on all ArcGIS Server machines as a background process. It must be running for diagnostic tools to function. The scripts to start and stop the Monitor are installed under the <ArcGIS Server Installation directory>/scripts folder and are called startmonitor/stopmonitor.

Monitor can be started or stopped by running

% <ArcGIS Server Installation directory>/scripts/startmonitor

% <ArcGIS Server Installation directory>/scripts/stopmonitor

When ArcGIS Server is started by running the startserver script, it will check if the monitor is already running. If not, this script will start the monitor. Running the stopserver script will NOT stop monitor process.

By default, Monitor runs on port 2422. This port number and other properties are set in the file <ArcGIS Server Installation directory>/java/tools/monitor/monitor.xml. The following properties can be modified in this file:

• ListenerPort

  This property is used to specify the port number that the Monitor uses to listen for incoming requests. The accepted range of port numbers is from 1024 to 65535. By default, the Monitor uses port 2422.

• LogLevel

  This property is used to specify the level of information that the Monitor produces and records in its log file. The log file is created at <ArcGIS Server Installation directory>/java/tools/monitor/logs/monitor.log. This file will be overwritten every time the Monitor is started. The supported levels of log information are listed below in the increasing order of message importance:

  • FINE—Records verbose messages that include the request XML received and the response XML returned

  • INFO—Records messages related to normal operation of the Monitor, for example, a new client connection

  • WARNING—Records error messages that are not fatal

  • SEVERE—Records errors that require immediate attention

  If the Monitor is configured to record INFO level messages, all WARNING and SEVERE messages will also be recorded, and so on. By default, the Monitor is set at level INFO.

• SocketTimeout

  This property is used to specify the amount of time, in seconds, that the Monitor waits for a request from a client before timing out. The default value is 120 seconds.

• ProcessTimeout

  This property is used to specify the amount of time, in seconds, that the Monitor waits for a response from a local program, such as diagnostics, before timing out. The default value is 120 seconds. The time-out periods are set such that they don't need to be modified unless the network (in the case of SocketTimeout) or machines (in the case of ProcessTimeout) are extremely slow.