Storage Agent disabled with error ANR0110E due to Time Differences

A TSM storage agent can become disabled if there is greater than 60 minutes of difference in time between the TSM storage agent machine and TSM server machine. When an administrator session initiates with the storage agent, this time difference will cause the storage agent to disable itself.

When routing an administration command to the storage agent, such as QUERY SESSION or QUERY MOUNT, the ANR0110E error message is witnessed in the server activity log. After changing the time difference between the TSM storage agent and TSM server to less than 60 minutes, and then issuing ACCEPT DATE on the TSM server, the storage agent disables itself. This can be observed by issuing the QUERY STATUS command on the storage agent console and checking for the availability parameter value, which should say ‘disabled.’

Since the storage agent is unsure about the time, it has disabled itself. Any subsequent storage agent requests will fail. Recycling the storage agent will cause it to become available for use again.

For the PDF version of this document, send a blank email, with subject line “Storage Agent disabled with error ANR0110E due to Time Differences”, to TSM Assist

System date/time change causes ANR0110E unexpected date sessions disabled

Every time the server is started and for each hour thereafter, a date and time check occurs. An invalid date can be one of the following:

• Earlier than the server installation date and time.
• More than one hour earlier than the last time the date was check
• More than 30 days later than the last time the date was checked.

As a preventive measure, when TSM detects an invalid date or time, server sessions become disabled so that the TSM server does not perform undesired actions based on the new system date/time. For example, expiration processing uses the system date/time to determine when to purge files from server storage.

An error message (ANR0110E) is displayed and expiration, migration, reclamation, and volume history deletion operations are not allowed. You may either change the system date if it is in error, or issue the ACCEPT DATE command to force the server to accept the current system date as valid. Use the ENABLE SESSIONS command after you issue the ACCEPT DATE command to re-enable the server for client node activity.

In order to correct this problem, perform the following commands on the TSM server:

• Run the QUERY STATUS command and check for the availability parameter value, which should say ‘disabled’
• Run the ACCEPT DATE command, which tells the TSM Server that you are aware of and accept the date/time change.
• Run the ENABLE SESSIONS command, which re-enables the TSM Server again for normal operations.
• Re-run the QUERY STATUS command and check for the availability parameter value, which should say ‘enabled’

For the PDF version of this document, send a blank email, with subject line “System date/time change causes ANR0110E unexpected date sessions disabled”, to TSM Assist

Creating Multiple TSM Instances in a UNIX Environment

Note:
This exercise assumes that the platform is SuSE Linux Enterprise Server.
This will work well on other UNIX environments – Solaris/HP/AIX.
For AIX, change all references from /opt/tivoli to /usr/tivoli.

To run multiple Tivoli Storage Manager servers on the same machine, you must create unique port numbers TCPPORT, HTTPPORT, SHMPORT and TCPADMINPORT and maintain database and recovery log files that are separate to each TSM server instance. Each server instance needs its own ports and its own database and recovery log.

Take note that you need a separate license for each unique instance of the server in order to run additional servers.

When running multiple servers, note that running a server from a directory other than the installation directory is only supported by setting the DSMSERV_DIR environment variable. This is because the base install directory is used for more than just finding the executables; e.g. the license and message file. To run a server from a directory other than the installation directory, you set the DSMSERV_DIR environment variable to point to the installation directory and add /opt/tivoli/tsm/server/bin to your path.

This exercise assumes that both servers will be using the same IP address and clients will be connecting to both servers. In order to set this up, two unique TCPPORTS and HTTPPORTS must be used in the dsmserv.opt. Otherwise the ports will not be open for connection when both servers are running, because the server that starts up first claims the port and the port will be bound to that server.

The TSM server software code is installed in /opt/tivoli/tsm/server/bin. The same executable, but different TSM Disk Definition – dsmserv.dsk, TSM Server Options – dsmserv.opt, Volume History and Device Configuration files will be used in each instance. This exercise assumes that the user will be keeping a set of these files for one instance in /opt/tivoli/tsm/server_A and in /opt/tivoli/tsm/server_B for the other. However, any filesystem can be used.

From here on, assume that server_A is the default installed TSM environment and that server_b is to be created.

The following steps can then be implemented:

For server_A:

1. After halting server_A, change to the server_A instance directory.

cd /opt/tivoli/tsm/server_A

2. Edit’s its dsmserv.opt file (the bare minimum setup below):

commmethod tcpip
tcpport 1500
commmethod http
httpport 1520
devconfig devcfg_serverA.out
volhist volhist_serverA.out

3. server_A can be started up by using

3.1 The –o parameter of the dsmserv command to specify an options file name.
3.2 The following command if your shell is in the csh family:

setenv DSMSERV_DIR /opt/tivoli/tsm/server/bin
setenv DSMSERV_CONFIG=/opt/tivoli/tsm/serverA /dsmserv.opt
nohup /opt/tivoli/tsm/server/bin/dsmserv quiet &

3.3 The following command if your shell is in the ksh family:

export DSMSERV_DIR=/opt/tivoli/tsm/server/bin
export DSMSERV_CONFIG=/opt/tivoli/tsm/serverA/dsmserv.opt
nohup /opt/tivoli/tsm/server/bin/dsmserv quiet &

For server_b:

1. Change to the server_B instance directory.

cd /opt/tivoli/tsm/server_B

2. Edit’s its dsmserv.opt file (the bare minimum setup below):

commmethod tcpip
tcpport 1501
commmethod http
httpport 1521
devconfig devcfg_serverB.out
volhist volhist_serverB.out

3. Format the database and recovery logs files. For example, to format two 10 GB database volume and a 1 GB recovery log volume, enter:

dsmfmt –g –db server_B_dbvol1.dsm 10 server_B_dbvol2.dsm 10
dsmfmt –g –log server_B_logvol1.log 1

4. Initialize the database and recovery log in the new server directory:

dsmserv format 1 server_B_logvol1.log 2 server_B_dbvol1.dsm server_B_dbvol2.dsm

5. When the server is started, it searches the current directory for the existence of the dsmserv.dsk file. If the file is found, the names of the recovery log and database files are used for server operation. If the dsmserv.dsk file is not found in the current directory, Tivoli Storage Manager issues an error message (ANR0212E) and server initialization stops.

6. Create server_B’s dsmserv.dsk file with contents as follows:

/opt/tivoli/tsm/server_B/server_B_logvol1.log
/opt/tivoli/tsm/server_B/server_B_dbvol1.dsm
/opt/tivoli/tsm/server_B/server_B_dbvol2.dsm

7. The TSM lock management process creates the adsmserv.lock file in the same directory as the dsmserv command. If the adsmserv.lock file is unable to be created, locked or written to, the message indicates the server is already running. To allow for the creation of a second lock file, create a soft link in server_B’s instance directory to the dsmserv command.

ln –s /opt/tivoli/tsm/server/bin/dsmserv /opt/tivoli/tsm/server_B/dsmserv

8. server_B can be started up by using

8.1 The –o parameter of the dsmserv command to specify an options file name.

8.2 The following command if your shell is in the csh family:

setenv DSMSERV_DIR /opt/tivoli/tsm/server/bin
setenv DSMSERV_CONFIG=/opt/tivoli/tsm/serverB/dsmserv.opt
nohup /opt/tivoli/tsm/server/bin/dsmserv quiet &

8.3 The following command if your shell is in the ksh family:

export DSMSERV_DIR=/opt/tivoli/tsm/server/bin
export DSMSERV_CONFIG=/opt/tivoli/tsm/serverB/dsmserv.opt
nohup /opt/tivoli/tsm/server/bin/dsmserv quiet &

9. You will also want to use the SET SERVERNAME to give each server instance a unique name in order to be able to manage these multiple TSM servers from the same ISC/Admin Center interface at 5.3 – if the multiple servers all have the same servername, then you will not be able to create a unique Admin Center connection to each of these servers while they are running on the same system. Furthermore, you may also want to set each servers internal name to something other than the defaults (the system name) for to make it easier to differentiate which one you are working on. This must be done before any device paths or server to server communication is configured.

For the PDF version of this document, send a blank email, with subject line “Creating Multiple TSM Instances in a UNIX Environment”, to TSM Assist

ANR2400E During del volhist, when volume is of type REMOTE

ANR2400E is reported when running del volhist for volume of type=remote. This volume has already been defined in a storage pool or has been used previously to store export, database dump, or database backup information; or this volume may be owned by a library client or another TSM server.

TSM is not able to delete these volumes using the del volhist command. The volumes will never return to scratch.

This can be fixed by issuing the following command:

del volh tod=today type=remote volume=<VOLUME_NAME> force=y

This should only be used if a REMOTE volume no longer has valid data on it. Once executed, any server could use the volume to overwrite any existing data. Use caution whenever you delete any volume history entry.

For the PDF version of this document, send a blank email, with subject line “ANR2400E During del volhist, when volume is of type REMOTE”, to TSM Assist

TDP for MSSQL receives error ACO5716W, rc=406

When trying to run a Data Protection for SQL command, the following message is received: ACO5716W An error was encountered with Tivoli Storage Manager API initialization, rc = 406.

For example:

C:\Program Files\Tivoli\TSM\TDPSql>tdpsqlc backup * full
IBM Tivoli Storage Manager for Databases:
Data Protection for Microsoft SQL Server Version 5, Release 1, Level 5.0
(C) Copyright IBM Corporation 1997, 2002. All rights reserved.
ACO5716W An error was encountered with Tivoli Storage Manager API initialization, rc = 406.

Or, from within the TDP for SQL GUI:

This error can occur when the base-level TSM BA client is upgraded to a newer version. In certain situations, the TSM API runtime files get out of sync. This issue was solved by (un)installing all the TSM components in this order:
1. Uninstall Data Protection for SQL.
2. Uninstall the base-level TSM BA client.
3. Reinstall the base-level TSM BA client.
4. Reinstall Data Protection for SQL.
5. Retry the operation.

For the PDF version of this document, send a blank email, with subject line “TDP for MSSQL receives error ACO5716W, return code 406”, to TSM Assist

TDP for MSSQL receives error ACO5716W, rc=610

When trying to run a Data Protection for SQL command, the following message is received: ACO5716W An error was encountered with Tivoli Storage Manager API initialization, rc = 610.

For example:

C:\Program Files\Tivoli\TSM\TDPSql>tdpsqlc backup * full
IBM Tivoli Storage Manager for Databases:
Data Protection for Microsoft SQL Server Version 5, Release 1, Level 5.0
(C) Copyright IBM Corporation 1997, 2002. All rights reserved.
ACO5716W An error was encountered with Tivoli Storage Manager API initialization, rc = 610.

The TDP for SQL GUI will fail to open with the above error.

The return code of 610 means “DSM_RC_NLS_OPEN_TXT”, a failure to open a message repository. In this case, a file is missing from the API and TDP for SQL directories called dscenu.txt – in some instances, TDP for SQL will be missing this file when installed.
The file “dscenu.txt” will also be found in the baclient directory (by default, c:\Program Files\Tivoli\TSM\Baclient) – copying this file into the API directory (by default, c:\Program Files\Tivoli\TSM\API) and into the TDP for SQL directory (by default, c:\Program Files\Tivoli\TSM\TDPSql) will allow the TDP for SQL GUI to see the message repository and open properly.

For the PDF version of this document, send a blank email, with subject line “TDP for MSSQL receives error ACO5716W, return code 610”, to TSM Assist

TDP for MSSQL receives error ACO0004E

When trying to run a Data Protection for SQL command, the following message is received: ACO0004E An unknown error has been detected.

For example:

C:\Program Files\Tivoli\TSM\TDPSql>tdpsqlc backup * full
IBM Tivoli Storage Manager for Databases:
Data Protection for Microsoft SQL Server Version 5, Release 1, Level 5.0
(C) Copyright IBM Corporation 1997, 2002. All rights reserved.
ACO0004E An unknown error has been detected.

Or, from within the TDP for SQL GUI:

This error can occur when the base-level TSM BA client is upgraded to a newer version. In certain situations, the TSM API runtime files get out of sync. This issue was solved by (un)installing all the TSM components in this order:

1. Uninstall Data Protection for SQL.
2. Uninstall the base-level TSM BA client.
3. Reinstall the base-level TSM BA client.
4. Reinstall Data Protection for SQL.
5. Retry the operation.

For the PDF version of this document, send a blank email, with subject line “TDP for MSSQL receives error ACO0004E”, to TSM Assist

Enabling Backup/Archive Statistics and Logging

Enable the processing of all events to the TSM server activity log.

en ev actlog all node=*

Start logging events to the all receivers – and this includes the TSM server activity log.

beg ev all

Adjust the length of time that the activity log retains messages to avoid insufficient or outdated data. Here we set the activity log retention set to 180 days for management by date to 180 days

s actl 180 m=d

The SQL activity summary table contains statistics about each client session and server processes. These statistics are reflected in the daily report produced by Operational Reporting. Adjust the length of time that TSM retains these statistics. Here we set the server to retain the SQL activity summary table information for 60 days.

s sum 60

Using the undocumented server option CLIENTSUMMARYSTATISTICS OFF in the dsmserv.opt file will prevent clients from logging events to the summary table.

The event retention period for event records in the server database allows you to monitor completed schedules. An event record is created whenever processing of a scheduled command is started or missed. You can adjust the length of time that the server maintains event information to avoid insufficient or outdated data. Here we set the event retention period to 180 days.

s ev 180

Issue the QUERY STATUS command and verify the Activity Summary Retention Period, Event Record Retention Period and the Activity Summary Retention Period values.

For the PDF version of this document, send a blank email, with subject line “Enabling Backup/Archive Statistics and Logging”, to TSM Assist

Extracting a list of failed files from the Activity Log

Enable the processing of all events to the TSM server activity log.

en ev actlog all node=*

Start logging events to the all receivers – and this includes the TSM server activity log.

beg ev all

Adjust the length of time that the activity log retains messages to avoid insufficient or outdated data. Here we set the activity log retention set to 180 days for management by date.

s actl 180 m=d

To display all client originating failure messages, run:

select nodename, date(date_time) as DATE, time(date_time) as TIME, msgno, message from actlog where originator=’CLIENT’ and severity=’E’ order by 1,2

As of TSM Server 3.7, the DateFormat option in the Server Options file has been deprecated. The date format is now governed by the Locale in which the TSM server is running.

Note that TSM keeps only one version of an event record in the database. If a client schedule is changed, all previous event records for that schedule are removed from the database.

For the PDF version of this document, send a blank email, with subject line “Extracting a list of failed files from the Activity Log”, to TSM Assist

SCRATCH volumes become PRIVATE when checked in

After volumes were checked in the library as SCRATCH, TSM changed their status to PRIVATE.

Inspect the activity log for ANR8356E and ANR8778W.

q ac s=ANR8356E
q ac s=ANR8778W

If you have surpassed your activity log retention period, the SELECT statement below will identify these tapes.

select volume_name from libvolumes where status=’Private’ and last_use is Null and volume_name not in (select volume_name from volumes) and volume_name not in (select volume_name from volhistory where type in (‘BACKUPFULL’, ‘BACKUPINCR’, ‘DBSNAPSHOT’, ‘DBDUMP’))

This situation typically happens when volumes were previously labelled in a different TSM server; or when a tape what should have been checked in as SCRATCH is checked in as PRIVATE.

The resolution is to check in the volumes using the following command:

label libv <LIBR_NAME> labels=b search=b checkin=scr overwrite=y waitt=0

For the PDF version of this document, send a blank email, with subject line “SCRATCH volumes become PRIVATE when checked in”, to TSM Assist