Backups to OST using DD fails with Error 2106:Disk storage is down

Issue

After updating DataDomain plug-in (version 2.5.0.3) and running thetpconfig command to change the password on a NetBackup 7.5.0.4 media server configured for OpenStorage [OST(DD)] storage, all subsequent backups written to the DataDomain storage have failed with the error Disk storage is down (2106) reported.

Error

Error nbjm (pid=4340) NetBackup status 2106, EMM status Storage server is Down or unavailable.
Disk storage is down (2106)

Environment

NetBackup 7.5.x OpenStorage(DataDomain) Environment:

• Master server: NBU_Master - Windows 2008 R2 (x64) Enterprise server SP1 - running NetBackup 7.5.0.4 
• Media server: OST_DD_Media - Windows 2008 R2 (x64) Enterprise server SP1 - running NetBackup 7.5.0.4 
• Disk storage: DataDomain_SU01 | Disk Type: OST(DD) | Media Server: OST_DD_Media | Disk Pool: OST_DD_Pool01 
• Disk_Pool: OST_DD_Pool01  - Storage Server: OST_DD_Storage01 

Note: All server names listed above are generic names used only for the examples in this document

Cause

Troubleshooting:

{Checked the OST(DD) configuration on the upgraded Media server}:

Opened a command prompt to ..\NetBackup\bin\admincmd (Note: On UNIX/Linux servers, the path to these commands is /usr/openv/netbackup/bin/admincmd), and ran the following commands: 

> nbdevquery -liststs -stype DataDomain -U
- Checked the State : DOWN 

> nbdevquery -listdp -stype DataDomain -U
- Disk_Pool: OST_DD_Pool01 = UP - Storage Server: OST_DD_Storage01 = DOWN 

> nbdevquery -listdv -stype DataDomain -U
- OST_DD_Storage01 = DOWN | Disk Volume name: ddboost - Flag: InternalDown 

> bpstsinfo -pi
- Properly displayed the DataDomain Plug-in information 

Syntax to change the state of an OST(DD) disk volume:
> nbdevconfig -changestate -stype -dp - dv -state  
  > nbdevconfig -changestate -stype DataDomain -dp OST_DD_Pool01 -dv ddboost -state UP
- Command completed succesfully

> nbdevquery -liststs -stype DataDomain -U
- Checked the State: DOWN 

> nbdevquery -listdp -stype DataDomain -U
- Disk_Pool: OST_DD_Pool01 = UP - Storage Server: OST_DD_Storage01 = DOWN 

> nbdevquery -listdv -stype DataDomain -U
- OST_DD_Pool01 = DOWN | Disk Volume name: ddboost - Flag: InternalDown 

No Changes? 

> bpstsinfo -si - Properly displayed the DataDomain storage server information

Opened the NetBackup Administration Console > Credentials > Storage Servers.

- Highlighted Storage Server OST_DD_Storage01 in the top pane
- In the bottom pane BOTH the master and media servers are listed as media servers for the Storage Server

Cause:Both the media server AND the master server were specified as "media servers" under Credentials> Storage Servers; and the OST(DD) plug-in and the NetBackup Credentials for the Storage Server had only been updated on the media server, but not the master server, and the mis-match in the "plug-in" and the "Storage Server" credentials between the two "media servers" configured for the storage servers cause the OST(DD) Disk Pool and Storage Servers to go to a DOWN state.

Solution

Copied the OST(DD) 2.5.0.3 plug-in installation files to the NetBackup master server, and performed the following actions: 

1. Upgrade the OST(DD) plugin on the master to version 2.5.0.3 - to match the version on the media server
Cycle the NetBackup services. 

To stop and start the NetBackup services on a Windows master or media server:
Open a command prompt to \NetBackup\bin and run:

> bpdown -v -f

Then, start the services:

> bpup -v -f

Note: On a UNIX/Linux server these commands would be executed:
# /usr/openv/netbackup/bin/goodies/netbackup stop
# /usr/openv/netbackup/bin/goodies/netbackup start
 

2. Run the tpconfig -update command to change the OST(DD) credentials for the master server:

Open a command prompts to the \volmgr\bindirectory (UNIX/Linux: /usr/openv/volmgr/bin directory) and run:

> tpconfig -update -storage_server -stype DataDomain -sts_user_id -password

3. Run the following NetBackup command to confirm the password change on the master server allows the master server to access the OST(DD) storage server: 

Open a command prompts to \netbackup\bin\admincmd (UNIX/Linux server:/usr/openv/netbackup/bin/admincmd):

> bpstsinfo -si

Note: The bpstsinfo -si command (run from the master server) should display a similar output to what was displayed when this command was run on the media server. 

Wait a few minutes (10 to 20 minutes) before proceeding.

4.  Run the following command from the master server to check the current state of the OST(DD) storage servers, disk volumes, and disk pools.

Note: below are the states that should be seen after the above actions have been performed:

> nbdevquery -liststs -stype DataDomain -U 
- State: UP

> nbdevquery -listdp -stype DataDomain -U 
- Disk_Pool: OST_DD_Pool01 = UP - Storage Server: OST_DD_Storage01 =UP

> nbdevquery -listdv -stype DataDomain -U 
- OST_DD_Pool01 = UP | Disk Volume name: ddboost - Flag: InternalUP