The NBDB database transaction log (NBDB.log) may become too large or corrupt for proper NetBackup operation.

Problem

Problem occurs when NetBackup Relational Database Manager service failed to start. When running following command shows Database unavailable.

usr/openv/db/bin/nbdb_ping
Database [NBDB] is not available.

Error

There may be errors reported by the ASA database engine in the /usr/openv/db/log/server.log file as well. These errors would look similar to the following:Error: Cannot open transaction log file -- Can't use log file "/usr/openv/db/data/NBDB.log" since it is shorter than expected

Cause

Problem occurs when transaction log for the ASA database that runs under NetBackup to become corrupt or too large for proper operation. For example, in a file system that limits file size to 2GB, the transaction log will be truncated and corrupt once it reaches the 2Gb limit. The transaction log is truncated during online catalog backup, but it might be possible that catalog backups did not take place to prevent the log from growing too large.

The ASA database failing due to transaction log problems can include these errors:

- Catalog backups and regular backups can hang.

- Operator is unable to cancel queued jobs.

- The EMM database does not come back up after restarting NetBackup

Solution

Rebuild the transaction log via the following steps:-

1. Stop NetBackup if it is not already stopped:/usr/openv/netbackup/bin/goodies/netbackup stop

 2. Remove NBDB and BMRDB from auto_start option

/usr/openv/db/bin/nbdb_admin -auto_start NONE

3. Start the ASA database engine/usr/openv/db/bin/nbdbms_start_server

4. To add the necessary environment variables to your shell, run the command: /usr/openv/db/vxdbms_env.sh

Note 1: please make note of the period at the beginning of the line.
 

Note 2: if using 'csh' for a shell, the command is 'source /usr/openv/db/vxdbms_env.csh' without the quotes.

Note 3: depending on the root user's shell environment, this command could fail with a "Paremeter not set" or "unbound variable" error.  If one of these errors occurs, run the "set +u" command and then reattempt the ". /usr/openv/db/vxdbms_env.sh" command.
 

5. Change directories to where the NBDB resides:
 cd /usr/openv/db/data 

  

6. Remove or rename the bad transaction log:
 mv NBDB.log NBDB.log.bad

If BMR is used, also rename the BMR transaction log:

 mv BMRDB.log BMRDB.log.bad

 7. To force database recovery, while still in the /usr/openv/db/data directory, run the command (please note, the NBDB.log will not be created until step 10):

/usr/openv/db/bin/dbeng9 -f /usr/openv/db/data/NBDB.db

If BMR is used , force recovery of the BMR database:
/usr/openv/db/bin/dbeng9 -f /usr/openv/db/data/BMRDB.db

(Note: for NetBackup 7.0.1,  it is running on ASA11.  This command is :  /usr/openv/db/bin/dbeng11 -f NBDB  or  /usr/openv/db/bin/dbeng11 -f BMRDB) 

NOTE: If the dbeng11 -f NBDB should fail with the error "A database server with that name has already started", please check /usr/openv/tmp and move the sqlany directory to different name and all the sqlany files to different names and restart the database server on step 3 above and the dbeng11 -f NBDB should work.

At this point of recovery, no transaction log will be created just yet until the database is opened.

8. Stop the database by running the command: /usr/openv/db/bin/nbdbms_start_server -stop

9. Add NBDB and BMRDB to auto_start option (Add BMRDB only if using BMR)

/usr/openv/db/bin/nbdb_admin -auto_start NBDB

/usr/openv/db/bin/nbdb_admin -auto_start BMRDB

10. To start the database, run the command: 

/usr/openv/db/bin/nbdbms_start_server
 

11. To test if the database is operational, run the command:

/usr/openv/db/bin/nbdb_ping

which should return something similar to the following:
 

Database [NBDB] is alive and well on server [NB_master1].