Previous Topic: Best Practices - How to Recover a Stand-alone Server from a Disaster Using the Disaster Recovery OptionNext Topic: How to Restore Data from Cloud Storage


Best Practices - How to Recover a Arcserve Backup Server from a Disaster without Using the Disaster Recovery Option

Arcserve Backup lets you perform a full disaster recovery of a Arcserve Backup server without installing the Disaster Recovery Option. To enable this capability, you must complete the steps described in Recover the Arcserve Backup server described below. If the Arcserve Backup server that you are recovering is a primary or stand-alone server, you must also complete the steps in Recover the Arcserve Backup Database below.

The procedure consists of the following tasks:

  1. Perform a full backup and restore of the Arcserve Backup server.
  2. Recover the Arcserve Backup server.
  3. Recover the Arcserve Backup database.
  4. Reactivate the existing Arcserve Backup database (optional).
  5. Recover the Job Queue Session.

    Important! You must restore the Job Queue to the Arcserve primary or stand-alone server. Do not restore the Job Queue to a member server of any Arcserve domain.

  6. Recover the Active Directory.

    Note: This task applies only to Arcserve Backup servers that function as a domain controller.

  7. Confirm Arcserve Backup licenses, if needed.

Be aware of the following behavior:

To recover the Arcserve Backup server

Important! You must have performed at least one full backup of the Arcserve Backup server before you can recover the Arcserve Backup server.

  1. Reinstall the operating system on the Arcserve Backup server.

    Verify that the hard disk partitions, hardware, and the operating system (version, edition, and service pack) configurations are identical to the configurations that were backed up.

  2. Reinstall Arcserve Backup, agents, and options into the same directories as the original installation.
  3. After you install Arcserve Backup, open the Merge utility and merge the media used for the last full backup.
  4. After the merge is complete, open the Restore Manager, and verify that the Restore files to their original location(s) option is selected.

    Locate the full backup sessions.

    Select the backup sessions for the machine, excluding all the following Arcserve Backup-specific sessions:

  5. Click Options on the toolbar.

    The Restore Manager, Options dialog opens.

    Click the Operations tab, click Restore Registry Files and Event logs, and click OK.

    The Options dialog closes.

  6. Click Submit on the toolbar to submit the restore job.

    The Session User Name and Password dialog opens.

  7. On the Session User Name and Password dialog, complete the fields that follow, as required, and then click OK.

    On the Session User Name and Password dialog, click Edit to modify the User Name, Password, and IP address for the selected session.

    The Enter User Name and Password dialog opens.

  8. On the Enter User Name and Password dialog, specify the User Name and Password for the Arcserve Backup server, click the Apply [User Name and Password] to All Rows check box to apply the user name and passwords specified to all sessions.

    Note: When you are editing IP addresses and passwords, you must edit the individual IP address and session password for each individual session.

    Click OK.

    The Enter User Name and Password dialog closes.

  9. Click OK to close the Session User Name and Password dialog.

    Note: After you click OK, a message box labeled Arcserve Backup may open and prompt you to specify the IP addresses for the sessions that require authentication to submit the restore job. If the Arcserve Backup dialog opens, you must specify all the IP address for all sessions to submit the job, and then click OK.

    The restore job is submitted.

  10. When the restore job is complete, restart the computer.

    Be aware of the following behavior:

  11. After the computer restarts, do one of the following:
  12. Right-click the Data folder and select Properties from the pop-up menu.

    The Properties dialog opens.

  13. Click the Security tab and then click Advanced.

    The Advanced Security Settings for Data opens.

    Note: On Windows Server 2008 systems, click Edit on the Data Security Detailed Settings screen.

  14. Click the Permissions tab and select the Replace permission entries on all child objects shown here that apply to child objects option and click OK.

    Note: On Windows Server 2008 systems, click the Replace the existing inheritable permissions on all descendants with inheritable permissions from this object option.

    If the Arcserve Backup Database is configured such that the database data files are stored in a different directory, repeat Steps 12, 13, and 14 on this folder to change its security attributes.

  15. Open the Windows Services Manager and start the SQL Server service (ARCSERVE_DB).
  16. Do one of the following:
  17. Open the Windows Computer Management Console, click Local Users, and then click Groups.

    The following group name should appear:

    SQLServer2008MSSQLUser$MACHINENAME$ARCSERVE_DB
    

    Note: The value of MACHINENAME should be the name of your computer.

  18. Record this Group name.

    Return to the Data folder (see Step 12).

    Right-click the Data folder and select Properties from the pop-up menu.

    The Properties dialog opens.

  19. Click the Security tab and then click Add.

    The Select Users, Computer, and Groups dialog opens.

  20. Click Locations and then click Local Machine.

    Add the Group that you recorded in step 14 and click OK.

    The Select Users, Computer, and Groups dialog closes.

  21. Click the Advanced tab and then click the Permissions tab.

    Select the Replace permission entries on all child objects shown here that apply to child objects option and click OK.

    Note: If the Arcserve Backup database is configured such that the database data files are stored in a different directory, repeat Steps 12 to 21 on this folder to change its security attributes.

  22. Open the Windows Services Manager and start the SQL Server service (ARCSERVE_DB).

To recover the Arcserve Backup database

Important! Arcserve Backup will not be available until you recover the database. Error messages may appear in the Arcserve Backup Activity Log that you can ignore.

  1. Open the Windows Services Manager and start the Arcserve Database Engine service.
  2. Open the Restore Manager.

    From the restore methods drop-down list, select Restore by Session.

    Locate and select the Arcserve Backup Database session as the restore source.

    Click the Destination tab and verify that the Restore files to their original location(s) option is selected.

    Note: If the Arcserve Backup database you are recovering is stored in an independent local SQL Server instance, you should select the "master" database to restore before restoring the Arcserve Backup "asdb."

  3. Open the Restore Options by clicking Options on the toolbar.

    Click the Operation tab, select the Disable Database Recording option, and click OK.

    The Options dialog closes.

  4. Right-click the Arcserve Database session and select Agent Option from the pop-up menu.
  5. Click the Restore Options tab, select the Force Restore over existing option, and click OK.

    Note: If you do not choose this option, the restore job may fail, and the Database Engine will not start. For troubleshooting assistance, see the steps under the section To Reactivate the existing Arcserve Backup database.

  6. Click Submit on the toolbar to submit the restore job.

    Note: After you click Submit on the toolbar to submit the restore job, you must specify a user name and password on the DBAgent tab on the Session User Name and Password dialog.

    During the restore process, the Database Engine service may pause or stop and the Manager Console may respond slowly. Because the Database Engine is unavailable during the recovery process, clients cannot connect to it. Therefore, Error E1516 [Staging] may be recorded in the Activity Log: "Cannot inquiry the database (Error=4294967293)." This behavior is normal during the database recovery process.

    After the restore job completes successfully, the Database Engine will automatically resume and Arcserve Backup will return to normal operations.

To reactivate the existing Arcserve Backup database

This is an optional task. If the restore job fails because the correct options were not selected in the previous steps, the database may have been in an offline state while the restore job was in progress. As a result, the Database Engine could not access the Arcserve Backup database during the restore. The following steps describe how to reactivate the Arcserve Backup database.

  1. Browse to the Arcserve Backup Home directory and locate asdbe_start.bat.
  2. Execute asdbe_start.bat.

    Note: The script uses a Microsoft SQL CLI utility "sqlcmd" to run a series of commands that will bring the Arcserve Backup database online.

    After you execute the script, the Database Engine service resumes.

  3. Repeat the steps in the section To recover the Arcserve Backup database.

    Note: Verify that you specify the Disable Database Recording and Force Restore over existing options before you start the job.

To Recover the Job Queue Session

  1. After the Arcserve Backup server starts, open the Restore Manager, locate, and select the Job Queue session.

    Note: When you select this session, Arcserve Backup requires a merge job of the Job Queue session.

    Click Yes to continue the recovery of the Job Queue session.

  2. From the Destination tab, specify an alternate location to restore the Job Queue session.
  3. Click Submit on the toolbar to submit the job to restore the Job Queue session to an alternate location.

    Note: Verify that the alternate location is an empty directory.

  4. After the Job Queue session is restored to the alternate location, open the Server Admin and do the following:
    1. Locate the Arcserve Backup primary server or stand-alone server.
    2. Right-click the Arcserve Backup server and select Stop all services from the pop-up menu.

    All Arcserve Backup services stop.

  5. Access the alternate location and copy all Job Queue files under the folder that you restored to the following directory: ARCSERVE_HOME\00000001.qsd
  6. From the Server Admin, restart all Arcserve Backup services by doing the following:
    1. Locate the Arcserve Backup primary server or stand-alone server.
    2. Right-click the Arcserve Backup server and select Start all services from the pop-up menu.

    All Arcserve Backup services start.

    Note: The status of the backup job that you used for restoration is in a "crashed" state. When the Job Queue was backed up, that job was in an active state, but the corresponding process was not running. Therefore, that job is now in a crashed state and error message E1311 is written to the Activity Log: Job has crashed. This behavior is normal while the Job Queue is being restored.

  7. If the Arcserve Backup is not a domain controller, go to the section, Confirm Arcserve Backup product licenses. Otherwise, continue to Recover the Active Directory.

Recover the Active Directory

  1. Restart the system.

    After the system restarts, press F8.

    The Advanced Option Menu appears.

  2. Select Directory Service Restore Mode and start the system in the Restore mode.

    Start Arcserve Backup.

    Error message E3073 occurs:

    Unable to logon as user, user =Administrator,EC=Logon Failure or W3073 Unable to logon as user, user =Administrator,EC=Logon Failure
    
  3. Open the Restore Manager and select the Source tab.

    From the Restore methods drop-down list, select Restore by Session.

    Locate and select the System State session.

    Perform one of the following options:

    For Windows Server 2003 systems:

    For Windows Server 2008 or later systems:

  4. On the System State Restore Options dialog, click Make the Restored Copy of the Active Directory Authoritative and click OK.
  5. Click Options on the Toolbar.

    The Global Options dialog opens.

  6. Click the Operation tab.

    Click Restore Registry Files and Event logs and click OK.

    The Global Options are applied.

  7. Perform one of the following options:

    After the restore job is complete, restart the system.

Confirm Arcserve Backup product licenses

You should confirm the product license after full restore is complete. The current Arcserve Backup license is restored back to the original state when you perform a full backup. If you have applied new licenses after a full backup, or the licenses were dynamically assigned to other servers, it is possible to encounter license errors. You should register or adjust product licenses accordingly.

Error Messages

After you recover the Arcserve Backup server, you may discover errors, warnings, and failure audits in the system Event Log similar to the error messages listed below, depending on how your system is configured. Such messages are caused by the intermediate state of recovery, or are related to the startup order in which Arcserve Backup and the SQL Server services started.

Arcserve Backup may report the following errors during the intermediate state of recovery.

Error 8355

This error message is reported when a "service broker" disabled setting is detected in the MSDB that was recovered. You can safely ignore this error because this is the default behavior of a system database recovery that is limited to SQL Server 2008 Express Edition. SQL Server 2000 and SQL Server 2005 are not affected by this behavior.

You can suppress this error by doing the following:

  1. Open a Windows Command Line window.
  2. Connect to the Arcserve Backup database (ARCSERVE_DB) by executing the following sqlcmd:
    SQLcmd -S <machine name>\<Instance name>
    

    For example:

    C:\Users\Administrator>sqlcmd -S localhost\ARCSERVE_DB
    
  3. Confirm the value of service_broker on msdb is 0:
    select name,is_broker_enabled from sys.databases
    go
    
  4. Execute the command using the following arguments:
    alter database msdb set enable_broker
    go
    
  5. Confirm the value of service_broker on msdb is 1:
    select name,is_broker_enabled from sys.databases
    go
    Quit
    

Close the Command Line window.

Arcserve Backup corrects the following error conditions after the recovery process is complete:

Error 615

This error message is reported when the 'master' database is restored during recovery without using the Arcserve Backup Disaster Recovery Option and SQL Server is hosting a single Arcserve Backup database. This error may not occur when SQL Server 2008 Express Edition hosts the Arcserve Backup database.

The Arcserve Backup Agent for Microsoft SQL Server will not back up the tempdb database even when a full instance is selected. Tempdb is also excluded by the Arcserve Backup server and file system agent when the normal file system is backed up. However, tempdb is recorded as an existing database in the SQL Server master database, so when the master is restored, the SQL Server service reports that it cannot find tempdb.

Error 15466

This error message is reported when the system state is restored during recovery without using Arcserve Backup Disaster Recovery Option. Either SQL Server 2008 Express Edition or SQL Server can host the Arcserve Backup database.

When you recover the Arcserve Backup server without using the Disaster Recovery Option, the Windows operating system and SQL Server are reinstalled, and the Service Master Key (SMK) of SQL Server is created. The Service Master Key (SMK) is used to encrypt all database master keys and all server-level secrets such as credential secrets or linked server login passwords.

They key is a 128-bit 3DES key. The SMK is encrypted using DPAPI and the service account credentials. When the system state is restored but SQL Server sessions have not yet been restored, the system state is overwritten by the restore operation. However, the SQL Server instance is not yet overwritten. The SMK is in the system state, so it has been recovered to the old one, which is therefore inconsistent with the SQL Server instance. At this time, the recovery procedure requires that you restart the operating system.

During the restart, SQL Server reads the SMK and checks it against the SQL database. Because the SMK and SQL Server database are inconsistent, the error occurs.

Error 17113

This error message is reported when the user permission settings for either the file or its containing folders are incorrect. Using the procedures described in this topic, you will adjust these permissions. The error will be corrected after you adjust the permissions.

Errors not related to the recovery process

If the Arcserve Backup services and SQL Server services do not start in the proper sequences, SQL Server error messages may appear in the system log. This behavior is a known issue. For more information, see the readme file.