Creating and using emergency recovery media

Creating and using emergency recovery media

Emergency recovery media enable you to recover your system if your hard disk experiences an unrecoverable error. You should create emergency recovery media immediately after you install your system, and again each time you add or change system hardware and software.

The emergency recovery media consist of:

emergency recovery boot media (CD or floppy diskettes) created with emerg_disk(ADM)
emergency recovery backup tapes created with emerg_tape(ADM)
backup media, containing additional user and application data

The emergecny recovery boot media and tapes are usually created right after you install your system, and do not need to be re-created unless you change your system configuration. Additional backups should be taken periodically, so that user and application data can be preserved.

The following topics describe:

Keys to recovering your system

To ensure that you always have a set of emergency recovery media and data backups, we strongly recommend that you:

Create emergency recovery boot media and tapes immediately after installing and configuring your system.
If you install additional system or application software at a later time, create new emergency recovery boot media and tapes.
Back up your system on a regular basis (daily or weekly), using standard UNIX tools or a third-party backup solution.
The length between backups is correlated to the amount of possible data loss. If you back up your system each day, you can only lose up to one day's data. If you only back up your system once a month, up to a month of data might be lost if your hardware fails.

The emergency recovery boot media contains a small, bootable image from which you can boot your system.

The emergency recovery tapes contain an image of critical filesystems.

By first booting your system with the emergency boot media and then restoring it from the emergency tapes, you restore all the system software and filesystems. that existed when you created the boot media.

NOTE: If your system does not have a tape drive, you should still create emergency recovery boot media and then back up the entire system onto other media or across the network.

Performing regular system backups ensures that data added to your system after the creation of the emergency recovery media can also be restored. Data created after the most recent backup may be lost if your hard disk fails.

The system owner can back up the entire system on a regular basis. Individual users without owner privileges can perform backups of their own directories.

To ensure that you back up not only data but also any extended file attributes (such as privileges and ACLs), use cpio.

See the cpio manual page for more information.

Creating emergency recovery boot media

For instructions on creating an emergency recovery boot CD, see the emergency_disk(ADM) manual page. The following is an example of creating boot diskettes.

Use the format(ADM) command to create two formatted diskettes. For example, this command creates a formatted, 1.44MB diskette in the primary diskette drive:

format -v /dev/rdsk/f03ht
Label the diskettes with the name of the system, date created, and their name and number. For example:
```
mymachine		mymachine
22 June 2001		22 June 2001
Emergency Recovery	Emergency Recovery
Diskette 1 		Diskette 2
```
NOTE: Emergency recovery disks are customized for a particular system and only work on that system.
Switch from the graphical environment to the system console, by pressing <Ctrl><Alt><Esc>. Log in to the system console as root.
Ensure that /var is mounted.
Make sure there are no active users on the system. Enter:

who
If root is the only user logged in, bring the system to single-user mode with this command:

shutdown -y -g0 -i1
If other users are listed, bring the system to single-user mode with this command:

shutdown -y -g300 -i1
The -g300 flag in this command allows users 5 minutes (300 seconds) to close their files and log out. A broadcast message from root provides warnings that the system is coming down.
After the system enters single-user mode, determine in which filesystem you will create the recovery media. The filesystem must contain at least 30 MB of free space.
To determine which filesystems have enough space, enter:

/sbin/dfspace
For example, you might see:
```
/        : Disk space:  195.10 MB of  429.00 MB available (45.48%)
/stand   : Disk space:    5.82 MB of    9.99 MB available (58.28%)
/home    : Disk space:  238.86 MB of  479.00 MB available (49.87%)
/tmp     : Disk space:    7.98 MB of    8.00 MB available (99.85%)
/var/tmp : Disk space:    9.99 MB of    8.99 MB available (90.00%)
```
In this example, only the root (/) and /home filesystems have enough space to be used in creating the emergency recovery media.
Enter:

/sbin/emergency_disk -d pathname diskette

pathname is the filesystem you selected. diskette is the diskette drive where you will insert the diskette (either diskette1 or diskette2).
The following example creates an emergency recovery diskette on the first diskette drive, using the /home filesystem as its working directory:

/sbin/emergency_disk -d /home diskette1
When prompted, insert the first formatted disk into the appropriate diskette drive.
Creating each emergency recovery disk takes about fifteen minutes.
When prompted, remove the first diskette, insert the second diskette and press <Enter>.
When prompted, remove the diskette from the diskette drive.
Store the emergency recovery disks in a secure location.

NOTE: When you try to create a set of emergency floppies, you may fail and encounter messages stating that modules such as /dev/dsk/c0b0t0d0s? and /home2 could not be found. If this occurs, you should comment out the /home2 line in /etc/vfstab.

After you create the emergency recovery diskettes, do one of the following:

If your system has a tape drive, you should now create emergency recovery tapes.
If your system does not have a tape drive, then you should now create a full system backup onto the network or other media.

Creating emergency recovery tapes

Emergency recovery tapes allow you to restore your system and its data to the configuration stored on the media. To create emergency recovery tapes:

Locate and label tapes for this procedure. Include the system name, date the tapes were created, and sequence number.
The number of tapes needed varies according to the size and configuration of your system.
Switch from the graphical environment to the system console, by pressing <Ctrl><Alt><Esc>. Log in to the system console as root.
If the system is already in single-user mode, skip to step 5. Otherwise, make sure there are no active users on the system. Enter:

who
If root is the only user logged in, bring the system to single-user mode with this command:

shutdown -y -g0 -i1
If other users are listed, bring the system to single-user mode with this command:

shutdown -y -g300 -i1
The -g300 flag in this command allows users 5 minutes (300 seconds) to close their files and log out. A broadcast message from root provides warnings that the system is coming down.
At the single-user prompt (#), insert the first emergency recovery tape into the appropriate cartridge tape drive.
Create the emergency recovery tapes using either step a or step b.
a.
Use this step only if only one hard disk is installed, and this disk also contains partitions for other operating systems. Enter:

/sbin/emergency_rec -e tape tapesize
The -e option tells emergency_rec to back up the entire primary hard disk.
tape is the tape drive location of your inserted tape (ctape1 or ctape2).
tapesize is the size of the tape specified either as a number of 512-byte blocks or as a number with a suffix of k, M or G to indicate kilobytes, megabytes or gigabytes. For example, a tapesize of 512000 or 250M would indicate that the tape can hold 250MB of data. If the hard disk is larger than the capacity of a single tape, you are prompted to insert additional tapes as needed.
NOTE: You can omit tapesize if the capacity of a single tape is larger than the size of the entire hard disk.
Wait for this command to finish processing and then go to Step 7.

b.
Use this step for a multiple hard disk system or a single disk system that does not meet the conditions in Step 8a.
Enter:

/sbin/emergency_rec tape
tape is the tape drive location of your inserted tape (ctape1 or ctape2).
The emergency_rec command (without the -e option) backs up:
- all active partitions on your primary hard disk
- the /home, /home2, /var, and /usr filesystems from the secondary hard disk, if they exist there
  NOTE: The emergency_rec command does not back up any other filesystems on the secondary disk, nor any filesystems on additional hard disks.
  To back up these filesystems and volumes, see ``After creating emergency recovery tapes''.
  You are prompted to insert additional tapes as needed.
Remove the last tape from the tape drive.
Store the emergency recovery tapes with the emergency recovery floppy disks in a secure location.

Handling tape error messages

The following error messages may appear when you create emergency recovery tapes:

Cannot determine mountpoint for device <device_name>
This message appears when a slice cannot be associated with a filesystem. This prevents the indicated slice from being backed up or restored.
Duplicate mountpoint <mnt_pt> detected with device <device_name>
This message appears when the determination of what filesystems the slices are associated with produces a duplicate. Common causes of this error include using old style device names (dev/dsk/1s1 instead of /dev/dsk/c0b0t1d0s1), or having entries commented out in /etc/vfstab. If you encounter this message, you must fix the problem before making the emergency recovery tape.

After creating emergency recovery tapes

After you create the emergency recovery tapes, either:

Begin regularly scheduled backups of your system as soon as you begin using it, if you used Step 6a of ``Creating emergency recovery tapes''.
Separately back up any data that was not copied onto the emergency recovery tapes, if you used Step 6b of ``Creating emergency recovery tapes''.
Specifically, you need to back up:
- any filesystems not on the primary hard disk
- the /home, /home2, and /usr filesystems, if they are defined on a disk other than the primary or secondary hard disk
The easiest way to do this is to make a full system backup. Then, you can begin a regular schedule of incremental backups. You must use the cpio(C) command to perform these backups if you want to retain privilege on the files you back up.

Emergency recovery with an encapsulated or mirrored root disk

There are three phases to performing an emergency recovery with an encapsulated or mirrored root disk:

WARNING: If the following procedures are not adhered to exactly, it may result in your system entering an unrecoverable state.

During the following process, use the vxprint(ADM) command to obtain information about the encapsulated root disk and the disk to which it is mirrored. In this example, an initial install was done with all additional filesystems created, and mirrored onto another disk. Your installation may differ, so you may have a different number of partitions on the root disk. This procedure is valid for recovering from a disaster where the primary and mirror disks both fail, and ODM was used only to encapsulate the root disk and mirror it. Information on any other ODM disks should be saved so that they may later be restored.

Creating the Emergency Recovery boot media and tapes

Bring the machine down to init state 1 and log in as root
Run vxconfigd(ADM) to start up the ODM configuration daemon
Use vxplex(ADM) to disconnect the mirrored root plexes, for example:

vxplex dis home-02
vxplex dis home2-02
vxplex dis rootvol-02
vxplex dis standvol-02
vxplex dis swapvol-02
vxplex dis tmp-02
vxplex dis var-02
Set up the kernel to ignore the ODM configuration:
- Make a backup copy of /etc/conf/pack.d/vol/space.c using the following command:
  
  cp /etc/conf/pack.d/vol/space.c /etc/conf/pack.d/vol/space.c.old
  then edit /etc/conf/pack.d/vol/space.c, and change the two lines:
```
#define VOL_ROOTDEV_IS_VOLUME
#define VOL_SWAPDEV_IS_VOLUME
```
  to:
```
#undef VOL_ROOTDEV_IS_VOLUME
#undef VOL_SWAPDEV_IS_VOLUME
```
- Make a backup copy of /etc/conf/init.d/kernel using the following command:
  
  cp /etc/conf/init.d/kernel /etc/conf/init.d_kernel.old
  then edit /etc/conf/init.d/kernel to move the ``swp1'' entry located just after the ``vol2'' entry to just below the ``vol1'' entry, and change the line from:
```
swp1::sysinit:/sbin/swap -a /dev/vx/dsk/swapvol > /dev/sysmsg 2>&1
```
  to:
```
swp1::sysinit:/sbin/swap -a /dev/swap > /dev/sysmsg 2>&1
```
  Change the initdefault entry from ``3'' to ``1'' to avoid having to quickly press the spacebar in Step 6.
- Make a backup copy of /etc/inittab using the following command:
  
  cp /etc/inittab /etc/inittab.old
  then edit /etc/inittab in the same manner as you edited /etc/conf/init.d/kernel.
- Make a backup copy of /etc/vfstab using the following command:
  
  cp /etc/vfstab /etc/vfstab.old
  then edit /etc/vfstab, and make the following type of substitutions on the non-commented lines:
  
  Replace ``/dev/vx/[r]dsk/home'' with ``/dev/[r]dsk/c0b0t0d0s4''
  
  Replace ``/dev/vx/[r]dsk/var'' with ``/dev/[r]dsk/c0b0t0d0sb''
  
  Replace ``/dev/vx/[r]dsk/home2'' with ``/dev/[r]dsk/c0b0t0d0sc''
  
  Replace ``/dev/vx/[r]dsk/tmp'' with ``/dev/[r]dsk/c0b0t0d0sd''
  You should also comment out any other ODM-defined slices.
- Make a backup copy of /etc/swaptab using the following command:
  
  cp /etc/swaptab /etc/swaptab.old
  then edit /etc/swaptab, replacing ``/dev/vx/dsk/swapvol'' with ``/dev/swap''
- Run the following command:
  
  touch /etc/vx/reconfig.d/state.d/install-db
- Run the following command to obtain the major/minor numbers of the root slice:
  
  ls -l /dev/dsk/c0b0t0d0s1
- Replace the major/minor numbers for /.io/bootdisk/[r]root:
  
  rm /.io/bootdisk/*root
  mknod root b MAJOR MINOR
  mknod rroot c MAJOR MINOR
  where MAJOR and MINOR are the major and minor numbers displayed by the ls in the previous step.
Rebuild the kernel:

/etc/conf/bin/idbuild -B
Reboot using the init 6 command.
If you changed the initdefault entry in Step 4, then you do not need to interrupt the reboot. If you did not change the initdefault entry, when you see the system logo appear, immediately press <Space> to enter the Bootstrap Command Processor. Enter the command initstate=1, followed by boot. This will cause the machine to boot into single-user mode.
If you edited the initdefault entry in Step 4, change the initdefault entry back to ``3''.
Run the emergency_disk(ADM) and emergency_rec(ADM) commands.

Bringing the system back after emergency media is created

Undo the steps in step 4 of the previous procedure to restore the ODM configuration. When you use mknod(ADM) to create the /.io/bootdisk/*root nodes, use the major and minor numbers listed by the following command:

ls -l /dev/vx/dsk/rootvol
Rebuild the kernel again:

/etc/conf/bin/idbuild -B
Reboot using the init 6 command. When you see the system logo appear, immediately press <Space> to enter the Bootstrap Command Processor. Enter the command initstate=1, followed by ``'boot'''. This will cause the machine to boot into single-user mode.
When the system comes back up, log in as root, and run vxconfigd(ADM) to start the ODM configuration daemon
Re-attach the plexes disconnected earlier, for the mirrored disk.
For example:

vxplex att home home-02
vxplex att home2 home2-02
vxplex att rootvol rootvol-02
vxplex att standvol standvol-02
vxplex att swapvol swapvol-02
vxplex att tmp tmp-02
vxplex att var var-02
You should also remove the comments from other ODM-defined slices you previously commented out.
Reboot using init 6 and let the machine boot all the way into multiuser mode.

Performing recovery when the primary and mirrored root disks fail

Boot from the Emergency Recovery diskettes or CD.
A message should be displayed indicating that the system is not sane.
Continue to the Emergency Recovery menu and perform the Restore Disk(s) process.
After the recovery has completed, select the option to Reboot. When you see the system logo appear, immediately press <Space> to enter the Bootstrap Command Processor. Enter the command initstate=1, followed by ``boot''. This will cause the machine to boot into single-user mode.
Enter the following command:

rm /etc/vx/reconfig.d/state.d/install-db
Run vxconfigd to remove all information about the mirrored disk:

vxplex -o rm dis home-02
vxplex -o rm dis home2-02
vxplex -o rm dis rootvol-02
vxplex -o rm dis standvol-02
vxplex -o rm dis swapvol-02
vxplex -o rm dis tmp-02
vxplex -o rm dis var-02
vxedit rm disk01
Run /etc/vx/bin/vxunroot.
Remove the VOLPUBLIC partition from both the root disk and the mirrored disk, if they exist:

prtvtoc -f /tmp/vtoc /dev/rdsk/c0b0t0d0s0
Edit the /tmp/vtoc file and change the line that is similar to this:
```
14	0xe	0x201	2048	2064384
```
to:
```
14	0x0	0x0	0	0
```
Then rewrite the VTOC:

edvtoc -f /tmp/vtoc /dev/rdsk/c0b0t0d0s0
Reboot using the init 6 command. When you see the system logo appear, immediately press <Space> to enter the Bootstrap Command Processor. Enter the command initstate=1, followed by ``boot''. This will cause the machine to boot into single-user mode. Ignore the vxconfigd errors.
Run vxinstall to re-install ODM and re-encapsulate root. After the three reboots take place, you can re-mirror the root disk, add any other ODM disks, and restore the data to them. The machine will be running in multiuser mode.

Recovering an unusable system

If your system will not boot, your system software is corrupted beyond repair, or your hard disk has been reformatted or replaced, you can use your emergency recovery boot media, tapes, and incremental backups to restore your system.

Before attempting to restore the system, ensure that your motherboard, hard disks, memory, and peripherals are in good working order. Do so by running any hardware diagnostics included with your system by the manufacturer. While this recovery procedure restores all the system software, applications, and data on the recovery media to your hard disk, it does not ensure proper operation of the system hardware.

To recover the system:

Boot the system using the emergency recovery boot media (diskettes or CD). Enter your licensing information when prompted.
Correct the system damage or restore data from the emergency recovery tapes.
The emergency recovery main menu provides options to start a limited command-line shell to restore data from emergency recovery tapes, to mount or unmount all filesystems (if data is accessible on the hard disk), and to reboot the system.

Use the up or down arrow keys or the <Tab> key to select the desired choice and then press <Enter>.

If you select to invoke an emergency shell, a shell prompt is displayed and the following commands are available:

cat(C) chroot(ADM) cpio(C) date(C)

dd(ADM) echo(C) edvtoc(ADM) fdisk(ADM)

find(C) fsck(ADM) grep(C) ksh(C)

labelit(ADM) ln(C) ls(C) mkdir(C)

mkfs(C) mount(ADM) prtvtoc(ADM) rm(C)

stty(C) vi(C)

Use these commands to investigate and fix the problem. To exit the shell and return to the main menu, press <Esc>.

NOTE: Some of these commands (for example, thefind command) are limited-function shell aliases for the regular commands. If the hard disk is functional and mounted, you can execute the full-function commands from the hard disk by specifying the full path of the command (for example, /mnt/usr/bin/find).

If you select to reinstall your system, insert the emergency recovery tapes when prompted. Wait for the processing of the last tape to complete before going to the next step.
If you select to mount filesystems, the filesystems are mounted. If you then invoke the emergency shell, you can access all the commands on your system. If your hard disk is severely damaged, this option may not be available.
If you select to unmount filesystems, the filesystems are unmounted. If your hard disk is severely damaged, this option may not be available.
You can choose to reboot your system. If you used the emergency recovery shell to repair your system, remove the emergency recovery boot media from the drive before selecting the option to reboot your system.

Remove the emergency recovery boot media. If you restored the system from the emergency recovery tapes, you can remove the last tape from the tape drive at this time.
Press <Esc> to reboot your system.
You may notice error messages similar to the following:
```
UX: initprivs: WARNING: File ``file'' fails validation: entry ignored
UX: initprivs: WARNING: X entries ignored in ``/etc/security/tcb/privs''
```
This is because the date stamp for the inode was changed during the restore process.
You can fix these errors after your system boots into multi-user mode, by logging in as root and entering the following command:

/etc/security/tools/setpriv -x
If you created any full or incremental backups of your system after creating the emergency recovery tapes, restore those backups using the restore method corresponding to the backup method you used to create them.
For example, if you created backups using the cpio(C) command, use the same method to restore the archive.

cat(C)	chroot(ADM)	cpio(C)	date(C)
dd(ADM)	echo(C)	edvtoc(ADM)	fdisk(ADM)
find(C)	fsck(ADM)	grep(C)	ksh(C)
labelit(ADM)	ln(C)	ls(C)	mkdir(C)
mkfs(C)	mount(ADM)	prtvtoc(ADM)	rm(C)
stty(C)	vi(C)