January 27, 2004

EYEWI disk failure

EYEWI lost a disk on Saturday morning.

The good: Jumpstart solves most everything, and /home was being backed up via fssnap and ufsdump to the standalone DLT drive. That got me most of the data restored, other than /usr/local, which really only had the rcs.mgr data in it. NetBackup was pretty happy with the restored /home/opt directory when I moved that into place.

The bad: The Jumpstart script for mirroring the drives does not perform (properly) the final metattach on the mirror sets, so the mirror only includes the first half. I did the metattach on all the remaining Jumpstarted servers, and I’ll be fixing that in the Jumpstart script. The upshot is that the second disk was useless for booting, and didn’t have /home or /usr like it should have.

What I did:

  • Use spare disk from ROJ’s AMANDA scratch to copy / from the surviving drive. Also did a uffsrestore from the catalog backup tape.
  • Use second of ROJ’s AMANDA scratch disks and the survivng disk to Jumpstart the system.
  • ufsrestore from the catalog backup tape onto the new system.
  • Move /home/opt to /home/opt.old.
  • Install NetBackup.
  • Move /home/opt to /home/opt.new, and move /home/opt.old to /home opt.
  • Enter NetBackup GUI console to fix the device files for the tape drives (110T, which was /dev/rmt/0* became /dev/rmt/1*, and Overland library went from /dev/rmt/1* to /dev/rmt/0*).
  • Fix /etc/system with increased shared memory configuration (restored from / partition on spare AMANDA disk).

There were a few other oddments scattered in there, as well - getting the system on the private net on bge1 (so I could run the NetBackup install from the cdserver), and things like that.

Last night’s backups didn’t work, since they fired off before I had updated /etc/system with the shared memory bumps. I manually fired off backups for a few key systems, though, and they worked. I also tested a restore of some data on ASHTI, and that worked.

I think things are stable and working as well as they used to now.

Posted by Rowan Littell at January 27, 2004 11:43 AM
Comments