Advanced Procedures: Using SCSI/RAID or other devices

Advanced LRP users may wish to boot LRP from a SCSI drive, RAID array, flash disk, or some other device that does not have support directly compiled in as part of the kernel. There are two possible ways to do this:

  1. Re-compile the LRP kernel, including support for the desired device

  2. Since LRP boots with an initial ramdisk (root.lrp), module support for the desired devices can be added at runtime, if everything required is archived as part of root.lrp.

I will cover the second method here.

OVERVIEW

The key to booting LRP from a SCSI disk, RAID array, or anything else that is not supported by the kernel without modules is folding the required files into the initial ramdisk (root.lrp) and adding support for the necessary devices before the kernel tries to access them. This is actually why initrd (initial ramdisk) support was first added to the linux kernel...the fancy footwork LRP does to turn the initial ramdisk into a full blown file system came later. The fact that LRP makes use of a simple .tgz file for the initial ramdisk actually makes the task of supporting odd hardware much easier. All you have to do is add some files to your running LRP system and back them up as part of root.lrp.

BEFORE YOU GET STARTED

Setting up a RAID array on an LRP system is about as complex as drive configuration gets under linux. I assume if you're attempting this, you have a solid understanding of linux block devices (especially IDE and/or SCSI drives), and are very familiar with disk partitioning. I also assume you are familiar with setting up hard disks for LRP, have gone through at least one of the two procedures outlined above, and thoroughly understood what you did.

You will need a few things, so try to track them all down before you get started.

  1. The Software-RAID-HOWTO. Download this, print it out, and read it completely before you go any further. Then read it again.

  2. A working LRP system with the hard disk and controller installed.

  3. Hard disk support utilities (fdisk, fsck, mke2fs, and the like). Typically packaged as hdsupp.lrp

  4. The RAID tools package (raid.lrp).

  5. A kernel compiled with RAID support.

  6. Kernel modules to support your HDD controller, SCSI support (if your HDD is SCSI), and the RAID modes you plan to use. For my Adaptec SCSI controller running RAID1 (mirrored), I needed:

    scsi_mod.o - Kernel SCSI support
    sd_mod.o   - SCSI Disk support
    aic7xxx.o  - Adaptec SCSI driver
    raid1.o    - Raid 1 personality

    Note

    The RAID kernel does not include built-in IDE support. If you are using IDE drives, you will have to load IDE support as a module, just like the SCSI support listed above.

  7. The ext2.o kernel module for your LRP kernel version. This should be available from the same place you got the IDE support module or kernel, above.

  8. <optional> The hdsupp_s.lrp package, which is used to check ext2 data partitions at boot time (and doesn't include the disk formatting utilities for size/security reasons).

PRELIMINARIES

Setup and configure a conventional floppy based LRP distribution of your choice. This will be the basis for your new SCSI+RAID system.

Warning

MAKE ONE OR MORE BACKUPS OF YOUR LRP DISK. You will be heavily modifying the startup scripts on one of these disks. If you mess anything up, LRP will crash horribly (no /etc directory, no inittab file, no boot) and you will have to boot from one of your backup disks.

Clearly label your original, backup, and (soon to be) RAID floppies, so you don't get them confused later.

ADDING BOOTSTRAP MODULE SUPPORT

Replace the kernel on your RAID LRP floppy with the RAID kernel and reboot. You will see a lot of errors about kernel-module version mismatches, but you should get to a login prompt. You'll update the modules later to get rid of the errors.

Note

If you don't have enough room on your disk for the new kernel, delete some of your LRP packages or modules.lrp to get some elbow room. You've got backups, and can add the packages back later when you're booting off the HDD.

Create a directory where you would like to put the boot-strap modules. I've put mine in various locations, but finally settled on /boot/modules.

Copy the modules required for SCSI+RAID (or whatever your odd boot device is) to the directory you just created. Verify everything works by loading the modules manually using insmod. If you have any problems at this point, fix them before continuing. For my system, I ran the following commands (you may need to supply parameters to some of the modules...adjust accordingly):

insmod /boot/modules/scsi_mod.o
insmod /boot/modules/sd_mod.o
insmod /boot/modules/aic7xxx.o
insmod /boot/modules/raid1.o

If the above worked, you need to create a file that tells the init scripts which modules to load. I used the file /boot/modules/modules.conf, with the same format as the /etc/modules you should already be familiar with.

Now, it's time to configure linuxrc to load the modules when LRP boots. I added the following lines to /linuxrc just before the code that queries the kernel command line for the 'boot' option:

#
# Load any modules required to boot
#
MODDIR=/boot/modules
if [ -r $MODDIR/modules.conf ] ; then
  # Loop over every line in modules
  echo 'Loading modules:'
  (cat $MODDIR/modules.conf ; echo) | # make sure file ends with LF
  while read module args
  do
        case "$module" in
        \#*|"") continue;;
        esac
        echo "  ${module}:"
        /sbin/insmod $MODDIR/${module%.o}.o $args
  done
  echo "  Done"
fi

Check to make sure the /dev directory contains all the devices you need for your hardware. You may also need to update /var/lib/lrpkg/root.dev.mk to create additional device nodes. I added the following lines on my system to support the newly added RAID functionality:

# RAID Devices
makedevs md b 9 0 0 15 >null 2>&1

At this point, double check everything, then backup root.lrp. Double check everything again, and as a final test, remove the modules you manually added (using rmmod), and run the linuxrc script manually. This will re-install all the LRP files except root, which shouldn't hurt anything, load the bootstrap modules, and create any new devices you added. If linuxrc won't run, doesn't load the modules, or doesn't create the devices, try to figure out why and fix it. Back up root.lrp when you're done.

Reboot your machine.

If you got everything correct, your system should boot off the floppy and load the bootstrap modules prior to the loading the LRP packages. You can verify the modules all loaded by looking at /proc/modules once the system is booted. If you goofed and the modules didn't load, fix whatever's wrong. If you really goofed and the linuxrc script didn't run (or crashed with syntax errors), your system will not boot, and you'll have to start over (you did make backup disks, right?).

FINISHING UP

Now that your system is properly loading bootstrap modules, you can partition your disks, create RAID arrays, format filesystems, and generally make use of your drives. You can even do things like load all your LRP files off a RAID5 ext2 partition if you want. The only requirement is that syslinux, the linux kernel, and root.lrp are available on a FAT partition syslinux can boot from, and even this is not mandatory if you use a different boot loader (grub, lilo, ldlinux, etc).

To get your system booting off your new device, see Method 1: Booting LRP from a hard disk, above.

Creating a RAID array

First, partition your drives, creating appropriate (typically equal sized) partitions with a partition type of 0xFD (raid-autodetect).

Load the raid.lrp package, editing /etc/raidtab as appropriate (see the Software-RAID-HOWTO). Backup raid.lrp to save your changes to /etc/raidtab.

Make the raid devices using mkraid.

At this point you can treat the newly created md device like any other harddisk partition. Format it using mke2fs (see the Software-RAID-HOWTO for some RAID-specific mke2fs parameters). You can load LRP packages from it by setting the PKGPATH kernel parameter, and mount the partition at run-time either manually or using /etc/fstab entries. See Methods 1 and 2, above, for details.

Booting LRP from a RAID array

You can setup LRP to boot from a RAID1 (Mirrored) array if you so desire. For additional background on the following procedure, see the Boot + Root + Raid + Lilo : Software Raid HOWTO. Basically, you get LRP booting of a hard disk, then migrate the single hard disk partition to a RAID1 Mirror.

I will describe the system I setup, which boots from a mirrored array (/dev/md0) made up of two partitions, /dev/sda1, and /dev/sdb1.

First, setup LRP to boot off your hard disk. This is covered in Method 1, above. Make sure you partition /dev/sda and /dev/sdb identically (at least for the two partitions you plan to mirror). If you don't have two identical drives, you can used unmatched drives, but make sure the partition on your secondary drive (/dev/sdb in this case) is at least as large as the partition on the primary drive (/dev/sda), or you will be unable to add it to the RAID array later.

Create an /etc/raidtab entry for a RAID1 array with a bad drive. My raidtab file looked like:

raiddev /dev/md0
  raid-level            1
  nr-raid-disks         2
  nr-spare-disks        0
  chunk-size            4
  persistent-superblock 1
  device                /dev/sda1
  raid-disk             0
  device                /dev/sdb1
  failed-disk           1

Warning

MAKE SURE NONE OF THE PARTITIONS INVOLVED ARE MOUNTED!

Run fdisk and change the partition type on /dev/sda1 to 0xFD

Make the raid device:

mkraid /dev/md0

Delete /dev/boot, and replace it with a link to /dev/md0. This is required so you don't accidentally backup data to just /dev/sda1, which would corrupt the array.

rm /dev/boot
ln -s /dev/md0 /dev/boot

Run fdisk and change the partition type on /dev/sdb1 to 0xFD

Add /dev/sdb1 to the raid array:

raidhotadd /dev/md0 /dev/sdb1

Edit /etc/raidtab and changed 'failed-disk' to 'raid-disk'

Edit /etc/raidboot.conf and set RAIDBOOT=YES and RAIDBOOTDEV=/dev/md0 so the boot device will be switched to /dev/md0 the next time you boot.

Backup raid.lrp

NOTES

You still set boot=/dev/sda1,msdos in syslinux.cfg. This is because when linuxrc runs, the RAID arrays are not autodetected yet (this happens after linuxrc exits). If one of your disks dies, you may have to edit syslinux.cfg on the working disk, but you should be able to boot off either disk.

While it's important you NEVER write data directly to a partition that is part of a RAID array (use the proper md* device instead), it's OK for syslinux and linuxrc to READ data from a single disk when booting. As long as you're only reading data, you're not in danger of corrupting your RAID array.

TESTING

You might want to test loosing a drive (just unplug it from the SCSI or IDE cable) and booting off the other disk before you get too far along in installing LRP (in case you accidentally loose your data). After you verify you can boot with one disk 'failed', hook everything back up. When you reboot, your raid should automatically rebuild itself, as indicated in /proc/mdstat.

You might also want to test manually rebuilding your array, which you would have to do if a disk actually died and was replaced with a new, unformatted disk. A good way to test this without having a spare third disk is to 'wipe' your current disk by running the following commands (count is the number of blocks on the partition as reported by fdisk).

Shutdown the array, so we can play with the individual partitions:

raidstop /dev/md0

Wipe all data from one of the disks. This will look like a failed disk to the kernel RAID support, since the RAID superblock got trashed:

dd </dev/zero >/dev/sdb1 bs=1024 count=21488

Restart the RAID array. You should see errors indicating /dev/sdb1 is not available. The array should run, but will be in degraded mode.

You are now the proud owner of a RAID array with a broken disk. Take a look at /proc/mdstat to see the differences between the degraded RAID and how things normally look.

I used the following procedure to recover from this (NOTE: If you actually were starting with a 'new' disk, you would first have to create an appropriate partition with a type of 0xFD):

raidhotadd /dev/md0 /dev/sdb1

This adds /dev/sdb1 back to the array. You can verify this by looking at /proc/mdstat, which should list sbd1[1] and sda1[0] as part of md0, and will indicate the mirror reconstruction is in progress (or complete).

Verify the listing in /proc/mdstat matches your /etc/raidtab file so you won't have any unpleasant surprises in the future. On my system, the files look like:

/proc/mdstat:

  Personalities : [raid1] 
  read_ahead 1024 sectors
  md0 : active raid1 sdb1[1] sda1[0] 21376 blocks [2/2] [UU]
  unused devices: <none>

/etc/raidtab:

  raiddev /dev/md0
    raid-level            1
    nr-raid-disks         2
    nr-spare-disks        0
    chunk-size            4
    persistent-superblock 1
    device                /dev/sda1
    raid-disk             0
    device                /dev/sdb1
    raid-disk             1

Make sure that the device & raid-disk settings in /etc/raidtab match the device[raid-disk] settings in /proc/mdstat:

mdstat:           raidtab:           
-------------------------------------
sda1[0]           device    /dev/sda1
                    raid-disk 0        
-------------------------------------
sdb1[1]           device    /dev/sdb1
                    raid-disk 1        
-------------------------------------

If you ever need to re-construct your RAID array, the raid-disk numbers are important, especially if you are running anything other than RAID1 (like RAID0 or RAID5).