Technical Memorandum
Title: Linux Raid Configuration for Dell PowerVault (FIber-Channel) 660F and 224F with PowerEdge 2550 server running RedHat Linux 7.0.
Date: 24 September, 2001
A. Zaimi
S.J. Hanson
Psychology Department-RUMBA Labs
Rutgers University
Newark, NJ 07102
Starting with a PowerVault 660F and 224F units and 2 PowerEdge servers with factory installed RedHat 7.0, we initially configured 1 PowerEdge server under Linux to access the PowerVault units. We connected the two PowerVault together with the fiber connectors (lower connector of each side of the PowerVault 660F went into the upper connector of the same side in the PowerVault 224F), and we connected the fiber cord from one of the upper slots on the back of PowerVault 660F to the first PowerEdge server (to one of the two fiber QLogic cards). Figure 1 shows the configuration.
On the back of each PowerVault unit, there is a small switch; PowerVault 660F needs to be set to 0 and PowerVault 224F needs to be set to 1 for a Master/Slave configuration (Figure 2).
After powering up the first PowerEdge server connected to the PowerVault, we observed whether or not it detected the PowerVault. During the boot up the PowerEdge server did appear to detect the PowerVault 660F based on boot messages. Boot messages also indicated that the QLogic driver was loaded by the kernel, but subsequently it also indicated an error comunicating to the PowerVault unit. Any attempts to access the PowerVault unit, for example: fdisk /dev/sdb (the scsi unit to which the PowerVault would correspond, /dev/sda is the local disk), would result in an error: Unable to communicate with device....
The connection between the PowerVault and the PowerEdge server is made through a fiber channel and the QLogic 2200 adaptor (see http://www.qlogic.com/products/qla2200.html) which allows for communication bandwidth up to 200 MB/s (or 1Gbit) in full duplex. Each PowerEdge server came with 2 QLogic fiber cards, but apparently only one card is needed to communicate between a PowerEdge to the PowerVault. Two cards were probably installed as each PowerEdge was a Dual Processor. The other card of the first PowerEdge server (and both cards of the second PowerEdge) were useless therefore we took advantage of this fact to utilise one of these cards (from the second PowerEdge) to install it on a windows NT machine.
Dell Technical Support indicated that the PowerEdge servers would not be able to communicate to the PowerVault as we did not have the most recent QLogic driver for the PowerEdge Linux server. Once the new QLogic drivers were downloaded and installed (after downloading it from the QLogic website: http://www.qlogic.com/bbs-html/csg_web/adapter_pages/fc_adapters.html) the same error messages indicating lack of communication between the Power Vaults and QLogic card/PowerEdge appeared. This implied that it was not the recency of the driver causing the communication link problem. Further investigation indicated that in order for the PowerEdge to talk to the PowerVault, we needed to configure the PowerVault Arrays first by unmasking the Logical Unit Numbers (LUNs) with some special microcode-PowerVault controller firmware.
Once we were aware that the a proper configuration procedure involved unmasking the LUNs of the PowerVault Arrays first, we investigated possible options to PowerVault configuration and the micro-controller. At present there is no direct option to perform such a configuration from a Linux server since Dell does not have such a software running under Linux. One other possibility involved accessing the PowerVault microcontroller to unmask the LUNs by using a laptop through a serial connection to the fiber slots of the PowerVault. This would provide a direct connection to the microcontroller on the PowerVault and would require using a special micrococde sequence. This microcode command sequence, however, was unavailable from Dell. The final possibility we considered was to temporarilly configure the PowerVault through the Array Manager Software tool that Dell ships with the PowerVault but which runs under the Windows NT/2000 operating systems. Once configured it should be possible to access the disk as SCSI through standard Linux command line disk functions.
Concurrently we configured a generic Dell precision 410 workstation with microsoft windowsNT 4.0 SP5 in order to use the Array Manager Tools under NT. We removed one of the QLogic fiber cards from the second PowerEdge and put it into the PCI slot of the Dell workstation. The QLogic cards are 64bit PCI cards but they works just as well in the 32bit PCI slots which we had in our NT workstation (64bit PCI cards generally work in 32bit slots but their performance may be slower).
We installed the QLogic 8.0 driver in the windows machine, rebooted, and the QLogic card was detected. We installed Dell OpenManage Array Controller software, version 4.0 and connected the fiber cord from the PowerVault 660F to the Dell workstation with NT and rebooted the workstation. When it rebooted the QLogic bios detected the PowerVault 660F. When we started Array Manager we were able to see the PowerVault 660F and PowerVault 224F as unit 1 and unit 2.
It would be desirable for our application to have one single volume consisting of the two PV units, the 660F and the 224F. Each of the units has 14 disks for a total of 28 disks, making up a maximum total storage of 1000GB. But we found out that the Open Manage tool does not allow to create a group with more than 16 disks which means that we were forced to have more than one group. Given that we have 14 disks per PowerVault we found it fit to have each PowerVault (with 14 disks) as a separate group-this meant that each PowerVault will be a separate volume from the stand point of the disk organisation from the Linux prospective. So, we created two groups under the Disks entry by selecting the disks units for each PowerVault units, set them to Raid level 0+1, and initialised each group (each group separately, this takes 4-6 hours per group and it is equivalent to low level formating, it probably could be skipped but we didn't).
Again for our application the Raid level 0+1 (striping after data mirroring) was most appropriate. Its advantageges are that this level offers redunancy (two copies of the data) and it spreads the data over the disks for speed (striping). The disadvantage is that we loose half of the Arrays' space because of the redunancy, i.e. the maximum total Array space we could have was 1000GB and we now only had available a total of 500GB using Raid level 0+1. We could have used Raid level 5 which offers some redunancy and good performance (speed) and waists only 25% of the overall maximum Array space. But the security of the data was an important issue for our application so we chose to have less space available but to better data protection from any eventual disk corruption in the PowerVault Arrays.
We disconected the PowerVault from the Dell NT workstation, and reconnected it to the PowerEdge server running Redhat Linux 7.0 and restarted the server so that it potentially would detect the PowerVault. After reboot, in the boot mesages we could see that /dev/sdb and /dev/sdc (which are the device names under which the PowerVault identifies itself-it shows up as a scsi drive) were recongnised and this seemed to eliminate the error mesages in the earlier transaction.
After the linux server was able to see the two Arrays, we needed to create a partition and hae a filesystem for each of the two volumes. These are operations that need to be performed on any new (blank) disk installed on a linux machine; refer to a Linux manual for further details or consult the manual pages of each command for instructions and internal options. To create the partitions we used the command fdisk which is a menu driven program for creation and manipulation of partition tables (type man fdisk to read the manual of the program). Any other program to manipulate the partition table for hard disk would work in a similar fashion. We typed:
fdisk /dev/sdb
and created a new primary partition (by selecting n-for new) and giving it the whole size of the volume (245GB). We did the same for /dev/sdc:
fdisk /dev/sdc
Once the partition tables were saved to disk, we needed to write a file system to each volume (to the new partition we just created for each volume) and for this we used the command mke2fs which is used to create a Linux second extended file system on a device, usually a disk partition (type man mke2fs for the manual pages). We typed:
mke2fs /dev/sdb1
mke2fs /dev/sdc1
Now the two volumes were ready to be mounted:
mount /dev/sdb1 /volume1
mount /dev/sdc1 /volume2
and the mounting operation was successful. Now the Raid Array was ready to read and write data.
Then we tested the reading and writing efficiency and speed for big files (larger than 100MB). The read/write operations were a success and the speed of the read and write process was astonishing: we were reading and writing files of size 1GB within 30 seconds (write=17s, read=32s) compared to the local scsi raid drive of the PowerEdge server (write=47s, read=31s) and compared to our SUN Enterprise server (write=63s, read=9s). At last, we made permanent entries in the /etc/fstab to have the Arrays automatically mounted at boot time:
/dev/sdb1 /volume1 ext2 defaults 1 2
/dev/sdc1 /volume2 ext2 defaults 1 2
So the PowerVault was successfully configured under Linux although the entire procedure was difficult especially given the uncertainty of whether Linux OS would be able to recognise the Raid Arrays as valid units (drives). Fortunately, Linux is a good choice for this kind of configuration and we had little further problem in getting the Arrays functional once the PowerVault Firmware was initialised and configured correctly from the Array Manager Tool which we intialized independently under the Windows NT environment.
This document was generated using the LaTeX2HTML translator Version 2002 (1.62)
2003-10-28