CONTEXT
These notes are here partially for my own reference / for future; and in case they are of use to someone else.
The machine used for this work was a Dell t300 with internal Perc6i SATA HW Raid Controller (LSI MegaRaid chipset based)
In case it is of interest to anyone, it appears this controller is OK with the 3rd party drives used to upgrade storage capacity: "HITACHI Deskstar 7K2000 HDS722020ALA330 (0F10311) 2TB 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive -Bare Drive" -- as per the NewEgg URL for reference: http://www.newegg.ca/Product/Product.aspx?Item=N82E16822145298&nm_mc=TEMC-RMA-Approvel&cm_mmc=TEMC-RMA-Approvel-_-Content-_-text-_-
(This is not an endorsement of Newegg, etc, just a link for clarity in case someone else wants to consider using such disks as 'possibly OK' in their Dell 6i raid config).
(Additional disclaimer, using 3rd party drives in your Dell gear is clearly not supported by Dell, you do so at your own risk, etc etc. Your server may implode if you attempt this. Don't blame me if you decide to do this and things don't work well for you :-)
Note: Why would you want to do this?
- Single ProxVE host with generous amount of local storage in HW raid
- Note that most Linux distros are not happy to boot from a volume > 2Tb in size
- so a work-around is to carve out 2 LUNs on the raid-controller; the first is some 'modest' size like 100Gigs for the OS; and the second LUN is intended for "bulk data". In the case of ProxVE, I'm installing ProxVE initially into just the 100Gig LUN; but subsequently want to reconfigure it so that the ~3+Tb LUN is used for VM Image storage; and the 100Gig LUN is used for ProxVE platform, swap, etc.
TDC Oct-1-2010
CONCISE SUMMARY OF LVM REARRANGE ON PROXVE:
TARBALL BACKUP:
proxmox:~# cd /var/lib/vz proxmox:/var/lib/vz# tar cvf /var-lib-vz.tar ./*
CHECK STATUS OF LVs:
proxmox:/# lvdisplay --- Logical volume --- LV Name /dev/pve/swap VG Name pve LV UUID cj7Dfz-9y1C-6cSw-0wGa-clta-k91i-a2lygU LV Write Access read/write LV Status available # open 1 LV Size 4.00 GB Current LE 1024 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 254:0 --- Logical volume --- LV Name /dev/pve/root VG Name pve LV UUID zeaXvP-J4cO-3ybZ-AseL-V7E3-ZcHE-9PGdts LV Write Access read/write LV Status available # open 1 LV Size 24.25 GB Current LE 6208 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 254:1 proxmox:/#
CHECK VOLUME GROUP STATUS - total capacity ?
proxmox:/# vgdisplay --- Volume group --- VG Name pve System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 5 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 97.15 GB PE Size 4.00 MB Total PE 24871 Alloc PE / Size 7232 / 28.25 GB Free PE / Size 17639 / 68.90 GB VG UUID 6s2HNY-mWlw-RVve-XlYC-z2LU-PW1G-Gpbpe3 proxmox:/#
so: add the free space - 68.90 GB - to the root LV. Thus:
proxmox:/# lvresize -L +68.90G /dev/pve/root Rounding up size to full physical extent 68.90 GB Extending logical volume root to 93.15 GB Logical volume root successfully resized proxmox:/#
Now check status:
proxmox:/# vgdisplay --- Volume group --- VG Name pve System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 6 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 1 Act PV 1 VG Size 97.15 GB PE Size 4.00 MB Total PE 24871 Alloc PE / Size 24871 / 97.15 GB Free PE / Size 0 / 0 VG UUID 6s2HNY-mWlw-RVve-XlYC-z2LU-PW1G-Gpbpe3
NEXT UP: PREP THE SDB DRIVE FOR ADDITION
USE PARTED BECAUSE IT IS >2Tb in size, Fdisk will choke.
26 apt-get install parted
RUN PARTED AND....
proxmox:/# parted /dev/sdb GNU Parted 1.8.8 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: DELL PERC 6/i Adapter (scsi) Disk /dev/sdb: 3895GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags (parted) mklabel Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue? Yes/No? yes New disk label type? [msdos]? gpt (parted) mkpart primary 0% 100% (parted) set 1 lvm on (parted) p Model: DELL PERC 6/i Adapter (scsi) Disk /dev/sdb: 3895GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 3895GB 3895GB primary lvm (parted) q Information: You may need to update /etc/fstab.
ADD NEW STORAGE INTO LVM:
proxmox:/# pvcreate /dev/sdb1 Physical volume "/dev/sdb1" successfully created proxmox:/# vgextend pve /dev/sdb1 Volume group "pve" successfully extended proxmox:/# vgdisplay --- Volume group --- VG Name pve System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 11 VG Access read/write VG Status resizable MAX LV 0 Cur LV 2 Open LV 2 Max PV 0 Cur PV 2 Act PV 2 VG Size 3.64 TB PE Size 4.00 MB Total PE 953470 Alloc PE / Size 24871 / 97.15 GB Free PE / Size 928599 / 3.54 TB VG UUID 6s2HNY-mWlw-RVve-XlYC-z2LU-PW1G-Gpbpe3 proxmox:/#
OK: Create the new LV, pve-data:
lvcreate -L 3.54T -n /dev/pve/data pve
check that space is allocated:
proxmox:~# vgdisplay --- Volume group --- VG Name pve System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 12 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 3 Max PV 0 Cur PV 2 Act PV 2 VG Size 3.64 TB PE Size 4.00 MB Total PE 953470 Alloc PE / Size 952861 / 3.63 TB Free PE / Size 609 / 2.38 GB VG UUID 6s2HNY-mWlw-RVve-XlYC-z2LU-PW1G-Gpbpe3
Check the LVs to be certain all is well:
proxmox:~# lvdisplay --- Logical volume --- LV Name /dev/pve/swap VG Name pve LV UUID cj7Dfz-9y1C-6cSw-0wGa-clta-k91i-a2lygU LV Write Access read/write LV Status available # open 1 LV Size 4.00 GB Current LE 1024 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 254:0 --- Logical volume --- LV Name /dev/pve/root VG Name pve LV UUID zeaXvP-J4cO-3ybZ-AseL-V7E3-ZcHE-9PGdts LV Write Access read/write LV Status available # open 1 LV Size 93.15 GB Current LE 23847 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 254:1 --- Logical volume --- LV Name /dev/pve/data VG Name pve LV UUID ZWW2x1-UegY-EUAT-8Mk8-T1BM-M9dj-0rds2W LV Write Access read/write LV Status available # open 1 LV Size 3.54 TB Current LE 927990 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 254:2 proxmox:~#
FORMAT THE THING:
(note this takes a few minutes - 15min? - it is a big filesystem after all)
proxmox:/# mke2fs -j /dev/pve/data mke2fs 1.41.3 (12-Oct-2008) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 237568000 inodes, 950261760 blocks 47513088 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=4294967296 29000 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000, 214990848, 512000000, 550731776, 644972544 Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 37 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. proxmox:/#
WHILE THAT IS GRINDING ALONG;
Open another SSH session
Resize the ROOT FILESYSTEM to use up the space available to it.
another shell session: resize my root fs.
proxmox:~# resize2fs /dev/pve/root 24419328 resize2fs 1.41.3 (12-Oct-2008) Filesystem at /dev/pve/root is mounted on /; on-line resizing required old desc_blocks = 2, new_desc_blocks = 6 Performing an on-line resize of /dev/pve/root to 24419328 (4k) blocks. The filesystem on /dev/pve/root is now 24419328 blocks long.
(NOTE THIS STEP TAKES A LITTLE WHILE ALSO - GO GET A COFFEE/ETC)
ONCE FS FORMAT IS DONE (IN OTHER SHELL SESSION:) Finally, mount it, untar, we're good.
proxmox:/# more /etc/fstab # <file system> <mount point> <type> <options> <dump> <pass> /dev/pve/root / ext3 errors=remount-ro 0 1 /dev/pve/data /var/lib/vz ext3 defaults 0 1 UUID=9249231d-b67c-4dae-9691-cc66aa63be64 /boot ext3 defaults 0 1 /dev/pve/swap none swap sw 0 0 proc /proc proc defaults 0 0 proxmox:/# mount /var/lib/vz proxmox:/# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/pve-root 96143116 765168 90494968 1% / tmpfs 2019720 0 2019720 0% /lib/init/rw udev 10240 592 9648 6% /dev tmpfs 2019720 0 2019720 0% /dev/shm /dev/sda1 516040 45016 444812 10% /boot /dev/mapper/pve-data 3741402976 201364 3551149260 1% /var/lib/vz proxmox:/# cd /var/lib/vz proxmox:/var/lib/vz# tar xvf /var-lib-vz.tar ./dump/ ./images/ ./lock/ ./lost+found/ ./private/ ./root/ ./template/ ./template/qemu/ ./template/iso/ ./template/cache/ proxmox:/var/lib/vz# pwd /var/lib/vz proxmox:/var/lib/vz# ls -la total 48 drwxr-xr-x 9 root root 4096 Sep 30 21:46 . drwxr-xr-x 32 root root 4096 Sep 30 20:19 .. drwxr-xr-x 2 root root 4096 Sep 3 09:21 dump drwxr-xr-x 2 root root 4096 Aug 31 06:14 images drwxr-xr-x 2 root root 4096 Sep 30 20:29 lock drwx------ 2 root root 16384 Sep 30 20:17 lost+found drwxr-xr-x 2 root root 4096 Sep 20 07:03 private drwxr-xr-x 2 root root 4096 Sep 20 07:03 root drwxr-xr-x 5 root root 4096 Sep 30 20:19 template proxmox:/var/lib/vz#
SO:
- Root FS has 700meg used, 90 gig free. - VZ Data FS has 3.4 Tb free.
proxmox:/var/lib/vz# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/pve-root 92G 748M 87G 1% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw udev 10M 592K 9.5M 6% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/sda1 504M 44M 435M 10% /boot /dev/mapper/pve-data 3.5T 197M 3.4T 1% /var/lib/vz proxmox:/var/lib/vz#