Sysadmin:Services:XenServerSetup

New CS Hardware (November 2014)

As of November 2014, we are working on installing new hardware to provide services to CS Students/Faculty/Staff. Details about what goes where and how resources are allocated can be found here.

Hardware Specs

Model	Silicon Mechanics A346 (2U Chassis), Supermicro Motherboard
CPU	2x 8-Core AMD Opteron 6308@3.5Ghz w/4MB L2 Cache
Ram	64GB DDR3 (16 slots available total)
Storage	6 3TB SAS HDD's configured in RAID10 (/dev/sda, 3 Striped Mirrored Pairs, 9TB effective storage, used for VM backing) + 2 240GB SSD's in RAID0 (/dev/sdb, used for Dom0 Backing)
Network	2x Gigabit Ethernet Onboard, 4x Gigabit Ethernet on PCIe, 2x 10G Ethernet on PCIe

Hard Drive Partitioning

Hard Disks are split using LVM, which is nested to some degree. The 9TB of rotational media is allocated exclusively for use by Xen to back virtual machines. Xen handles much of the partition creation on that end. ext4 is used for the most part as the underlying filesystem.

The 9TB Raid Drive forms a volume group (/dev/vmstore) split into two parts using LVM: one 1TB Logical Volume (/dev/vmstore/local) for local VM storage (NOT in production) and one 7TB Logica Volume (/dev/vmstore/shared) for production VM's that may be mirrored to other machines in the future. The rest of the drive is left over as extra storage for later use.

There is some reasoning behind the apparent complexity to this system: we may use Direct Replicated Block Device (drbd) in the future to mirror all VM data to another machine. Hypothetically, the shared LV /dev/vmstore/shared will be mirrored across both machines, taking all the suboordinate volumes with it.

Volume Layout (Super-Detailed version)

Device Location	Type	Filesystem	Size	Mounted at	Usage
/dev/sda	Raw Disk		9TB		Points to main 9TB RAID Array, not mounted directly
/dev/sda1	Linux Extended	LVM	9TB		Space allocation for use by LVM (acts as PV)
/dev/sdb	Raw Disk		239GB		Contains files for dom0
/dev/sdb1	Linux Primary	ext2	255MB	/boot	Contains boot files for dom0
/dev/sdb2	Linux Extended		239GB
/dev/sdb5	Linux Logical	LVM	239GB
/dev/vmctrl	LVM VG (on /dev/sdb5)		239GB
/dev/vmctrl/root	LVM LV	ext4	213GB	/	dom0 root filesystem (debian)
/dev/vmctrl/swap	LVM LV	Linux Swap	9GB		dom0 swap partition
/dev/sdb5	Linux Logical	LVM	239GB
/dev/vmstore	LVM VG		9TB		Main volume group for organizing vm storage
/dev/vmstore/local	LVM LV		1TB		Logical Disk for local vm storage (backs volume group)
/dev/vmstore/shared	LVM LV		7TB		Logical Disk for shared vm storage (backs volume group)
/dev/vmstore-local	LVM VG (on /dev/vmstore/local)		1TB		Volume group for local storage volumes
/dev/vmstore-shared	LVM VG (on /dev/vmstore/shared)		7TB		Volume group for shared storage volumes
/dev/vmstore/local/admin	LVM LV	ext4	100GB	/mnt/vmdata-local/admin	Administration storage for local virtual machines
/dev/vmstore/local/config	LVM LV	ext4	1GB	/mnt/vmdata-local/config	Xen Config file storage for local virtual machines
/dev/vmstore/shared/admin	LVM LV	ext4	400GB	/mnt/vmdata-shared/admin	Administration storage for local virtual machines
/dev/vmstore/shared/config	LVM LV	ext4	1GB	/mnt/vmdata-shared/config	Xen Config file storage for local virtual machines

Local LV

/dev/vmstore/local also acts as a physical drive to back another Volume Group (/dev/vmstore-local) which actually contains the logical partitions for the virtual machines. This has no direct physical counterpart, it is placed on top of the logical volume /dev/vmstore/local. /dev/vmstore-local contains two logical volumes with ext4 filesystems: config (1GB, for xen config files) and admin (100GB, for misc. admin tools). The rest is allocated by Xen and should end up containing several LVM volumes for use by Virtual Machines. These can be mounted on Dom0 for maintinence.

Shared LV

/dev/vmstore/shared has a similar layout to it's local counterpart. It contains two logical volumes: config (1GB), and admin (500GB), as well as volumes for any VM's that have been created.

VM Resource Allocation

The CS server architecture is backed by 5 Xen Virtual Machines running Debian Wheezy. Naming conventions are definitely going to change in the near future. All MAC addresses must be in the 00:16:3e:xx:xx:xx range; this is a hardware range that has been reserved for Xen VM's

Name	HDD Space	RAM	Cores	MAC Address	IP Address	Services
dom0	240GB (SSD)	1GB	1	0c:c4:7a:07:cd:ab	159.28.230.117	Xen VM Management
vm0 (home)	3TB	16GB	4	00:16:3e:00:00:00	159.28.230.240	NFS (/clients), SSH
vm1 (net)	500GB	4GB	2	00:16:3e:00:00:01	159.28.230.241	DNS, DHCP, LDAP, CUPS
vm2 (web)	2.5TB	24GB	4	00:16:3e:00:00:02	159.28.230.242	Apache, Mailman, MySQL
vm3 (wiki)	500GB	8GB	2	00:16:3e:00:00:03	159.28.230.243	MediaWiki, Sage