Table of Contents
Goal is to provide a storage device trough network, and be able to use it over the network. We will use LVM2 to create the storage device, gnbd to export mass storage, and clvm to create a clustered logical volume. You must use the kernel 2.6.12.
Architecture available for the test
3 nodes with 2x9.1G scsi hard drive
1 server
Tools needed already included in IGGI release:
http://sources.redhat.com/dm/
http://sources.redhat.com/lvm2/
http://sources.redhat.com/cluster/
cvs -d :pserver:[email protected]:/cvs/dm co device-mapper
cvs -d :pserver:[email protected]:/cvs/lvm2 co LVM2
cvs -d :pserver:[email protected]:/cvs/cluster co cluster
device-mapper: ftp://sources.redhat.com/pub/dm/
lvm2: ftp://sources.redhat.com/pub/lvm2/
Lvm2 should be built with the options: --with-clvmd --with-cluster=shared
ccs: Cluster Configuration System
cman: Cluster Manager, tools to manage the cluster
gfs: Global File System
dlm: Distributed Lock Manager
gulm: other kind of lock manager pour GFS
iddev: Identify device
magma: cluster/lock manager API abstraction library cluster
gnbd: GFS Network Block Device
fence: cluster I/O fencing system
Yo be able to run ccsd dameon, we need an /etc/cluster/cluster.conf file, wich describes the name for the cluster and a method to acces node. We will use the manual access method trough IP address.
<cluster name="clusterv" config_version="1">
<clusternodes>
<clusternode name="iggi.guibland.com">
<fence>
<method name="human">
<device name="last_resort" ipaddr="12.12.12.253"/>
</method>
</fence>
</clusternode>
<clusternode name="node1.guibland.com">
<fence>
<method name="human">
<device name="last_resort" ipaddr="12.12.12.1"/>
</method>
</fence>
</clusternode>
<clusternode name="node2.guibland.com">
<fence>
<method name="human">
<device name="last_resort" ipaddr="12.12.12.2"/>
</method>
</fence>
</clusternode>
<clusternode name="node3.guibland.com">
<fence>
<method name="human">
<device name="last_resort" ipaddr="12.12.12.3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevice name="last_resort" agent="fence_manual"/>
</cluster>
You must install cluster, dkms-cluster, clvm, lvm2 on server. Now load load modules cman dlm gnbd gfs dm-mod and start the ccsd service on the server. You can do that automatically with the setup_server_cluster.pl script. It probe all needed modules, create the configuration file and start ccsd.
[root@iggi ~]# urpmi clvm dkms-cluster cluster libxml2-devel To satisfy dependencies, the following packages are going to be installed: binutils-2.16.91.0.2-3.1.20060mdk.i586 cluster-1.01.00-3mdviggi.i586 clvmd-2.02.09-2mdviggi.i586 dkms-2.0.5.9-3.1.20060mdk.noarch dkms-cluster-1.01.00-3mdviggi.i586 dkms-minimal-2.0.5.9-3.1.20060mdk.noarch gcc-4.0.1-5mdk.i586 glibc-devel-2.3.6-4.1.20060mdk.i586 kernel-source-2.6.12.26mdk-1-1mdk.i586 ............ [root@iggi ~]# setup_server_cluster.pl gfs_srv - load cman kernel module - load gnbd kernel module - load gfs kernel module - load dlm kernel module - load dm-mod kernel module - service ccsd stop Stopping ccsd: [ OK ] - Generate /etc/cluster/cluster.conf - Backup of /etc/cluster/cluster.conf configuration - service ccsd start Starting ccsd: [ OK ]
CCSD is now started, we can join the cman, and check we are now a member.
[root@iggi ~]# cman_tool join [root@iggi ~]# cman_tool status Protocol version: 5.0.1 Config version: 1 Cluster name: clusterv Cluster ID: 26846 Cluster Member: Yes Membership state: Cluster-Member Nodes: 1 Expected_votes: 4 Total_votes: 1 Quorum: 3 Activity blocked Active subsystems: 0 Node name: iggi.guibland.com Node addresses: 12.12.12.253
Server has joined the mandriva cluster:
[root@iggi ~]# cman_tool nodes Node Votes Exp Sts Name 1 1 4 M iggi.guibland.com
Now install you can install clvmd dkms-cluster cluster kernel-2.6.12 on all nodes
[root@iggi ~]# urpmi --parallel cluster clvmd dkms-cluster cluster libxml2-devel kernel-2.6.12 copy of file /var/cache/urpmi/partial/parallel.cz To satisfy dependencies, the following packages are going to be installed: clvmd-2.02.09-2mdviggi.i586 libdevmapper1.02-1.02.09-1mdviggi.i586 .........
Copy the configuration file on all nodes:
[root@iggi ~]# mput2 $NKA -- /etc/cluster/cluster.conf /etc/cluster/cluster.conf
You can create the configuration automatically with the setup_client_cluster.pl script. It probe all needed modules. Once you have copied the /etc/cluster/cluster.conf file from you server, you can start ccsd on all nodes:
[root@iggi ~]# mput2 $NKA -- /etc/cluster/cluster.conf /etc/cluster/cluster.conf [root@iggi ~]# rshp2 -v $NKA -- setup_client_cluster.pl gfs_client <node1.guibland.com> [rank:1]: - load cman kernel module <node2.guibland.com> [rank:2]: - load cman kernel module <node3.guibland.com> [rank:3]: - load cman kernel module <node1.guibland.com> [rank:1]: - load gnbd kernel module <node2.guibland.com> [rank:2]: - load gnbd kernel module <node3.guibland.com> [rank:3]: - load gnbd kernel module <node1.guibland.com> [rank:1]: - load gfs kernel module <node1.guibland.com> [rank:1]: - load dlm kernel module <node1.guibland.com> [rank:1]: - service ccsd stop <node3.guibland.com> [rank:3]: - load gfs kernel module <node2.guibland.com> [rank:2]: - load gfs kernel module <node2.guibland.com> [rank:2]: - load dlm kernel module <node2.guibland.com> [rank:2]: - service ccsd stop <node1.guibland.com> [rank:1]:Stopping ccsd:[ OK ] <node3.guibland.com> [rank:3]: - load dlm kernel module <node3.guibland.com> [rank:3]: - service ccsd stop <node2.guibland.com> [rank:2]:Stopping ccsd:[ OK ] <node3.guibland.com> [rank:3]:Stopping ccsd:[ OK ] [root@iggi ~]# rshp2 -v $NKA -- service ccsd start <node1.guibland.com> [rank:1]:Starting ccsd:[ OK ] <node2.guibland.com> [rank:2]:Starting ccsd:[ OK ] <node3.guibland.com> [rank:3]:Starting ccsd:[ OK ] [root@iggi ~]# rshp2 -v $NKA -- ps axf | grep ccsd <node1.guibland.com> [rank:1]:10201 ? Ssl 0:00 ccsd clusterv <node2.guibland.com> [rank:2]: 2921 ? Ssl 0:00 ccsd clusterv <node3.guibland.com> [rank:3]: 2961 ? Ssl 0:00 ccsd clusterv [root@iggi ~]# rshp2 -v $NKA -- service cman start <node2.guibland.com> [rank:2]:Starting cman:[ OK ] <node1.guibland.com> [rank:1]:Starting cman:[ OK ] <node3.guibland.com> [rank:3]:Starting cman:[ OK ]
We need to join the fence domain on nodes and server.
[root@iggi ~]# fence_tool join [root@iggi ~]# rshp2 -v $NKA -- fence_tool join
We want CLVM service, so we need to start clvmd service on all nodes and server.
[root@iggi ~]# service clvmd start Starting clvmd: [ OK ] [root@iggi ~]# gexec -n 0 service clvmd start 0 Starting clvmd: [ OK ] 1 Starting clvmd: [ OK ] 2 Starting clvmd: [ OK ]
We want to export GNBD on nodes, so we need to start gnbd_serv service on all nodes.
[root@iggi ~]# gexec -n 0 gnbd_serv -v [root@iggi ~]# rshp2 -v $NKA -- ps axf | grep gnbd <node3.guibland.com> [rank:3]: 4820 ? S 0:00 /sbin/gnbd_serv -v <node1.guibland.com> [rank:1]:11033 ? S 0:00 /sbin/gnbd_serv -v <node2.guibland.com> [rank:2]: 4777 ? S 0:00 /sbin/gnbd_serv -v
We have done a lot of things, on server and node side. Let's diplasy what kind of service is available.
[root@iggi ~]# cman_tool status Protocol version: 5.0.1 Config version: 1 Cluster name: clusterv Cluster ID: 26846 Cluster Member: Yes Membership state: Cluster-Member Nodes: 4 Expected_votes: 4 Total_votes: 4 Quorum: 3 Active subsystems: 0 Node name: iggi.guibland.com Node addresses: 12.12.12.253 [root@iggi ~]# cman_tool nodes Node Votes Exp Sts Name 1 1 4 M iggi.guibland.com 2 1 4 M node2.guibland.com 3 1 4 M node3.guibland.com 4 1 4 M node1.guibland.com [root@iggi ~]# cman_tool services Service Name GID LID State Code Fence Domain: "default" 2 2 run - [1 2 3 4] DLM Lock Space: "clvmd" 3 3 run - [1 2 3 4]
Nodes and server are now ready, we need to export our storage devices. Gnbd tools will be used. To export the /dev/sdb device on node node1.guibland.com:
[root@iggi ~]# ssh n1 Last login: Wed Nov 8 15:41:51 2006 from iggi.guibland.com [root@node1 ~]# gnbd_export -c -v -e n1export -d /dev/sdb gnbd_export: created GNBD n1export serving file /dev/sdb
-c Reads from the exported GNBD will take advantage of the linux page cache. This option is used with -e.
-v Increase the verbosity of the output. This option is the most useful with -l. If it is used along with -l, an extended list of information on each exported device will be printed.
-e n1export Export a device as a GNBD with the Device name gnbdname. You must also specify the pathname of the device with the -d option. Once a GNBD has been exported, clients can import it with gnbd_import export the specified GNBD
-d /dev/sdb Specify the device to export as a GNBD. This option is used with -e. pathname may be either a block device or a regular file. Usually block devices are used, because this increases GNBD performance. the device to create a GNBD on
Now we can check that node1 export a GNBD:
[root@node1 ~]# gnbd_import -e n1 n1export
We can check that the disk /dev/sdb is currently exported
[root@node1 ~]# gnbd_export
Server[1] : n1export
--------------------------
file : /dev/sdb
sectors : 17930694
readonly : no
cached : yes
timeout : no
To mount an exported gnbd devices
[root@iggi ~]# gnbd_import -v -i n1 gnbd_import: incorrect mode for /dev/gnbd_ctl. removing gnbd_import: creating /dev/gnbd_ctl gnbd_import: created directory /dev/gnbd gnbd_import: created gnbd device n1export gnbd_recvd: gnbd_recvd started [root@iggi ~]# gnbd_import -v -i n2 gnbd_import: created gnbd device n2export gnbd_recvd: gnbd_recvd started [root@iggi ~]# gnbd_import -v -i n3 gnbd_import: created gnbd device n3export gnbd_recvd: gnbd_recvd started
To display the created devices
[root@iggi ~]# ls /dev/gnbd/* /dev/gnbd/n1export /dev/gnbd/n2export /dev/gnbd/n3export
To List imported GNBD
[root@iggi ~]# gnbd_import
Device name : n1export
----------------------
Minor # : 0
Proc name : /dev/gnbd0
Server : n1
Port : 14567
State : Close Connected Clear
Readonly : No
Sectors : 17930694
Device name : n2export
----------------------
Minor # : 1
Proc name : /dev/gnbd1
Server : n2
Port : 14567
State : Close Connected Clear
Readonly : No
Sectors : 17930694
Device name : n3export
----------------------
Minor # : 2
Proc name : /dev/gnbd2
Server : n3
Port : 14567
State : Close Connected Clear
Readonly : No
Sectors : 17930694
Create a physical volume for each exported GNBD
[root@iggi ~]# pvcreate /dev/gnbd0 /dev/gnbd1 /dev/gnbd2 /dev/cdrom0: open failed: Read-only file system Attempt to close device '/dev/cdrom0' which is not open. Physical volume "/dev/gnbd0" successfully created Physical volume "/dev/gnbd1" successfully created Physical volume "/dev/gnbd2" successfully created
To display the Physicxal volume use pvdisplay
[root@iggi ~]# pvdisplay --- NEW Physical volume --- PV Name /dev/gnbd0 VG Name PV Size 8.55 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID tGa4v3-0Pw7-QNna-ePtI-g5YX-PQ8R-R31o9K --- NEW Physical volume --- PV Name /dev/gnbd1 VG Name PV Size 8.55 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID Y10Q3r-jGTj-rZzW-KUU0-0YJg-2eRH-YlzOZ8 --- NEW Physical volume --- PV Name /dev/gnbd2 VG Name PV Size 8.55 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID 3cJToH-32jL-zS9O-zXGn-DTaE-pUuD-qqtS2r
Create the volume group
[root@iggi ~]# vgcreate cluster_vg /dev/gnbd0 /dev/gnbd1 /dev/gnbd2
Volume group "cluster_vg" successfully created
Display the volume group
[root@iggi ~]# vgdisplay
--- Volume group ---
VG Name cluster_vg
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 3
Act PV 3
VG Size 25.64 GB
PE Size 4.00 MB
Total PE 6564
Alloc PE / Size 0 / 0
Free PE / Size 6564 / 25.64 GB
VG UUID RtAe1x-0jjT-Z7Jp-rLZ0-zq33-aTh3-0EntEW
Now we have to create the CLVM logical volume in an existing volume group (cluster_vg)
[root@iggi ~]# lvcreate -L 25.64G cluster_vg
/dev/cdrom0: open failed: Read-only file system
Rounding up size to full physical extent 25.64 GB
Logical volume "lvol0" created
Display attributes of a logical volume
[root@iggi ~]# lvdisplay --- Logical volume --- LV Name /dev/cluster_vg/lvol0 VG Name cluster_vg LV UUID 9x5q0A-Po14-ffge-Ll5f-68pl-yaMb-fqfONT LV Write Access read/write LV Status available # open 0 LV Size 25.64 GB Current LE 6564 Segments 3 Allocation inherit Read ahead sectors 0 Block device 250:0
Now we can format the device in GFS, and use DLM on the clusterv cluster.
[root@iggi ~]# gfs_mkfs -p lock_dlm -t clusterv:data -j 6 /dev/cluster_vg/lvol0 This will destroy any data on /dev/cluster_vg/lvol0. It appears to contain a GFS filesystem. Are you sure you want to proceed? [y/n] y Device: /dev/cluster_vg/lvol0 Blocksize: 4096 Filesystem Size: 6524260 Journals: 6 Resource Groups: 100 Locking Protocol: lock_dlm Lock Table: clusterv:data Syncing... All Done
-p lock_dlm LockProtoName is the name of the locking protocol to use. Acceptable locking protocols are lock_dlm, lock_gulm.
-j 6 The number of journals for gfs_mkfs to create. You need at least one journal per machine that will mount the filesystem.
-t iggi:data The lock table field appropriate to the lock module you're using. It is clustername:fsname. Clustername must match that in cluster.conf; only members of this cluster are permitted to use this file system. Fsname is a unique file system name used to distinguish this GFS file system from others created (1 to 16 characters).
All is ok and ready ! We can now mount the device :)
[root@iggi ~]# mkdir /mnt/test_vg
[root@iggi ~]# mount -t gfs /dev/cluster_vg/lvol0 /mnt/test_vg
[root@iggi ~]# df
Filesystem Size Used Avail Use% Mounted on
/dev/hdc6 13G 1.9G 11G 16% /
/dev/hdc2 49G 4.7G 45G 10% /mnt/windows
/dev/hda 1.5G 1.5G 0 100% /mnt/cdrom
/dev/mapper/cluster_vg-lvol0
25G 32K 25G 1% /mnt/test_vg
[root@iggi ~]# mount | grep gfs
/dev/mapper/cluster_vg-lvol0 on /mnt/test_vg type gfs (rw)
Now we can copy data on it, and it works :)
[root@iggi ~]# cp -f /var/install/cluster/media/main/ke* /mnt/test_vg/ [root@iggi ~]# ls /mnt/test_vg/* /mnt/test_vg/kernel-2.6.12.26mdk-1-1mdk.i586.rpm /mnt/test_vg/kernel-xen0-2.6.17.5mdv-1-1mdv.i586.rpm /mnt/test_vg/kernel-2.6.17.5mdv-1-1mdv.i586.rpm /mnt/test_vg/kernel-xenU-2.6.17.5mdv-1-1mdv.i586.rpm /mnt/test_vg/kernel-smp-2.6.12.26mdk-1-1mdk.i586.rpm /mnt/test_vg/keychain-2.6.6-1mdviggi.noarch.rpm /mnt/test_vg/kernel-source-2.6.12.26mdk-1-1mdk.i586.rpm
New dlm and gfs services are now available on cluster
[root@iggi ~]# cman_tool services Service Name GID LID State Code Fence Domain: "default" 2 2 run - [1 2 3 4] DLM Lock Space: "clvmd" 3 3 run - [1 2 3 4] DLM Lock Space: "data" 4 4 run - [1] GFS Mount Group: "data" 5 5 run - [1]
lvdisplay lvremove cluster_vg vgdisplay vgremove cluster_vg pvdisplay pvremove /dev/gnbd2 /dev/gnbd1 /dev/gnbd0 gnbd_import -R [root@iggi ~]# rshpn gnbd_export -R gnbd_export: removed GNBD n1export gnbd_export: removed GNBD n2export gnbd_export: removed GNBD n3export [root@iggi ~]# rshp2 -v $NKA -- service clvmd stop <node1.guibland.com> [rank:1]:Stopping clvmd: [ OK ] <node2.guibland.com> [rank:2]:Stopping clvmd: [ OK ] <node3.guibland.com> [rank:3]:Stopping clvmd: [ OK ] [root@iggi ~]# rshp2 -v $NKA -- killall /sbin/gnbd_serv [root@iggi ~]# rshp2 -v $NKA -- service ccsd stop <node3.guibland.com> [rank:3]:Stopping ccsd:[ OK ] <node2.guibland.com> [rank:2]:Stopping ccsd:[ OK ] <node1.guibland.com> [rank:1]:Stopping ccsd:[ OK ]