Rent Configuration in libvirt and SanLock

Keywords: xml CentOS vnc

In previous tests, sanlock's leases were saved in NFS (disk_lease_dir="/var/lib/libvirt/sanlock"), and libvirt's automatic lease management function (auto_disk_leases=1) was used. Here we will try some new configuration methods.

Store leases directly in shared storage (automatic lease management)

There is no way to store leases directly in shared storage in libvirt and sanlock-related configurations and documents. The "disk_lease_dir" variable of lease location can be configured only when automatic lease management is used. I wonder if the lease can be automatically saved in shared storage by modifying this configuration to share storage. If you can, there must be relevant judgment statements in the source code, so go in and explore.

Analysis source code

Analyse libvirt's analysis of the "disk_lease_dir" variable.

  • Look at the source code for reference to the "disk_lease_dir" variable:
$ tar xvf libvirt-1.3.4.tar.gz
$ cd libvirt-1.3.4/

$ grep -rHn disk_lease_dir
src/locking/test_libvirt_sanlock.aug.in:6:{ "disk_lease_dir" = "/var/lib/libvirt/sanlock" }
src/locking/lock_driver_sanlock.c:134:    p = virConfGetValue(conf, "disk_lease_dir");
src/locking/lock_driver_sanlock.c:135:    CHECK_TYPE("disk_lease_dir", VIR_CONF_STRING);
src/locking/libvirt_sanlock.aug:20:   let entry = str_entry "disk_lease_dir"
src/locking/sanlock.conf:34:#disk_lease_dir = "/var/lib/libvirt/sanlock"
src/locking/sanlock.conf:40:# at 'disk_lease_dir' *MUST* be given a different host ID.
src/locking/sanlock.conf:66:# content of disk_lease_dir) to make sure sanlock daemon can
ChangeLog:80123:	$ augtool get /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir
ChangeLog:80124:	/files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir = /var/lib/libvirt/sanlock
tools/virt-sanlock-cleanup.in:29:LOCKDIR=`augtool get '/files@sysconfdir@/libvirt/qemu-sanlock.conf/disk_lease_dir'`
  • It can be inferred from experience that the location of the real parsing "disk_lease_dir" variable may be in lock_driver_sanlock.c. Go in and check:
$ vi src/locking/lock_driver_sanlock.c
  • It can be inferred preliminarily from the statement "S_ISDIR(st.st_mode)" that the "disk_lease_dir" variable must be a directory, otherwise it will report an error directly.
...

static int virLockManagerSanlockLoadConfig(const char *configFile)
{
    virConfPtr conf;
    virConfValuePtr p;
    char *tmp;

...

    p = virConfGetValue(conf, "auto_disk_leases");
    CHECK_TYPE("auto_disk_leases", VIR_CONF_ULONG);
    if (p) driver->autoDiskLease = p->l;

    p = virConfGetValue(conf, "disk_lease_dir");
    CHECK_TYPE("disk_lease_dir", VIR_CONF_STRING);
    if (p && p->str) {
        VIR_FREE(driver->autoDiskLeasePath);
        if (VIR_STRDUP(driver->autoDiskLeasePath, p->str) < 0) {
            virConfFree(conf);
            return -1;
        }
    }

...

static int virLockManagerSanlockSetupLockspace(void)
{
    int fd = -1;
    struct stat st;
    int rv;
    struct sanlk_lockspace ls;
    char *path = NULL;
    char *dir = NULL;
    int retries = LOCKSPACE_RETRIES;

    if (virAsprintf(&path, "%s/%s",
                    driver->autoDiskLeasePath,
                    VIR_LOCK_MANAGER_SANLOCK_AUTO_DISK_LOCKSPACE) < 0)
        goto error;

...

        if (stat(dir, &st) < 0 || !S_ISDIR(st.st_mode)) {
            virReportError(VIR_ERR_INTERNAL_ERROR,
                           _("Unable to create lockspace %s: parent directory"
                             " does not exist or is not a directory"),
                           path);
            goto error;
        }

...

Conclusion:

From the simple source code analysis, if the "disk_lease_dir" variable is not a directory, it can not be used, there is no automatic distinction between "shared storage" or "NFS" statements.

Practical testing

  • Create a volume on shared storage to save the lease.

Select a node to create a volume:

$ lvcreate -n sanlock -L 5G storage

Activate the volume at another node:

$ pvscan --cache
$ lvchange -ay storage/sanlock
  • Set it on this volume to automatically manage the lease:
$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/auto_disk_leases 1
$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir "/dev/storage/sanlock"
  • Restart the libvirt service:
$ systemctl restart libvirtd
$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Five 2017-04-21 18:45:56 CST; 1s ago
     Docs: man:libvirtd(8)
           http://libvirt.org
  Process: 59501 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 59501 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/libvirtd.service
           ├─12993 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           └─12994 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

4 Month 2118:45:56 Test2 systemd[1]: Starting Virtualization daemon...
4 Month 2118:45:56 Test2 systemd[1]: Started Virtualization daemon.
4 Month 2118:45:56 Test2 dnsmasq[12993]: read /etc/hosts - 6 addresses
4 Month 2118:45:56 Test2 dnsmasq[12993]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
4 Month 2118:45:56 Test2 dnsmasq-dhcp[12993]: read /var/lib/libvirt/dnsmasq/default.hostsfile
4 Month 2118:45:56 Test2 libvirtd[59501]: libvirt version: 2.0.0, package: 10.el7_3.5 (CentOS BuildSystem <http://bugs.centos.org>, 2017-03-0...tos.org)
4 Month 2118:45:56 Test2 libvirtd[59501]: hostname: test2
4 Month 2118:45:56 Test2 libvirtd[59501]: internal error: Unable to generate lock space /dev/storage/sanlock/__LIBVIRT__DISKS__: Superior directory does not exist or is not a directory
4 Month 2118:45:56 Test2 libvirtd[59501]: QEMU Initialization state driver failed: internal error: Unable to generate lock space /dev/storage/sanlock/__LIBVIRT__DISKS__...Is a catalog
4 Month 2118:45:56 Test2 libvirtd[59501]: Driver state initialization failed

This means that the "disk_lease_dir" variable cannot be set as a volume to automatically manage leases directly on shared storage, which is consistent with previous source code analysis results.

Manual lease management

Configure libvirt

  • Disable automatic lease management:
$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/auto_disk_leases 0
  • Delete the automatic lease management directory configuration:
$ augtool -s rm /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir
  • Allow disks to have no lease configurations:
$ augtool -s set /files/etc/libvirt/qemu-sanlock.conf/require_lease_for_disks 0
  • Restart the libvirt service:
$ systemctl restart libvirtd
$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Five 2017-04-21 19:00:03 CST; 5s ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 60745 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─12993 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ├─12994 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           └─60745 /usr/sbin/libvirtd

4 Month 2119:00:03 Test2 systemd[1]: Starting Virtualization daemon...
4 Month 2119:00:03 Test2 systemd[1]: Started Virtualization daemon.
4 Month 2119:00:03 Test2 dnsmasq[12993]: read /etc/hosts - 6 addresses
4 Month 2119:00:03 Test2 dnsmasq[12993]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
4 Month 2119:00:03 Test2 dnsmasq-dhcp[12993]: read /var/lib/libvirt/dnsmasq/default.hostsfile

Create a lease file (at Test1 or Test2 nodes)

  • Prepare to create a lease on shared storage/dev/mapper/ray and modify its permissions:
$ chown sanlock:sanlock /dev/mapper/raw
  • Create Lockspace with the name "libvirt":
$ sanlock direct init -s libvirt:0:/dev/mapper/raw:0
init done 0
  • Create a resource named "test1" that belongs to Lockspace named libvirt:
$ sanlock direct init -r libvirt:test1:/dev/mapper/raw:1048576

Join Lockspace (Test1 and Test2 nodes)

Two nodes are added to Lockspace using different Host ID s (they need to be re-added after each boot).

  • Test1 node:
$ sanlock client add_lockspace -s libvirt:1:/dev/mapper/raw:0
add_lockspace
add_lockspace done 0

$ sanlock client status
daemon 581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1
p -1 helper
p -1 listener
p -1 status
s libvirt:1:/dev/mapper/raw:0
s __LIBVIRT__DISKS__:1:/var/lib/libvirt/sanlock/__LIBVIRT__DISKS__:0
  • Test2 node:
$ sanlock client add_lockspace -s libvirt:2:/dev/mapper/raw:0
add_lockspace
add_lockspace done 0

$ sanlock client status
daemon b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2
p -1 helper
p -1 listener
p -1 status
s libvirt:2:/dev/mapper/raw:0
s __LIBVIRT__DISKS__:2:/var/lib/libvirt/sanlock/__LIBVIRT__DISKS__:0
  • View the created lease:
$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000038190 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000037427 0002 0001
01048576                              libvirt                                            test1 0000000000 0000 0000 0

Create virtual machine disks (Test1 and Test2 nodes)

Create a volume on shared storage as a disk.

  • Select a node to create and initialize the volume:
$ lvcreate -n test4_sanlock -L 5G storage
$ dd if=cirros-0.3.4-x86_64-disk.img of=/dev/storage/test4_sanlock
  • Activate the volume at another node:
$ pvscan --cache
$ lvchange -ay storage/test4_sanlock

Create a new virtual machine (Test1 and Test2 nodes)

  • Configure virtual machines:
$ vi test4_sanlock.xml

<domain type='kvm'>
    <name>test4_sanlock</name>
    <memory>262144</memory>
    <vcpu>1</vcpu>

    <os>
        <type arch='x86_64' machine='pc'>hvm</type>
        <boot dev='hd'/>
    </os>

    <devices>
        <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/dev/storage/test4_sanlock'/>
            <target dev='hda' bus='ide'/>
        </disk>
        
        <lease>
            <lockspace>libvirt</lockspace>
            <key>test1</key>
            <target path='/dev/mapper/raw' offset='1048576'/>
        </lease>
        
        <input type='tablet' bus='usb'/>
        <input type='mouse' bus='ps2'/>

        <graphics type='vnc' port='-1' listen = '0.0.0.0' autoport='yes' keymap='en-us'/>
    </devices>
</domain>
  • Define virtual machines:
$ virsh define test4_sanlock.xml
 Definition field test4_sanlock (from test4_sanlock.xml)

$ virsh list --all
 Id Name Status
----------------------------------------------------
 - Tes1_sanlock Closed
 - Tes2_sanlock Closed
 - Tes3_sanlock Closed
 - Tes4_sanlock Closed
 - test_sanlock Closed

Start the virtual machine (Test1 and Test2 nodes)

  1. Start the virtual machine test4_sanlock on the Test1 node:
$ virsh start test4_sanlock
//The domain test4_sanlock has started

$ virsh list --all
 Id    Name                         state
----------------------------------------------------
 2     test4_sanlock                  running
 -     test1_sanlock                  Close
 -     test2_sanlock                  Close
 -     test3_sanlock                  Close
 -     test_sanlock                   Close

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040118 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039355 0002 0001
01048576                              libvirt                                            test1 0000040075 0001 0001 1

The resource test1 owner belongs to Host ID 1, or Test1 node.

  1. Start the virtual machine test4_sanlock on the Test2 node:
$ virsh start test4_sanlock
//Error: Start field test4_sanlock failed
//Error: resource busy: Request lock failed: Error - 243

$ virsh list --all
 Id    Name                         state
----------------------------------------------------
 -     test1_sanlock                  Close
 -     test2_sanlock                  Close
 -     test3_sanlock                  Close
 -     test4_sanlock                  Close
 -     test_sanlock                   Close

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040200 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039437 0002 0001
01048576                              libvirt                                            test1 0000040075 0001 0001 1

$ sanlock direct dump /dev/mapper/raw
2017-04-21 20:02:54+0800 39406 [24937]: s2:r3 resource libvirt:test1:/dev/mapper/raw:1048576 for 2,10,65662
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire begin 2 0 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire leader 1 owner 1 1 40075 max mbal[0] 1 our_dblock 0 0 0 0 0 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire owner 1 1 40075 host_status 1 1 40139 wait_start 39396
2017-04-21 20:02:54+0800 39406 [24937]: r3 paxos_acquire owner 1 delta 1 1 40159 alive
2017-04-21 20:02:54+0800 39406 [24937]: r3 acquire_disk rv -243 lver 1 at 40075
2017-04-21 20:02:54+0800 39406 [24937]: r3 acquire_token held error -243
2017-04-21 20:02:54+0800 39406 [24937]: r3 release_token r_flags 0 lver 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 release_token done r_flags 0
2017-04-21 20:02:54+0800 39406 [24937]: r3 cmd_acquire 2,10,65662 acquire_token -243 lease owned by other host
2017-04-21 20:02:54+0800 39406 [24937]: cmd_acquire 2,10,65662 result -243 pid_dead 0
2017-04-21 20:02:54+0800 39406 [24933]: client_pid_dead 2,10,65662 cmd_active 0 suspend 0
2017-04-21 20:05:43+0800 39573 [24933]: cmd_register ci 2 fd 10 pid 65914
2017-04-21 20:05:43+0800 39573 [24933]: cmd_restrict ci 2 fd 10 pid 65914 flags 1
2017-04-21 20:05:43+0800 39574 [24938]: cmd_inquire 2,10,65914 ci_in 3 fd 15
2017-04-21 20:05:43+0800 39574 [24938]: cmd_inquire 2,10,65914 result 0 pid_dead 0 res_count 0 cat_count 0 strlen 0
2017-04-21 20:05:43+0800 39574 [24937]: cmd_acquire 2,10,65914 ci_in 4 fd 16 count 1 flags 0

The startup failed and the owner of test1 could not be obtained.

  1. Close the virtual machine test4_sanlock on the Test1 node:
$ killall -9 qemu-kvm

$ virsh list --all
 Id    Name                         state
----------------------------------------------------
 -     test1_sanlock                  Close
 -     test2_sanlock                  Close
 -     test3_sanlock                  Close
 -     test4_sanlock                  Close
 -     test_sanlock                   Close

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040262 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039498 0002 0001
01048576                              libvirt                                            test1 0000000000 0001 0001 1

When the virtual machine is shut down, the timestamp of resource test1 is changed to 0.

  1. Start the virtual machine test4_sanlock on the Test2 node:
$ virsh start test4_sanlock
//The domain test4_sanlock has started

$ virsh list --all
 Id    Name                         state
----------------------------------------------------
 2     test4_sanlock                  running
 -     test1_sanlock                  Close
 -     test2_sanlock                  Close
 -     test3_sanlock                  Close
 -     test_sanlock                   Close

$ sanlock direct dump /dev/mapper/raw
  offset                            lockspace                                         resource  timestamp  own  gen lver
00000000                              libvirt       581e732d-e4b1-4216-8b2f-1a63f08bb28d.Test1 0000040344 0001 0001
00000512                              libvirt       b888aabb-0f5f-45ce-a310-ff8021f514fd.Test2 0000039580 0002 0001
01048576                              libvirt                                            test1 0000039574 0002 0001 2

Start successfully, the owner of resource test1 becomes Host ID 2, or Test2 node.

summary

  • With automatic lease management, only NFS can be used to save lease files. The granularity of the lock is "disk". All virtual machines must use or not use sanlock. The lease files can only be stored in one place.
  • Using manual lease management, lease files can be saved using shared storage or NFS. The granularity is "virtual machine". Each virtual machine can choose whether to use sanlock or not according to its needs. The lease files can be saved separately or centrally according to their needs.

Posted by laanes on Thu, 23 May 2019 11:19:04 -0700

Hot Keywords

©2024 Programmer Group