kernel failed to re-read the partition table on /dev/xvda

Posted: August 16, 2014 in tech
Tags: , , ,

I really want to be able to resize partitions and file systems without having to reboot the virtual machine. It can be really difficult to get an outage on a production database system in our highly available environment. It works better in some virtualization hypervisors than others. It was working perfectly in CentOS virtual machines running in ESXi 5.1 but even that has had some issues recently, so perhaps there is more at play than just the Hypervisor.

We have started using the Oracle VM hypervisor for a small farm of Oracle database servers to save on licensing costs. This suite is Xen based but it has been extended (I believe) from the open-source software.

It’s usable. It’s not horrible. But as far as Virtualization goes it is not the best. But surprisingly cheap considering the name brand. That will probably be fixed after a while. 


 

The scenario is you need more disk space in the virtual machine. So using the OVM Manager you can grow the virtual disk image file. It’s an easy process in the GUI and should complete easily and without error.

At this point things usually go south because the system cannot make the kernel re-read the partition table, though I have found a possible way around the part of the problem. The key issue still remains, and here it is, the infamous error:

  • the kernel failed to re-read the partition table on /dev/xvda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.

Screen Shot 2014-08-16 at 9.32.29 AM

I’ve tried the usual suspects that can make the kernel re-read the partition table. These are partprobe, kpartx, partx, and ‘for i in `find /sys -name rescan |grep target`; do echo 1 > $i; done’. It doesn’t work under the Xen hypervisor. I think the partition table in question is borrowed from the hypervisor domain.

So what to do? Here’s my partial fix: I found that migrating the VM to another host in the Xen pool will at least show the increased virtual disk size, allowing the usual fdisk delete/recreate process to grow the partition.

But once the partition has been re-sized, even this does not work. I have migrated from host to host to host to host, and run the rescan on host and VM both. I looked at the kernel’s /proc/partitions file on the host and saw a “/dev/loop??” that seemed to correlate to the virtual disk image file in the repository.

Screen Shot 2014-08-16 at 9.50.03 AM

While migrating from host to host I looked at the host’s /proc/partitions file and determined the virtual disk is mounted as a loop device on the host:

Screen Shot 2014-08-16 at 10.14.28 AM

I’ve used kpartx, partprobe on this (/dev/loop2) but to no avail. The new partition 2 size does not get propagated to the kernel of the VM:

  • No change to size of physical volume /dev/xvda2.

Screen Shot 2014-08-16 at 10.25.55 AM

There has to be a way to get this pushed to the VM’s kernel. I just haven’t found it yet.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s