Oracle VM Repository Alarms

The Oracle VM platform for virtualization has some amazing gaps in functionality– see previous post about the disk size change problem for starters. But you want to know another big hole in functionality? Alarms! I would love to get alerts and alarms on certain conditions sent to my email or phone.

My embarrassing case in point: the other day a few thin-sized disks grew and filled up a repository. The virtual disks then started spewing SCSI errors. This could have been prevented if 1) the the disks were thick instead of thin, but even bettter would be option 2) some alarms about operating conditions such as repository utilization and performance utilization of the hosts.

So I this is a little script I wrote, you might want to sing it note for note, and be happy. I will eventually get some performance statistics put into my Teamquest environment so I can have performance graphs and alarms from that monitoring. But it’s been a while since I defined a User Table database in it…

In the meantime I have a simple repository utilization alarm script. You can download the script, below. It’s not gorgeous code but it gets the job done. You will need to get a few things working for it to work in your environment, namely you will need to get the Oracle VM 3.3 CLI running with SSH keys instead of password logins. The later version of OVM (starting with one of the 3.2 releases, I think) have added an administration port that can be accessed using SSH. I found the write-up here.

I have my cronjob running as root on the Oracle VM Manager host, the same one that is running the CLI port. After generating the SSH keys for root I skipped the ssh-add command the wiki above mentions, and simply ‘cat the public key file appending it to the oracle account authorized_keys file (eg, cat ~/.ssh/*.pub >>~oracle/.ssh/authorized_keys).

Next, as root, ssh to the admin port as the admin user. Answer yes to save the keys– if you have conflicts you should stop and clean it out of your .ssh/known_hosts file. Once you have authenticated it should authorize that newly added key. While you are in the CLI session you should test your functionality and compatibility of your Oracle VM Manager environment by issuing the command “list repository”. Your test should look something like this:

# ssh admin@localhost -p 10000
The authenticity of host '[localhost]:10000 ([127.0.0.1]:10000)' can't be established.
DSA key fingerprint is ya.da.ya.da.ya.da.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:10000,[127.0.0.1]:10000' (DSA) to the list of known hosts.
admin@localhost's password: 
OVM> list repository
Command: list repository
Status: Success
Time: 2015-02-11 15:16:43,417 CST
Data: 
 id:0004fb00000300005eexxxxxxx name:Repo1
 id:0004fb0000030000709xxxxxxx name:Repo2
 id:0004fb0000030000455xxxxxxx name:Repo3
OVM>

Go ahead and ‘exit’ after this has completed successfully. Does the output look similar?

Try the SSH connection again and if successful you should not have a password prompt nor a prompt to save the host key. In this second session, please copy the UUID from the earlier test and try the command below and using your UUID after the equal sign:

# ssh admin@localhost -p 10000
OVM> show repository id=
Command: show repository id=0004fb0000030000455xxxxxxx
Status: Success
Time: 2015-02-11 15:22:47,666 CST
Data: 
 File System = 0004fb0000030000455xxxxxxx [fs on NETAPP]
 Manager UUID = 0004fb00000100006edaaaaaaaaaaaaaa
 File System Free (GiB) = 503.32
 File System Total (GiB) = 2037.11
 File System Used (GiB) = 1533.79
 Used % = 75.0
 Refreshed = Yes
 Presented = Yes
 Presented Server 1 = 4c:4c:45:44:00:52:38:10:aa:aa:aa:aa:aa:aa [host1]
 Presented Server 2 = 4c:4c:45:44:00:52:38:10:bb:bb:bb:bb:bb:bb [host2]
 VirtualDisk 1 = blahblah.img [blahblah.img]
 VirtualDisk 2 = blahblhablha.img [blahblahblha.img]
 VirtualCdrom 1 = 0004fb00001500000b984b7f06a25c25.iso [OEL 6.iso]
 Vm 1 = 0004fb0000060000ecccccccccc [virtmach1]
 Vm 5 = 0004fb0000060000ddddddddddd [testme]
 Id = 0004fb0000030000455xxxxxxx [Repo3]
 Name = Repo3
 Locked = false
OVM>

Your output should hopefully look similar to the above info so go ahead and exit out of the CLI. You should have at the minimum the following keywords in four named fields (see underlined words) for my OVMrepomon.pl script to work:

• File System Free (GiB)
• File System Total (GiB)
• File System Used (GiB)
• Used %

If you have those four fields, let us proceed because your next step is prepare your Perl environment with the modules necessary to retrieve, process, format the data, and send the alarm.

Your Perl executable needs three free modules. You may be able to find some of these in the package repository of your operating system distribution, but that would be more complicated than I want to write for this post. So run your yum commands, or dpkg/apt-get, or whatever… but ultimately you will probably be running CPAN to install a few of these modules.

• MIME::Lite  — this sends the SMTP alarm message
• HTML::HashTable — this is an easy way to make an HTML table for the email
• Net::OpenSSH — this is an easy way to make the SSH connection to the OVMM CLI port

Once you have installed the modules above, download my code and create an executable script from it.

There are seven lines that need to configured for your environment and they are located near the top of the script. They are in one section called “Global Basics” and they control who gets the alarms, the OVM host, and your thresholds for the alarms.

• # Global Basics to tweak for your installation:
• my $OVMHOST=’admin@hostname:10000′; # should be in the notation user@host:port for the OVM CLI
• my $MAILTO=’your best friends email address’; # who receives the alerts?
• my $MAILFROM=’no-reply’; # who do the alerts come from?
• my $MAILHOST=”smtpforwarder”; # who can forward the email alerts?
• my $DEBUG=0;
• my $PCused=”87″; # percent used threshold
• my $FSfree=”200″; # gigs free of file system threshold

Run the command a few times, preferable with DEBUG set to 1 or higher and the thresholds set to level to insure a message can be sent. Make sure you are getting the email messages and then turn debug off. Now you are ready to put it in your cron scheduler. Make sure cron is able to execute and deliver a message to your inbox with debug off, and finally adjust the thresholds to what you truly want to be alerted at.