Oracle VM Repository Alarms

Posted: February 11, 2015 in Oracle VM, perl, tech
Tags: , ,

The Oracle VM platform for virtualization has some amazing gaps in functionality– see previous post about the disk size change problem for starters. But you want to know another big hole in functionality? Alarms! I would love to get alerts and alarms on certain conditions sent to my email or phone.

My embarrassing case in point: the other day a few thin-sized disks grew and filled up a repository. The virtual disks then started spewing SCSI errors. This could have been prevented if 1) the the disks were thick instead of thin, but even bettter would be option 2) some alarms about operating conditions such as repository utilization and performance utilization of the hosts.

So I this is a little script I wrote, you might want to sing it note for note, and be happy. I will eventually get some performance statistics put into my Teamquest environment so I can have performance graphs and alarms from that monitoring. But it’s been a while since I defined a User Table database in it…

In the meantime I have a simple repository utilization alarm script. You can download the script, below. It’s not gorgeous code but it gets the job done. You will need to get a few things working for it to work in your environment, namely you will need to get the Oracle VM 3.3 CLI running with SSH keys instead of password logins. The later version of OVM (starting with one of the 3.2 releases, I think) have added an administration port that can be accessed using SSH. I found the write-up here.

I have my cronjob running as root on the Oracle VM Manager host, the same one that is running the CLI port. After generating the SSH keys for root I skipped the ssh-add command the wiki above mentions, and simply ‘cat the public key file appending it to the oracle account authorized_keys file (eg, cat ~/.ssh/*.pub >>~oracle/.ssh/authorized_keys).

Next, as root, ssh to the admin port as the admin user. Answer yes to save the keys– if you have conflicts you should stop and clean it out of your .ssh/known_hosts file. Once you have authenticated it should authorize that newly added key. While you are in the CLI session you should test your functionality and compatibility of your Oracle VM Manager environment by issuing the command “list repository”. Your test should look something like this:

# ssh admin@localhost -p 10000
The authenticity of host '[localhost]:10000 ([]:10000)' can't be established.
DSA key fingerprint is ya.da.ya.da.ya.da.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:10000,[]:10000' (DSA) to the list of known hosts.
admin@localhost's password: 
OVM> list repository
Command: list repository
Status: Success
Time: 2015-02-11 15:16:43,417 CST
 id:0004fb00000300005eexxxxxxx name:Repo1
 id:0004fb0000030000709xxxxxxx name:Repo2
 id:0004fb0000030000455xxxxxxx name:Repo3

Go ahead and ‘exit’ after this has completed successfully. Does the output look similar?

Try the SSH connection again and if successful you should not have a password prompt nor a prompt to save the host key. In this second session, please copy the UUID from the earlier test and try the command below and using your UUID after the equal sign:

# ssh admin@localhost -p 10000
OVM> show repository id=
Command: show repository id=0004fb0000030000455xxxxxxx
Status: Success
Time: 2015-02-11 15:22:47,666 CST
 File System = 0004fb0000030000455xxxxxxx [fs on NETAPP]
 Manager UUID = 0004fb00000100006edaaaaaaaaaaaaaa
 File System Free (GiB) = 503.32
 File System Total (GiB) = 2037.11
 File System Used (GiB) = 1533.79
 Used % = 75.0
 Refreshed = Yes
 Presented = Yes
 Presented Server 1 = 4c:4c:45:44:00:52:38:10:aa:aa:aa:aa:aa:aa [host1]
 Presented Server 2 = 4c:4c:45:44:00:52:38:10:bb:bb:bb:bb:bb:bb [host2]
 VirtualDisk 1 = blahblah.img [blahblah.img]
 VirtualDisk 2 = blahblhablha.img [blahblahblha.img]
 VirtualCdrom 1 = 0004fb00001500000b984b7f06a25c25.iso [OEL 6.iso]
 Vm 1 = 0004fb0000060000ecccccccccc [virtmach1]
 Vm 5 = 0004fb0000060000ddddddddddd [testme]
 Id = 0004fb0000030000455xxxxxxx [Repo3]
 Name = Repo3
 Locked = false

Your output should hopefully look similar to the above info so go ahead and exit out of the CLI. You should have at the minimum the following keywords in four named fields (see underlined words) for my script to work:

  • File System Free (GiB)
  • File System Total (GiB)
  • File System Used (GiB)
  • Used %

If you have those four fields, let us proceed because your next step is prepare your Perl environment with the modules necessary to retrieve, process, format the data, and send the alarm.

Your Perl executable needs three free modules. You may be able to find some of these in the package repository of your operating system distribution, but that would be more complicated than I want to write for this post. So run your yum commands, or dpkg/apt-get, or whatever… but ultimately you will probably be running CPAN to install a few of these modules.

  • MIME::Lite  — this sends the SMTP alarm message
  • HTML::HashTable — this is an easy way to make an HTML table for the email
  • Net::OpenSSH — this is an easy way to make the SSH connection to the OVMM CLI port

Once you have installed the modules above, download my code and create an executable script from it.

There are seven lines that need to configured for your environment and they are located near the top of the script. They are in one section called “Global Basics” and they control who gets the alarms, the OVM host, and your thresholds for the alarms.

  • # Global Basics to tweak for your installation:
  • my $OVMHOST=’admin@hostname:10000′; # should be in the notation user@host:port for the OVM CLI
  • my $MAILTO=’your best friends email address’; # who receives the alerts?
  • my $MAILFROM=’no-reply’; # who do the alerts come from?
  • my $MAILHOST=”smtpforwarder”; # who can forward the email alerts?
  • my $DEBUG=0;
  • my $PCused=”87″; # percent used threshold
  • my $FSfree=”200″; # gigs free of file system threshold

Run the command a few times, preferable with DEBUG set to 1 or higher and the thresholds set to level to insure a message can be sent. Make sure you are getting the email messages and then turn debug off. Now you are ready to put it in your cron scheduler. Make sure cron is able to execute and deliver a message to your inbox with debug off, and finally adjust the thresholds to what you truly want to be alerted at.

I really want to be able to resize partitions and file systems without having to reboot the virtual machine. It can be really difficult to get an outage on a production database system in our highly available environment. It works better in some virtualization hypervisors than others. It was working perfectly in CentOS virtual machines running in ESXi 5.1 but even that has had some issues recently, so perhaps there is more at play than just the Hypervisor.

We have started using the Oracle VM hypervisor for a small farm of Oracle database servers to save on licensing costs. This suite is Xen based but it has been extended (I believe) from the open-source software.

It’s usable. It’s not horrible. But as far as Virtualization goes it is not the best. But surprisingly cheap considering the name brand. That will probably be fixed after a while. 


The scenario is you need more disk space in the virtual machine. So using the OVM Manager you can grow the virtual disk image file. It’s an easy process in the GUI and should complete easily and without error.

At this point things usually go south because the system cannot make the kernel re-read the partition table, though I have found a possible way around the part of the problem. The key issue still remains, and here it is, the infamous error:

  • the kernel failed to re-read the partition table on /dev/xvda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.

Screen Shot 2014-08-16 at 9.32.29 AM

I’ve tried the usual suspects that can make the kernel re-read the partition table. These are partprobe, kpartx, partx, and ‘for i in `find /sys -name rescan |grep target`; do echo 1 > $i; done’. It doesn’t work under the Xen hypervisor. I think the partition table in question is borrowed from the hypervisor domain.

So what to do? Here’s my partial fix: I found that migrating the VM to another host in the Xen pool will at least show the increased virtual disk size, allowing the usual fdisk delete/recreate process to grow the partition.

But once the partition has been re-sized, even this does not work. I have migrated from host to host to host to host, and run the rescan on host and VM both. I looked at the kernel’s /proc/partitions file on the host and saw a “/dev/loop??” that seemed to correlate to the virtual disk image file in the repository.

Screen Shot 2014-08-16 at 9.50.03 AM

While migrating from host to host I looked at the host’s /proc/partitions file and determined the virtual disk is mounted as a loop device on the host:

Screen Shot 2014-08-16 at 10.14.28 AM

I’ve used kpartx, partprobe on this (/dev/loop2) but to no avail. The new partition 2 size does not get propagated to the kernel of the VM:

  • No change to size of physical volume /dev/xvda2.

Screen Shot 2014-08-16 at 10.25.55 AM

There has to be a way to get this pushed to the VM’s kernel. I just haven’t found it yet.

My previous two posts were about getting utilization statistics out of my Network Appliance filers into a Teamquest database for my IT Service Analyzer and Reporter charts. They are working great and I am using them in a production environment. The thing that bothered me about them is they seemed so slow. The volume stats report would take just over a second for four filer heads and the system stats script seemed to take FOREVER. I timed it. It was only five seconds for four filers but the feeling was still FOREVER.

timing old script

ptime of old volstats

timing the old systats

ptime of old systats

I knew what the problem was and I knew I would have to buckle down and learn SNMP even better, and especially learn Perl SNMP modules in order to tune it back my acceptable standards of runtime. That first script was a quick and dirty hack really, and like most hacks it is just functional. All the SNMP requests were running system commands that could be easily run and debugged from a command line. It’s a great way to learn and get something functional at the same time. But it’s like a baby eating from a bottle, it needs to grow up, eat solid food, go to school, and get a job to support itself. Or, in Perl terms, it needs to use pure Perl code to do the work instead of system commands.

So, enter version 2 of both scripts. My new volume stats script literally runs twice as fast as the old script. My new system stats, also quite literally, runs TEN times as fast. Woo hoo! How is that for tuning code and making things better?

timing new volstats

ptime of new volstats

timing new sys stats

ptime of new sys stats

These new versions run no system commands but do all work using the Net-SNMP Perl modules (not to be confused with Net::SNMP Perl modules). The process of learning the SNMP module took several days of trial and error around my other work. The biggest issue with Perl is the confusing amount of Perl modules available to do the same job. Often, a few google searches will reveal which module has the most support and I would choose that one. But in the case of the Perl SNMP modules there is no clear winner. Both have equal number of blogs and confused postings looking for help with the modules.

So I picked one. It was the wrong one initially, of course. I picked Net::SNMP to start with because it can be built using the CPAN shell (eg, ‘perl -MCPAN -e shell’). The other primary SNMP module being used is the one provided by the Net-SNMP command line packages. This can be more of a challenge to build, but more often than not it can just be installed as a package for your system, which is the easy route I chose. I used the OpenCSW package.

The reason I say that the Net::SNMP package was the wrong path is the challenge for an SNMP illiterate to understand SNMP and specialized MIBs. It appeared that you needed to know the confusingly long ID number of the statistic to use this module. I was (and am still) trying to learn about SNMP and could not figure out the proper way to find the statistics I wanted using this module. So I switched to the other package module which allowed me to use names for statistics that I was used to, like “df64AvailKBytes” to find the full and correct amount of Kilobytes available to a filesystem.

So I set off to learn the module. I started small with test scripts to just gather one or a few statistics. This allowed me to make some quick progress and learn how to address the desired statistics as a scalar, array, or hash, and to grow and process multiple statistics in relation to each other.

I ended up using the VarList method within the module. It allows the script to retrieve a bunch of statistics with a single connection. This is much more efficient than the old script which would make up to a dozen SNMP command requests to each filer head to get the desired statistics. This new method gets them once and then let me step through them one row at a time.

View/download my scripts here:

  1. new version 2 netapp volume stats script
  2. new version 2 netapp sys stats script

There is one thing that bothered me and I never figured out when I worked on the volume statistics script (the second one I tackled). When using the command line utilities the entire disk table can be requested using the name ‘dfTable’. This would not work using the Perl SNMP module even though ‘volTable’ and ‘ifTable’ would work. I do not understand the difference, but instead punted and again used the VarList method for named individual statistics with great success. If you know why, please make a comment. I wonder if I could shave a few tenths of a second off using dfTable… ;)

This is a follow on post to my previous article on getting the NetApp filer disk/volume/aggregate statistics charting using TeamQuest ITSAR (IT Service Analyzer and Reporter). So if are you interested in getting some other statistics on usage and utilization of your Network Appliance filers like the one below, read on.

NetApp systats Chart

Utilization statistics in ITSAR

This script and user table agent definition detail how to get the actual filer utilization such as CPU busy, network kilobytes in and out, and some other useful things for potential alerts. Potential alerts? Yes, some of the statistics that can be gathered using the SNMP agent are things like failed disks, failed power supplies, failed fans, the number of spare disks, and more. Simple peruse the Network Appliance SNMP MIB to see everything that is available to us. The table definition and my script can easily be extended before implementation to include the additional information you may be interested in.

Personally, I really trust the NetApp auto-support ability. Our NetApp filers are extremely capable of alerting us when a disk or anything fails. The filer heads are clustered and extremely redundant so I trust them (just not the devil inside, to quote a movie), so I might as well gather a few stats that I may track and alert on at a future time.

I won’t spend a lot of time covering the setup of SNMP on the filer or the TeamQuest host because that’s already done in the previous blog on the subject. Instead I will jump straight into the files and table setup for these new statistics.

The first step is to download the two additional files needed for the filer system statistics.

  1. The Network Appliance TeamQuest table definition for System statistics
  2. The Network Appliance Systats perl script

By now you have all the recommendations on hand and ready to go from my last blog… so save the files above to the same directory. Edit the script to configure the paths, username, password, and community string just like last time. Also make sure that the data directory has write privilege for the user that will be running the TeamQuest UTA which is usually daemon:root. Run it a few times to make sure it is working correctly, but take the time to make sure that the logfiles are writable by the user daemon after you are finished testing.

The script writes two files necessary for calculating the true network statistics. The SNMP statistic delivered is a number in bytes since the system last booted. I don’t think it needs to be stated, but this is not a very flexible statistic to work with for charting. It’s huge! And it gets humongous since the filers never, ever need to restart except for upgrades. I developed the script to use a log file to store the statistic from the last run and do a little math to give us a useful number for ongoing utilization. On execution the script operates like this in regards to the network statistics:

  1. Gets current network statistic
  2. Get last network statistic from log file
  3. Calculates difference
  4. converts to kilobytes
  5. saves current statistic (as read from filer) to the logfile

That’s it! It’s pretty easy to setup and run. The most difficult part of the setup was reading through all the many possible options for defining the statistics in the table definitions. I think I saved you a bit of work there – and in fact, some of the praise there goes to TeamQuest themselves. I was having issues with the way some of the statistics were being averaged and I opened a ticket with them. They were very patient with me and we got it resolved. Tickle me happy!

So import the table definition into your test or production database (“$manager/bin/tqtblprb -i -d testdatabase -f NetApp_sysStats.tbl”). And when that is done build your User Table Agent same as before but referencing the second script and the new table (USER:NetAppSysStats).

I may go ahead and setup some alerts on some of the statistics, there is more to be done!

NetApp all table data

NetApp systats

For as long as I have been using TeamQuest products I have wanted them to provide a solution for my Network Appliance brand filer devices. It was a desire that I could have have written a long time ago but frankly it was a low priority. I had a custom script that would run “df-Ah” on the filer, cut out the columns I wanted and write it to a CSV file that certain people could read in and make an Excel chart with. It was adequate so other higher priority items were worked and this languished for… a really long time. I now, finally, have something like the gauges chart below:

ITSAR chart of NetApp storage utilization

ITSAR chart of my NetApp appliances’ utilization

It finally happened because this last summer we migrated our Solaris web environment into zones. My previous job ran on the old systems and while I could have moved the stupid job I was holding out to force myself to get this written. So after a couple months of running it manually when the user needed the data I bit down and wrote the code necessary to get the capacity data into Teamquest. Basically I leveraged that inherent laziness in me to finally make myself get it done the proper way.

So, this blog documents my efforts to write a real solution to make a beautiful chart in my Teamquest IT Service Analyzer and Reporter where all the wild things go to get charted for management. I wanted more than just the storage utilization metrics we currently provide but that was the most important first step to accomplish and will be covered in this blog. A follow on blog should cover the  CPU, network, failed disk, failed power supply, and other interesting metrics that can be gathered, monitored, charted, and alerted on.

How to duplicate my results in your environment

The first item is get SNMP version 3 working on your filers under a limited user account on the filer. SNMP version 3 is necessary in today’s multi-terabyte world because the fields defined within SNMP version 1 and 2 by the MIB cannot account for the insane amount of “bytes” reported. Yes, it has to report in bytes. So be sure to download the Word Doc available at the NetApp community site and follow through step one. Yes, just the first step is all that is really needed, but don’t forget to set a password for the new user account who is allowed (and only allowed) to use SNMP.

Create a location for your scripts and data files. I like to put my scripts in /opt/teamquest/scripts, with a data directory underneath that. The Teamquest User Table Agent will run as the user ‘daemon’ and group root, so be sure to set appropriate permissions on the directory path and script for read and execution, and the data directory for write permission.

Make sure your system has snmp binaries — the Solaris ones are adequate and will probably be in /usr/sfw/bin if you installed the packages. The OpenCSW binaries are great, too. You will notice I am actually using the OpenCSW binaries at times but I have no good reason too– except that I typically like to have some base OpenCSW packages installed so that I have gtail, allowing me to tail multiple files at the same time.

Download the following files

  1. NetApp MIB for SNMP from NetApp
  2. My Script for NetApp Volume Statistics
  3. TeamQuest table definition for NetApp Volume

Drop the latest NetApp SNMP MIB into the data directory and copy my scripts to the script directory. Use “less” to look into the NetApp MIB and look at some of the options to get in there. There is a lot. I focused on the following values that I will use between this blog (volume statistics) and a future blog on system statistics:  dfTable, productVersion, productModel, productFirmwareVersion, cpuBusyTimePerCent, envFailedFanCount, envFailedPowerSupplyCount, nvramBatteryStatus, diskTotalCount, diskFailedCount, diskSpareCount, misc64NetRcvdBytes, and misc64NetSentBytes. If you see a “64” version of a statistic you will want to use that one to make sure that you are getting the real data figure out of the system.

Test your user and SNMP client with some command line operations before you start editing the script to run in your environment. A command would be like this ‘/opt/csw/bin/snmpget -v3 -n “” -u yourSNMPuser -l authNoPriv -A yourSNMPuserPassword -a Md5 -O qv -c yourcommunity -m /opt/teamquest/scripts/data/netapp.mib your-filername NETWORK-APPLIANCE-MIB::misc64NetRcvdBytes.0’. We will work on this statistic next time but today we are looking at the dfTable statistic for all the stats you want on your storage. So be sure to also test this different SNMP command: ‘ /usr/sfw/bin/snmptable -v3 -n “” -u yourSNMPuser -l authNoPriv -A yourSNMPuserPassword -a Md5 -c yourcommunity -m /opt/teamquest/scripts/data/netapp.mib yourfilername NETWORK-APPLIANCE-MIB::dfTable’ and marvel at the amount of data that comes across your terminal.

If all is successful with your command line tests then you are ready to edit the script and get it configured for your environment. You may be changing the path to the SNMP command and the MIB file, but you will definitely be changing the username, password, and community string. There are several other options to tweak too — do you want to import all volumes or just aggregates? Do you want to ignore snapshots? Test the script several times and make sure it is returning the data the way you want it. You will notice that you have to pass the filer names (comma separated, no spaces) in on the command line. This makes it easy to add and remove filers from your environment without adding or removing User Table Agents from your Teamquest manager, just simply edit the command line options passed to the script. Don’t forget to test with and without the -t=interval options for the TeamQuest format where the interval will match your desired frequency that the agent runs. And don’t worry about the extra options for snapshots or aggregates-only, this can be tweaked at any time to limit the data being absorbed by Teamquest and when you report or alert you can easily filter out what you don’t want.

When you are ready import the third file, the table, into your Teamquest database. You may want to use a test database for a while and then eventually add it to a production database. The command to import the table is “$manager/bin/tqtblprb -i -f NetApp_VolumeStats.tbl” but I heartily recommend you have the command line manual handy and consult it regularly for adding databases, tables, and deleting said items when things go wrong. IT happens.

Adding User Table Agent configuration

Adding User Table Agent configuration

When the table is entered into the database you are ready to add your very own User Table Agent. Connect to the TQ manager of the desired system using your browser. Set the database if you are not using the production database, and then click Collection Agents. On the far right you will see the link “Add Agent”, click that and then “User Table Instance”. Begin to enter the information that makes sense to you such as a name for this agent, the path to the executable, and the interval you want  collection to happen. The class and subclass must match exactly what is in the table file that was imported. It will be “USER” and “NetAppVolumes” unless you changed it. The Program arguments is where you pass in the comma separated list of filer names (no spaces!), a single space and -t=<interval>. Make sure to set that interval to equal what you have entered in below for the actual collection interval. After you save and apply the new settings you simply have to wait until the clock hits the time to match the next collection (every five minute increment of the hour if you are using the 300 second interval like I am).

Be sure to launch TQView and look at the table directly for accurate statistics, play with the filter options, etc. Tip: you can create a test database in ITSAR that only harvests from this test database so that you can test end to end.

table view of data

Using TQview to examine actual data gathered

You will notice that I dropped the actual FlexibleVolume volume type data from my gathering. It may be useful at some point in the future and it can be re-added with a simple edit to the script, but for this first stage all I care about is overall health of the filer and so my ITSAR chart for management is a simple global view of the filer cluster-pair. For this, I use the statistic “FilerCapacity” that the script calculates by summing all of the VolumeType “aggregate” on each filer node. You can see that I have a total of four nodes in my environment (names withheld to protect the innocent).

And that is it for the first stage! On to writing alerts and getting the system stats working.

Mars One Park

Posted: August 28, 2012 in fiction, mars

Neil sat on the park bench, chin and hands resting on the top of his cane. His sparse white hair waved in the breeze, keeping time with the arctic grass planted around the bench, the statues, and the dark red stone walking path through the park.

This was Mars One Park. “Built to honor the first successful human exploration of the red planet and inspire children and adults alike”, they said some time ago. They had invited him to come for the commemoration. He came, but did not speak. It was too painful.

And yet Neil was drawn back to it not long afterward and discovered that there was healing here. So he made it here regularly for the last year. He came sometimes specifically to remember the early times. He came sometimes to forget the later times. This was the place where both happened.

The arctic grass and the scrub trees, the deep red paving bricks for the path and the statues of the team, the other monuments and museum pieces of those early days mattered little to his memories. His eyes that were once so blue could still see the barren, pale marscape that greeted them on their landing and those first sols before they began shaping the planet to fit human activity. And, so, being here where it started, seeing with his mind’s eye the barren place of his past overlaid  with the richness of the present park, it allowed him to follow the trails of memories where he wanted.

The people sometimes distracted. Today appeared to be one of those days because the crowd was great and the buzz of conversation was strong in the air. Not packed, just busy with lots of walking and talking family groups. Perhaps a holiday. Neil considered the crowd, the length of the sol, and thought that it might be the one holiday he should not have come. It might be Mars Explorer day, the day to honor their landing. Yes, the more he considered it the more likely it seemed.

The crowds tended to move fast like a river and weave around him whether he was walking or sitting on his bench. They left him alone in his bubble of memories as if he had passed beyond their time. It wasn’t the crowd that was bothering him. It was one person. The young blonde man standing across the square was staring at Neil thoughtfully. He obviously already knew or was on the verge of recognizing Neil.

It appeared that he did. He was now walking purposely across the grass and the sidewalks directly to Neil.

He stopped a respectful distance in front and waited until Neil gave him his attention. Neil relaxed, thinking that this would be a more graceful encounter that he could more easily bear.

“Dr. Fellowes,” the newcomer began, “You are Dr. Neil Fellowes of the Mars One team, correct?”

He paused, respectfully waiting an answer from Neil as permission to continue. A man with a family walking passed jerked his head and slowed to look at them both.

Neil acknowledged he was with a nod and the young blonde man continued, “Forgive me for interrupting you at this beautiful park which must bring you such joy, and sadness. I really admire you, and the whole team, and wish they were still here with us as I am sure you do. May I sit with you? I would love to talk with you for a moment, or, for as long as you are willing.”

Neil shifted on the bench, leaned back, and gestured for the man to sit beside him. “I can’t promise a good conversation, but please, sit.”

The young man sat and both turned so that they were looking more towards each other. He proffered his hand to Neil and introduced himself, “I am Kurt Persson. I am a first generation Mars-son. I am in my second year of university and my heart still loves Mars history, I eat the classes and books up. Both my father and mother were early settlers from Sweden when they were young, and they had the honor of working with you for a time, on wind management. It was those stories that gave me my love of Mars history and enabled me to recognize you.”

Kurt’s voice took on a slight Swedish accent as began talking about his parents. Neil put two fingers on his temple as he concentrated on sifting through his memories. He smiled, “Ah, was it Harald and Kerstin Persson?”

“Yes!” Kurt said with a huge smile and obvious excitement.

Neil nodded, “Very brilliant engineers, individually, but as a team, phenomenal. They saved time, resources, and lives too, I know.”

Neil reached over and grabbed Kurt’s hand and squeezed it. “That time of life was one of my greatest times, oh the landing, the setup, that was exhilerating. But the things the teams like those with your parents did were remarkable. Are they still living?”

“Very much so,” Kurt answered, “Though I fear they are on the far side of the planet and I don’t get over to them often enough. They are working on a new, higher dome structure for larger cities, to house more above ground, but to change the planet less destructively in doing so. I think that was something you helped them see.”

They paused. Neil’s perspective was shifting and he was looking through the present to the barren past. The crowd disappeared out of his peripheral vision and the buzz of their conversation melted away. It was him and the beginning. And he just started sharing.

“This crater was absolutely beautiful in it’s barren starkness when we touched down. The stone and regolith had been practically untouched forever. A rover half a century before we arrived, and some deliveries of machinery and raw supplies in the few years leading up to our arrival. One over there,” he said pointing off his right shoulder, “around the southern ridgeline, another on the north, and one right just west of our touch down spot where we are sitting.

“The planet was raw, powerful hostility to life, but  it was at the same time raw loneliness, calling for me, for all of us to make it home.” He paused and the loneliness was tangible despite the crowd pressing them a moving around their little bubble of the park. The man and his family stood there facing them both and yet neither Neil nor Kurt noticed. The father leaned in and whispered to his wife and their children and gently beckoned another man and his family to come over.

“When we landed I was looking out the porthole window just this direction.” He motioned with arm in a straight line in the direction the bench was facing. “I saw the ridge where we felt was the most likely place to tunnel down for living quarters. This crater, Mars, it was going to be our home. It called us, it called all of us. But it didn’t want us yet. Not yet.”

Neil paused lost in memories. As Kurt looked slightly away from Neil he noticed the people around them. The large milling crowd had changed. The buzzing, deafening conversation has dropped a degree in their vicinity. People were stopped and listening. Just a few close by but it was spreading as more and more as he saw people whisper, point, and then squeeze in close.

“The sixteen of us were itching to get out of the lander immediately, ” he began again. “But we forced ourselves to take the proper time to run through the safety checks on our suits, the pressure locks, the radios. Everything went by the book not just once, not even just twice. We were determined to have no mistakes. That was our mantra and we stuck with it. This was our life and we could not go back if we did not succeed.”

He looked up to point again, straight ahead to the drill site and now entrance to the museum and noticed the dozen people gathered around listening to his story.

“Oh, hello,” Neil mumbled, momentarily distracted.

“Please, Dr. Fellowes”, a dark skinned man, the father of the family said, “Please continue if you don’t mind us listening too.”

Neil nodded, and the man promptly sat on the grass at his feet, gesturing his family and the others to sit also. They sat and squeezed in, surrounding the bench, cameras and devices came out for pictures and recordings. Neil glanced back to Kurt and tried to pick up where he left off.

“We ran the safety checks over and over, taking our time for two whole Sols. I even had time to connect up the wireless controllers to the equipment,” he motioned towards the south drop he had pointed out earlier. “And programmed them to move and meet us at the drill point there,” he finished pointing straight ahead over the sitting crowd.

“We finally began debarking on the third Sol, two by two, as it happened, like Noah’s animals. Nobody said anything about that the time, it’s just the limit of what we could squeeze in the airlock fully suited. I think it was Rachel who first made the joke about us being Noah’s pets but it was much later.

“I wasn’t the first one out, it was not Neil on the moon and Neil on Mars, though I jockeyed hard for that distinction — Neil Armstrong and Neil Fellowes, the first men on our first expeditions setting feet down off earth,” Neil was smiling big remembering it all. “I was second wave with Robbie. It was Rachel and Anton, me and Robbie. We were coming out as fast as we could. Nobody was saying anything. We just got out and moved off enough to make room for the others and stood here,” Neil gestured around them, “in the red silent wasteland, absorbed by the silence. Silence out of respect in part. But silence out of mostly shock I think.

“I didn’t even break the silence. I wanted too, so badly I wanted to. I had dreamed of it all my life, of mimicking the steps and words of my hero Neil Armstrong but on Mars. And here,” Neil stomped his foot, “and here I was.

“But we were still earthlings then. Earthlings in shock at being on Mars. We had all been completely enraptured as we came out the airlock and saw the beautiful pale red, barren view, and two moons! Two moons– Phobos,” Neil raised his left hand pointing the northwest, “was moving fast, coming over that ridgeline. And Deimos just hanging high up over there…” he raised his right hand to the north east.

“We were just silent and soaking it in. I think I know why- Neil, Neil had the blue marble called Earth that was home right there,” he said pointing to a blank sky.

“It was his anchor, keeping him focused. This,” he gestured around him, “this red land, was not ready for humans and yet it was our home. We were alone. We were earthlings. On the moon there was a blue and green planet to call home. Earthlings standing on Mars had almost nothing but the suits on our backs.

“So we kept coming out of the lander– eight and still no words had been spoken. I knew the moment was here and I thought perhaps I was destined for it after all. I tried to gather my thoughts and focus. I had written a dozen sayings that I thought passable though I never thought they compared to Armstrong’s ‘One small step’. I had prepared for it– and in fact we all did, we admitted later. But no one spoke. No one could make thoughts into words.”

Neil paused, a smile tugging at the corners of his mouth.

“Nine and ten were just coming out, Miranda and Sophie. All of sudden, Robbie… Robbie just opened his mouth to say precisely the wrong thing as he had knack for. At this singular event, an auspicious event, the most auspicious event in a century, an event that required a spectacular saying. And Robbie just spoke. Words said that cannot be unsaid.”

Neil leaned back, closing his eyes, the slight smile growing larger. “Still, it was pretty funny, later. We were shocked at first. Disbelief. Then angry… He killed our history making moment, never to be repeated. The death of a moment… And it was like we went through many of the stages of grief all in an instant for that moment we had been waiting for. Oh, we did finish with laughter, laughs for a lifetime at his expense…”

Neil leaned forward again, resting his chin on his hands on the cane again. His eyes were sparkling now, not the bright blue of his youth but the wise, aged grey eyes of lifetime exploring and seeing new things. The crowd was nearly thirty strong and enraptured, silently experiencing Neil’s story. Neil glanced around the crowd and spied the children from the first family to stop and motioned at them.

“Do you know what Robbie said?” he asked them. “We tried to change the recordings and to get something more magnificent for posterity but I’m afraid it was too late. Tell me, do you know what he said?” Neil asked the boy who looked to be about 10.

The boy looked shy, frozen in place. Neil smiled and gestured to the girl next to him, slightly younger he thought. “Or you?”

The big eyed, raven haired girl smiled shyly and nodded that she did. “Go ahead, tell us” Neil encouraged her.

She blushed, but loudly and proudly said, “Dang! I forgot to pee!”

The night was sultry

Posted: August 26, 2012 in fiction
Tags: , ,

Neil Fellows felt pain in his hands and forearms as he pushed the outer Armstrong Gate closed. He had stayed out too long and even the four layers of insulation, shielding, and life support paraphernalia of their suits could not protect for the length of time he loved to stay outside the bunker.

The gate connected, latched, and sealed off the Martian winter with a whimper of a sound. The atmosphere was too thin to make as much noise as one expected. But the physical jarring was not hampered by lack of atmosphere and the impact sent searing jolts of pain up his arms and seemingly directly into his eyes.

Neil paused just long enough to let the pain subside before proceeding down the slope towards the next gate, the garage, and eventually the underground bunker they called “home”. This entrance was the first entrance made into their bunker and was carved wide enough to bring the vehicles inside for safety and maintenance. Neil was the only one on the team who still used it for non-vehicular excursions because it was more challenging to use. But he couldn’t help it. He loved the name. Armstrong Gate. It was strong, powerful, and so emotive. Especially for Neil since it was a connection to his namesake, Neil Armstrong, the first man to step on another world. And here Neil Fellows was following in his footsteps on Mars.

He followed the driveway down into the dimly light tunnel to the next gate already warming up with the activity now that he was out of the direct cold. This large chamber was still cold since it was neither heated nor pressurized but it was a good deal better than the surface right now. Neil operated the man-sized pressure chamber door when he reached the end of the drive. The second door and chamber was large enough for some of their vehicles but required more resources. Resources that were very, very precious since they maintained life on a planet that would take their lives in an instant.

The effort and the impact of the door caused less pain this time. With a hum and a swoosh the atmosphere returned to the chamber and Neil smiled like always with the return of definite sounds instead of faint echoes of sounds that one experienced outside in the Martian atmosphere. The sounds were even more completed as he opened his mask and turned off the breathing machine, closing his eyes and again smiling as another sense returned: the smell of Mars.

When Neil was outside he never knew he was missing these senses. There was too much glory in being on Mars, too much data to collect with his eyes that he never noticed that nose and ears were removed from him. There was always that transition period of suiting up and waiting to step out. But that all energetic anticipation of stepping out onto the surface again that Neil didn’t mind the short-lived muteness and lack of olfactory sensations of the suit.

The smell of Mars. Here in the garage behind the two doors and the airlock of the Armstrong Gate it was the strongest. The garage and the whole bunker was dug underground to protect them from the solar radiation. Earth had a massive atmosphere to filter and protect but Mars was thin and exposed so living underground was the best alternative. In the garage the walls and floor were pure bedrock and martian soil and the smell was uniquely Martian.

Neil walked passed their exploratory vehicles and various other machinery to the doors that lead into their living and common quarters. They were also carved out of rock and so the smell of Mars was there with them in this space where they lived, and worked, and played, and ate, and it soon became just a smell of people. Not quite Mars, but Martians. Something a little different.

Neil entered through the strong doors and into definitely warmer living quarters. He paused in the “mud room” and removed the layers of the suit and hanging them up, connecting them up to recharge, shedding thirty pounds in a few minutes. Still chilled, he left his balaclava pulled up over his ears and head and walked down the hall into the large common area passing by the humidifier that kept all their skins from cracking and bleeding. That was another thing he loved about staying outside so long. Neil found he really appreciated the comfort of their living space– the heat, the air, the moisture control. Little things that could be taken for granted until one was deprived of them.

A sudden very distinctive earthly smell hit him like a wall as stepped into the common room. He saw most of the group gathered in the corner watching the video screen, laughing, talking, and eating. It was warm, it was inviting.

“The night was wet and hot, hot and wet, wet and hot; that’s humid. The night was humid.”

Neil stopped at the back of the room smelling the air. This was something else besides the reintroduction of his deprived sense. What was that smell? It was an earthly smell? It had been so long.

“Hey, Neil- you’re late for movie night!” Anitoly shouted to him, waving him forward.

The others turned around, smiling and beckoning him forward. “We almost don’t have any popcorn left!”

Popcorn. Neil smiled as he removed his balaclava and started forward to the group. That was the earthly smell. Popcorn.

“The night was sultry” said Mrs. Lift.

This is my first attempt to roast a bean from Burundi, and the latest in several recent exploratory coffees from Africa. This is my two hundred and thirty-second roast in my RK drum and Fiesta grill.

My grill is setup with two burners, marked “Left B” and “Right B” in the graph. I keep the drum over the right burner for controllable direct heat and the use the left burner to boost the chamber temp. The burner measurement is a percentage value using 1,000 as full heat. The roasting chamber temps are Fahrenheit.

The goal is use the left burner to slowly raise the temp for the drying phase and then raise the right burner (the direct heat) for actual roasting. Once the beans are into first snap I like to bring all burners to the lowest setting to draw it out a bit and prevent it from rushing into second snap.

I took 1,080 samples with my cheap temp-probe, or a roast of 18 minutes.

The black markers at the bottom are markers for the following notes:

  • sample 852, 14:20, first snap is heard
  • sample 876, 14:45, snaps are in groups
  • immediately into slow roll
  • sample 895, 14:55, rolling strong/fast
  • sample 928, 15:30, smokey, still rolling
  • sample 967, 16:07, mostly done with first snap
  • sample 1000, 16:30, open lid for 05
  • sample 1067, 17:40, a second snap or two is heard
  • sample 1072, flame out
  • sample 1080, into the roast cooler

I can’t wait to see how it tastes!

Safari Preview Swamps SSO Server

Posted: May 17, 2012 in tech
Tags: , ,

Sometimes when I log in to my SSO console I would see a lot of sessions for a single user. I can understand three, four, even six, between restarting the browser and using multiple browsers. I do that a lot since I am responsible for the SSO system and have to check out the multiple ways of authentication. But the excessive sessions for a few users bother me. I mean excessive like 20 or 40. And for ME!

It’s not a big deal — the SSO server (CAMS by CafeSoft) could handle it. But I needed to know what was going on. What if this is something that needs to be fixed? A hole that needs to be plugged? Besides that, I try to save money where we can and sometimes we hit our licensing limit for development and integration/testing and I don’t want to have to buy more seats if we do not need them.

By process of elimination I was able to narrow it down to either one particular browser or the Kerberos authentication that it was configured to do. I had been watching the logfiles to see the requests that were made and redirected to the authentication links, the unique session id assigned to it… and then nothing. That sessionId was never used. There was no success message logged with that session ticket.

Now it was time to get sneaky and figure out what the browser was doing. I was already thinking that it was related to the browser pre-fetching things to try and be faster. So I killed my browser and had another admin remove all my session from the system. With Wireshark running and capturing everything for my server subnet I launched Safari again and followed my Usual Methods.

  1. Launch browser, type over the address bar with the address of my dev server
  2. Admin in other room constantly refreshes and reports no new sessions as I am typing the address. That shoots down my initial thoughts of pre-fetching.
  3. I hit enter and observe the usual redirects and invisible kerberos login
  4. My assistant from the other room reports I now have one session. Everything is perfectly normal
  5. I begin typing on the address bar for a slightly different space on same server but different Apache virtual host and eventually hit enter. My assistant reports still only one.
  6. Launch new tab– Eureka! Assistant reports I now have sixteen sessions.

So… Safari is configured to display the “Top Sites” thumbnails in a new tab. And those “Top Sites” are not built based on cache but with new requests. Whaaaa????
Safari Tabs setting

It gets worse.

Remember I had Wireshark capturing in the background while I was doing this? I examined the packets and was able to determine what was going on. I see Safari making the request and the server responding with the 302 HTTP code to be redirected to the kerberos login page.

GET /CamsConsole/sessions/ HTTP/1.1
Host: *****.********
X-Purpose: preview
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)
AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5 Safari/534.55.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive

HTTP/1.1 302 Moved Temporarily
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 17 May 2012 16:04:36 GMT
Set-Cookie: BigIPDensitySecure=604176650.37663.0000; path=/
Vary: Accept-Encoding, User-Agent

There are several interesting things to note in this request. First – there is a special HTTP Header being used by Safari, X-Purpose: Preview. Second, there is a very notable lack of other HTTP headers. In fact, you could say just the basics of compression, the Agent, and KeepAlive. That last one is important by the way at figuring out what is going on.

There is one important clue in the reply from the webserver and it is not really that obvious until you look at the later packets.

When you filter a conversation in Wireshark it makes a new filter for stream=xxx to show you everything in that conversation. With most HTTP servers this stream can be open for a 100 requests, five minutes, or something in between. It’s all negotiated between client and server. This is part of the function of the “KeepAlive” header, it is the client telling the web server, “Hey, I support keeping this session open if you do”. So it stays open for further requests.

And if I thought I was done looking at everything that my browser was doing I would miss some other details. Because this conversation keeps repeating. Several times. Exactly the same.

So I clear the Wireshark filter to restore all the conversations to see them as they happen by time. The very next stream is the request to get logged in via Kerberos. I’ll try to keep this long post short and summarize — the browser says give me “this”, and server says “401- please login with ‘Negotiate’, and here’s your BigIP cookie”, and browser says “hey, I have ticket, here you go”, and the server says “OK, here’s your SSO cookie and here’s your BigIP cookie”.

And then I see that again. And again. Several times over multiple streams just like the original stream. And meanwhile the Cams authentication service creates a new session for me. And again. And again. As long as it keeps doing it.

And that is when it hits me. The Safari X-Purpose Preview function does not utilize cookies. It seemingly does not accept, keep, track, or submit any cookies. It never sends the BigIP or SSO cookie.

It’s just going to fill up my logs with useless new sessions and deplete my available licenses.

I’ve been scripting a lot of fancy things into our F5 BigIP LTM-1500 lately and this seems like another perfect way to solve this little problem. A simple iRule applied to the virtual server instance can intercept the HTTP Request before it even goes to the web server.

This iRule goes up near the top of the other iRule items (before any ProxyPass iRules if you use those) and acts on HTTP Requests only. When it detects that Safari header it responds with a 200 and a short message and never ever goes to the web server, Cams, or passes Go.

Haiku Preview Message

Haiku Preview Message

Yes.. I did wax a little more poetic than absolutely necessary, but that white box in the picture above is my short and quick way to stop Safari Preview from depleting my development Cams licenses.
iRule script download

There’s a fun little geeky comic online that you may have heard of, XKCD.

A while back the author had comic that resonated with me about password security. I’m not buff enough in my math skills to keep up understand the equation but I could follow the principle. The idea is that four (or so) random words is more secure than making an extremely complex password that has numbers and special characters embedded and replacing letters. The challenge with this somewhat standard practice of l33tspeak is it has to be written down. But… since we are people that love stories, four random words in real English will be more memorable because we can make up a story to remember it.

Here’s the famous comic.

Recently, while I was working on one of my other Perl scripts, I was on an online forum and saw a post about how to make a random sentence. That tickled my fancy and I came up with a quick and dirty little CGI script to generate a random sentence suitable for passwords. Unfortunately a lot of places have requirements to still include special characters and numbers but this little script will meet those requirements. The spaces meet special character requirements (most of the time) and a number between 1 and 99 is included.

The results? They are often amusing and poetic. Sometimes they are risque. It just depends on what is in your system’s local dictionary. Download Perl Password Poetry Producer


print "Content-type: text/html\n\nPassword Poetry Generator\n";

open (INWORDS,"< /usr/dict/words");
close INWORDS;

my $poem;
my $randiddly=int(rand(99));

if ($randiddly%2==1){ $poem= join" ",(map{$w[rand@w]}1),$randiddly,(map{$w[rand@w]}1); }
else { $poem= join" ",(map{$w[rand@w]}1..2),$randiddly; }

if ("$poem" !~/[A-Z]/){ $poem= join" ",$poem,( map{ ucfirst ($w[rand@w])}1);}
else { $poem= join" ",$poem, (map{$w[rand@w]}1) ; }

print "$poem\n


#some html code has been stripped for wordpress

Some sample passwords: