Using a Dynamic Alarm to Rescan Unavailable Datastores

Hi All,

So in our particular retail environment we have had an ongoing issue with Iscsi datastore availability in the case of a power outage.   We also have the same problem on in store deployments when the onsite technicians power on the components in the wrong order (ie power on the hosts before the iscsi array).   The result of this problem was of course escalations for retail stores being down for a significant period of time following a power outage.

Of course I googled the heck out of this problem -  and the only solution I have found was to change the boot delay on the ESXi hosts to exceed the array boot time.  That was not a great solution for us -  because it would delay starts on planned reboots without a corresponding power outage.

One day I was working on another reactive alarm and I got that light bulb over my head.   I thought "I wonder if there is an alarm type for datastore connectivity to hosts that I could use to trigger an action?".   It turns out there is a condition that can be monitored at the datastore level to alarm for this -  it is called Datastore State to All Hosts. In the screenshots below -   I created a custom rule with this condition.   For a review on setting up Powercli based alarms -  refer to my blog post here.

Now that we have the alarm setup  -   let's get into the Powercli code behind it.  In our particular implementation -  we only have to worry about 3 nodes in our clusters,  so my script is focused on that,   the methodology would also work with a larger cluster and a foreach-object loop -  but I am just going to use my already written script to show how I solved my particular problem, but the logic will scale.

# In response to a datastore alarm, remove all snapshots on the datastore.
# This is just for demonstration purposes.
# Find more info at http://blogs.vmware.com/vipowershell
$basePath = "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\scripts"
$ProgressPreference = "SilentlyContinue"
$env:APPDATA = "c:\Documents and Settings\All Users\Application Data"

Add-PSSnapin VMware.Vimautomation.Core -ea SilentlyContinue
$WarningPreference = "SilentlyContinue"
Connect-VIServer localhost -User "nobody\someone" -Password "xxxxxxxxxxxxxxxxxxxxxx" | Out-Null
$email = "jason.willey@gmail.com"
$datastoreId = "Datastore" + $env:VMWARE_ALARM_TARGET_ID
$datastore = Get-Datastore $datastoreId
$impactedhosts = $datastore |get-vmhost
$cluster = $impactedhosts |select -First 1 |get-cluster 
# host one check
$Hostone = $impactedhosts |where {$_.name -match "x1"}  
$Hostoneconnected = $Hostone.ConnectionState
$hostonestate = get-vmhost $hostone |get-datastore $datastore
$hostonestate = $hostonestate.Accessible
if ($Hostoneconnected -match "connected") {
$hostonestate = get-vmhost $hostone |get-datastore $datastore
$global:hostonestate = $hostonestate.Accessible }
#host two check
$Hosttwo = $impactedhosts |where {$_.name -match "x2"}  
$Hosttwoconnected = $Hosttwo.ConnectionState
if ($Hosttwoconnected -match "connected") {
$hosttwostate = get-vmhost $hosttwo |get-datastore $datastore
$global:hosttwostate = $hosttwostate.Accessible }
# Host Three check
$Hostthree = $impactedhosts |where {$_.name -match "x3"}  
$Hostthreeconnected = $Hostthree.ConnectionState
if ($Hostthreeconnected -match "connected") {
$hostthreestate = get-vmhost $hostthree |get-datastore $datastore
$global:hostthreestate = $hostthreestate.Accessible}
#  starting rescan loop
$global:counter = 0 
while ($global:hostonestate -match "False" -or $globalhosttwostate -match "False" -or $global:hostthreestate -match "False" -and $counter -lt 10 {
if ($global:Hostonestate -match "False") {Get-VMHostStorage -VMHost $Hostone -RescanAllHba -RescanVmfs |Out-Null
 $global:hostonestate = get-vmhost $hostone |get-datastore $datastore;
 $global:hostonestate = $$global:hostonestate.Accessible  }
if ($global:Hosttwostate -match "False") {Get-VMHostStorage -VMHost $Hosttwo -RescanAllHba -RescanVmfs |Out-Null
 $global:hosttwostate = get-vmhost $hosttwo |get-datastore $datastore;
 $global:hosttwostate = $global:hosttwostate.Accessible  }
if ($global:Hostthreestate -match "False") {Get-VMHostStorage -VMHost $Hostthree -RescanAllHba -RescanVmfs |Out-Null
 $$global:hostthreestate = get-vmhost $hostthree |get-datastore $datastore;
 $global:hostthreestate = $global:hostthreestate.Accessible  } 
sleep 120
$global:counter = $global:counter + 1
if $counter -eq 9 { $global:hosterror = "$datastore has been rescanned every 2 minutes for 20 minutes and is not available on all nodes of $cluster"

}
if ($globbal:hosterror -ne $null) {Send-MailMessage -To $email -Subject "Unable to rescan ISCSI storage in cluster $cluster" -Bodyashtml -Body "$global:hosterror"  -SmtpServer "m.nobody.com" -From "VMware@nobody.com"}

So I will skip all the logic for interpreting the alarm received as that is detailed in another post here.

First things first - I translate the actual datastore ID presented by the alert to a more script friendly datastore name and get the hosts that are supposed to be connected to this datastore. I also get the cluster name for use later on when we send an email alert in the case we were unable to successfully rescan.

$datastoreId = "Datastore" + $env:VMWARE_ALARM_TARGET_ID
$datastore = Get-Datastore $datastoreId
$impactedhosts = $datastore |get-vmhost
$cluster = $impactedhosts |select -First 1 |get-cluster 

From there - I now verify that each host is actually connected to vcenter, and verify if the datastore is viewed as available from the host side. I then repeat this logic for each possible host.

$Hostone = $impactedhosts |where {$_.name -match "x1"}  
$Hostoneconnected = $Hostone.ConnectionState
$hostonestate = get-vmhost $hostone |get-datastore $datastore
$hostonestate = $hostonestate.Accessible
if ($Hostoneconnected -match "connected") {
$hostonestate = get-vmhost $hostone |get-datastore $datastore
$global:hostonestate = $hostonestate.Accessible }

Now I set up the rescanning loop. In the case of our environment we have chosen to rescan every two minutes for twenty minutes and then send an email alert to second level support. I used $global: variables here so they will survive the loop. So after the rescan, it checks the status of the datastore again, and continues running the loop until all the datastores have returned, or the loop counter reaches 10. At the end of each pass of the loop there is a 120 second sleep to give us our two minutes between rescans. One the last iteration of the loop - it writes an error message that after 20 minutes of 2 minute rescans - the cluster still has a datastore availability problem.

$global:counter = 0 
while ($global:hostonestate -match "False" -or $globalhosttwostate -match "False" -or $global:hostthreestate -match "False" -and $counter -lt 10 {
if ($global:Hostonestate -match "False") {Get-VMHostStorage -VMHost $Hostone -RescanAllHba -RescanVmfs |Out-Null
 $global:hostonestate = get-vmhost $hostone |get-datastore $datastore;
 $global:hostonestate = $$global:hostonestate.Accessible  }
if ($global:Hosttwostate -match "False") {Get-VMHostStorage -VMHost $Hosttwo -RescanAllHba -RescanVmfs |Out-Null
 $global:hosttwostate = get-vmhost $hosttwo |get-datastore $datastore;
 $global:hosttwostate = $global:hosttwostate.Accessible  }
if ($global:Hostthreestate -match "False") {Get-VMHostStorage -VMHost $Hostthree -RescanAllHba -RescanVmfs |Out-Null
 $$global:hostthreestate = get-vmhost $hostthree |get-datastore $datastore;
 $global:hostthreestate = $global:hostthreestate.Accessible  } 
sleep 120
$global:counter = $global:counter + 1
if $counter -eq 9 { $global:hosterror = "$datastore has been rescanned every 2 minutes for 20 minutes and is not available on all nodes of $cluster"
}

The last step once the loop has been terminated is to check if the error message has been populated - send an email out to the second level support team letting them know that automatic rescanning has not been successful.

if ($globbal:hosterror -ne $null) {Send-MailMessage -To $email -Subject "Unable to rescan ISCSI storage in cluster $cluster" -Bodyashtml -Body "$global:hosterror"  -SmtpServer "m.nobody.com" -From "VMware@nobody.com"}

I am hoping that you find this useful, as I have heard this is a very common problem, especially with Iscsi implementations. Leave me a comment if you have any questions or suggestions, or just leave a comment if you had a similar problem and solved it in a different way.

Applying XenServer Updates

I've been playing with the XenServer 4.1 product on a low end spare server I have laying around. To it's credit XenServer is the only one the big three hypervisors that will run on this Pentium D 820 server, as Hyper-V requires DEP, 64bit and processor side virtualization, and ESX3i requires specific approved hardware to function. Granted, without the processor virtualization assistance, XenServer is limited to deploying Linux guests only, it's still enough to be able to play with the interface and deploy some basic machines.

While not a complex and deep as VMware Update Manager, I am impressed with Citrix's simple "just the servers" approach to patching. While we are deploying VMware Update Manager to handle the ESX hosts, I can't see us ever using the guest patching capabilities, nor could I see any large organization doing so. In my opinion , most large IT shops won't want to maintain a separate update process for virtual machines and a separate update process for physical machines, despite the ability to roll back changes and other benefits of VMware Update Manager.

One of the best features of Citrix's product is their "Check For Updates" launcher. It gives you a clear view of what updates you already have, and which updates you should apply. Along with the download links are links to the patch notes. While it is much more rudimentary than VMware Update Manager, I appreciate the simplicity.

The Citrix patching utilities obviously don't compare with VMware Update Manager, but they certainly have the advantage over the VMware patching website. :

I found the deployment of the updates once they were downloaded to be incredibly simple, and the updater to be intelligent when it came to evaluating the server receiving the patches. It first attempted to evaluate moving the guest to another host, which was impeded by the fact I only have one server. Once it realized there were no migration options, it suspended the VM while applying the update and rebooting. After communication was re-established with XenCenter, the guest was resumed automatically. End to end, a pretty simple and intelligent process.

I hope VMware and Citrix (as well as Microsoft) are learning from each other, and improving their products based on their competitors strengths and weaknesses. I would like to see VMware implement a "simple" host updater into a future version of VirtualCenter. I will also be watching to see if XenCenter implements some of the more advanced features of VMware Update Manager directly into XenCenter.

PowerShell for VMware

Rich over at VM/ETC just posted a great bunch of links to various PowerShell scripts that he has run across in his Internet travels, as well as some links on learning the PowerShell scripting language.

http://vmetc.com/2008/08/27/powershell-scripting-examples-for-vmware-virtual-infrastructure/

One other link I had planned on posting as well was one from the Planet VMware blog : Managing VMware with PowerShell Webinar

I expect that the VMware Developer Center's PowerShell scripting contest should yield many interesting new VMware PowerShell scripts once it's finished.

VMware Joins the SVVP

In a letter from the CEO of VMware today it was announced that VMware is in the Server Virtualization Validation Program. I'm fairly sure they weren't on the list yesterday when I checked. This officially ends any ambiguity of whether you will be fully supported using VMware with Microsoft products.

VMware in the SVVP is a great win for them, but I don't think it dulls Microsoft's new advantage of having the only virtualization platform that is guaranteed not to have to be reproduced on physical hardware.

You can read VMware's statement here :
The Console : VMware is in the SVVP

That's Why They Call It the Bleeding Edge...

As almost anyone running VMware must already know - VMware had a major bug in ESX 3.5 Update 2. http://kb2.vmware.com/kb/1006716.html I find it surprising that anyone is already running this update in production, as it was just released on 7/25.

My company was not affected, because we have a "forum vetting" policy for all new non security related updates. It's an unwritten rule that we always wait at least a month before progressing any major update, and we thoroughly check the company forums and non company sources for experiences that might be similar to what we will experience. This policy has paid dividends on the rare occasion when a major release has suffered issues. Using this method, we were able to avoid major issues with Windows 2003 SP1 and the some of the HP service packs and firmware in the past. By no means am I advocating using the internet as the sole source for testing, but it certainly is valuable as an additional check.

I have to give credit to VMware despite the circumstances today, as they are on top of making people aware of the issue. We received a call from our assigned SE this morning alerting us to the issue. Some issues are unavoidable, I'm just glad VMware handles them in the correct way.

Virtualization Certification Trifecta

For a long time, there was only one player in the Virtualization certification market - VMware with their VCP. I was waiting for them to broaden their certification beyond the initial certification, so I was please to see the announcement of the VCDX program.

VMware is no longer the only one in the virtualization certification race, as Citrix has released their Xen CCA, and Microsoft is preparing a virtualization certification as well.

So how do those certification programs compare in difficulty and return on investment?

Let's start with the first program out of the gate - the VCP.

• Difficulty: I found this certification to not be incredibly difficult for people that have spent a good amount of time with VMware, and have had access to the official training materials. This certification has some "paper certification" protection in the sense that to be a VCP you must have completed some form of official training. I'm not sure I completely agree with this requirement, as I have seen some passionate fans of the product being held back by the lack of training budget.
• Return on Investment: Very high in my opinion, but I think that large company HR departments aren't looking for the VCP yet , it seems to be more the consulting companies specializing in Virtualization that know to ask. A quick search of dice.com found 96 of the 1547 VMware jobs in the US looking for a VCP. Just for a point of reference - a search for MCSE found 1801 results. Here's an interesting article on the value of the VMware certification: http://servervirtualization.blogs.techtarget.com/2007/10/12/the-value-of-the-vcp-vmware-certified-professional/

Now the VMware VCDX - of course the tests are not fully released so this is all speculation based on the announcements that were made.

• Difficulty: I think this will be the king of Virtualization certifications for the near future. I think that for the rest of 2008 and probably well into 2009 there will be less than 1000 of them. I think the most difficult part for the majority of people will be the Submission of a successful VMware Infrastructure design plan.
• Return on Investment : I think this will be very high for high end consulting jobs, and elite enterprise customers - I don't see it being a big draw for the average job. Oddly enough - there are already 4 jobs from dice.com looking for the certification. The VMware Enterprise Administration exam is just being released this month.

Citrix CCA in XenServer - They appear to have recently released a new revision of this exam - A02 that is focused on the XenServer Enterprise Edition 4.1 product. http://www.citrixtraining.com/courses/course_list.cfm

• Difficulty : While I have not taken this exam (yet), from the material available it appears to be very similar to the VCP in difficulty level, without the requirement to take the training class that tends to restrict some people.
• Return on Investment : Hard to say. It appears to be a niche skill right now, and most of the people that are actually looking for Xen on dice.com are actually VMware consulting companies. I found no jobs looking for Xen CCAs It is possible that companies are training up internally rather than searching for external talent.

Microsoft's 70-652 Technical Specialist : Windows Server Virtualization, Configuring. The test is planned to be released this August. http://www.virtualization.info/2008/07/microsoft-releases-hyper-v-10-exam.html (link courtesy of Mark Bradley)

• Difficulty: Based purely on the published exam objectives, I would place the difficulty somewhere between VCP/Xen CCA and VCDX. I think anyone with a background in Windows clustering will tend to have a advantage on this exam.
• Return on Investment : Time will tell. There are already 16 jobs on dice.com looking for Hyper-V and the Microsoft marketing machine is only getting started. I think the strength of Microsoft name and marketing arm will drive the value of this certification very quickly.

I would definitely like to go for the trifecta of Virtualization Certification, with a VCP, XenServer CCA, and a MS TS in Hyper-V. Having a little product diversity never hurt the resume.

I would be interested to hear other people's opinion on the value of the various certifications, so feel free to leave a comment, I would be especially interested to hear from my new readers in the UK and Australia as to how you see the value of the various certifications in your local job markets.

High Density Virtualization Environments and Windows 2008 Datacenter Edition

In November of 2007, Microsoft made an announcement I truly found shocking - When purchasing Windows 2008 the customer is entitled to "unlimited virtual instances per license".

http://www.microsoft.com/windowsserver2003/datacenter/unlimitedvirtualization.mspx


Now that Hyper-V is reality, the consequences of this move are huge.

As an example - let's take a DL585 with 4 sockets and do the math on this using list price.

• 4 processors of datacenter edition @ ~$3000 per processor would be $12,000
• 1 single Windows 2003 server Standard license costs $1000.

The break even point on Datacenter edition is 12 servers (depending on the number of CALs that need to be purchased). If you are using Datacenter Edition, you effectively saved money on every copy of Windows required past that instance.

So you may be thinking "What is considered a high density server environment?". Lacking an official ruling on the term, I would place it at anytime you are averaging more than 3 virtual machines per processor (not core). For example a fully populated DL585 with dual core processors would need to have more than 24 guests to enter this category.

Going with the example number of 24 guests on the virtual machine with the Datacenter edition license, and the pricing from the example above - you have now saved $12,000 in budget without having to make any difficult decisions. The savings are even larger if you deploy virtualized instances of Windows 2003 Enterprise.

The best news is that this doesn't just apply to Hyper-V, it also applies to VMware, Citrix Xen, and assumedly any other virtualization software deployments as well.