Purple Screen of Death - Dell issues

You may notice I don't post a lot of issues on here. That's because we frankly don't have a whole lot of issues. I personally attribute it to using Best Practices, and regular maintenance. That said, things will still get lost in the weeds occasionally.

We ran into the Purple Screen of Death on one of our ESX 4.1 boxes yesterday. It is a Dell R610, and apparently had a hardware hiccup, and kicked out errors stating:

		Tue Apr 19 21:09:15 2011		PCIE Fatal Err: Critical Event sensor, bus fatal error (Bus 1 Device 0 Function 1) was asserted		0xA10002FBF9AD4DB1000413186FAA0101h
		Tue Apr 19 21:09:15 2011		Err Reg Pointer: OEM sensor, OEM Diagnostic data event was asserted		0xA00002FBF9AD4DB10004C11A7E011610h
		Tue Apr 19 21:09:15 2011		PCIE Fatal Err: Critical Event sensor, bus fatal error (Bus 1 Device 0 Function 0) was asserted		0x9F0002FBF9AD4DB1000413186FAA0001h
		Tue Apr 19 21:09:15 2011		Err Reg Pointer: OEM sensor, OEM Diagnostic data event was asserted		0x9E0002FBF9AD4DB10004C11A7E011610h

We rebooted the box, and it came back online just fine, but we didn't feel comfortable with it, so we stuck it in maintenance mode and had someone contact Dell. Dell reports that we need to update the Bios on it:

-----

Yes, it appears your system is affected by some of the microcode updates released from Intel on the 5500 and 5600 series processors.  That is likely the cause of these PCI errors.  The course of action we need to take is:

·         Update the BIOS

·         Update the iDRAC

·         Clear out the old log entries

·         Monitor for re-occurance.
------

So it's sitting in maintenance mode until someone has some time to love on it. The awesome thing is that we run N+1 (one more box than we need) so we have that luxury. I know plenty of people that refuse to listen to why you should go N+1 who would be scrambling to make a maintenance window to update it.

The downside to this whole fiasco was that when it hiccupped, it stayed online (as is the default with ESX), and held onto the Storage of it's VMs. Therefore, HA couldn't restart them on another box until someone manually SHUT OFF the pretty Purple-VM-Eater. As soon as they did that, all was well in the world and the phone stopped ringing.

Since I'm not fond on relying on manual intervention to make HA work, I found the command for auto-restart when a PSoD happens and applied to ALL our hosts:

esxcfg-advcfg -s X /Misc/BlueScreenTimeout

Were X = number of seconds before restart

I went with 30 seconds, that way I have the opportunity of seeing the screen if I so happen to be looking at it when it dies.

 
------
Dustin Shaw
VCP

All About the Design

It seems to me that you can always tell if an IT guy is worth his water by the way that he architects/designs solutions.

We picked up a small shop recently that their previous guy built them a heavy-built Dell T710 with Microsoft Hyper-V. They've got 5 VMs running on it, so nothing wrong with the basic idea behind it, but the execution is where I have an issue.

The machine was originally built (from factory) with the onboard SAS 6/iR RAID card with a set of four 450GB drives in a RAID 5 config. No hot spares. Issue number one.

He realized that he must have under built, so he then went off the deep end. He went out to NewEgg and bought four 2TB SATA drives, bought the caddies off EBay, and bought a PERC 6/i RAID controller directly from Dell (must've not found it cheaper somewhere else). This means the factory warranty from Dell (3 year NBD onsite) won't extend to anything except the RAID card. What's the most likely thing to die on a server again?

He then went in and reconfigured the machine to run off the PERC with a second RAID 5 set with the 2TB drives (again no hot spare), and directly attached one of the VMs to it for their main data storage. This is all fine and dandy, except for the fact that there is 800GB free on the first RAID set, and they are only (at this point) using 40GB on the second RAID set (leaving roughly ALL of it free).

And here comes the second part of the issue - much bigger than the previous config issues.

The PERC 6/i controller has issues running on drives 1.5TB and up. The latest firmware revision runs better than the previous one, but the controller is regularly (it's happened 3 times in a month, hence why the old guy went bye-bye) dropping a drive offline. Part of the issue is that when it drops the drive, it drops the RAID set offline at the same time.

When I got called in, the RAID set had dropped two drives offline (meaning bad, bad things could've happened to their data). I reseated the "bad" drive, imported the foreign config, and let the RAID set rebuild itself. I then proceeded to move their data off of that RAID set and remove the config from the environment.

They still have 800GB to grow into before we even need to consider doing anything. My first plan of order is to tell them that they need to get another 450GB drive and put it in as a hot spare. As I mentioned, this is a small shop, and the server sits on the floor in a closet. They only open the door when there is a problem, so they could easily loose a drive and not know it until someone asks "What's that red light for? It's been doing that for a month." Hence the driving need for a hot spare.

------
Dustin Shaw
VCP

We'll miss you ESX!

For those of you who haven't been paying attention in VMware, with the advent of vSphere (over a year ago), the official recommendation is to no longer use ESX, but now to use ESXi.

Why would I do something silly like that?

I'm glad you asked :-)

ESXi is lighter - not so heavy on the host box
No security console - aka, less security risk and things to patch
Less configuration options - easier to setup a new host

What are the downsides?


Oh, those... Well, if you believe VMware, none! If you listen to third party vendors, then you will hear a long list of agents that they've had to scramble to figure out new methods of doing. Some (like Quest's vRanger Pro) are switching to a VA (Virtual Appliance) format. Others are just pulling from the remote API calls. And others (like HP Systems Insight Manager or Dell OpenManage) are switching their management tools to use SNMP and CIM based calls for managing the hardware.

Why can't I just keep using ESX?

Oh, that's because ESX is going to disappear in "future major releases," so why wait until you are forced into the situation. Try to be a little forward looking; after all, isn't that why you're Virtualized in the first place? :-)

If you want to read more from VMware, you can go here:
http://www.vmware.com/products/vsphere/esxi-and-esx/overview.html
http://www.vmware.com/products/vsphere/esxi-and-esx/faqs.html

------

UPDATE 3/29/11

VMware made it official that as of vSphere 5 (coming out July-August 2011), ESX will no longer be included. ESXi will be the only way to go. Better start migrating!

Read more about it here.

------
Dustin Shaw
VCP