vRanger 5 and ESXi Performance Settings

Everyone should know that the previous recommendations from Quest were to increase the CPU Reservations to 1500 on ESX servers for the best performance from vRanger.

I just recently ran across an article (here) from Sept 2011 detailing that this is NOT the case for ESXi. There is no performance gain on ESXi by increasing the CPU reservations from the default 293.

------
Dustin Shaw
VCP

vRanger error "There is an error in XML document"

I ran into an issue with one of our repositories - every job in the particular repository was failing with the error:

An internal error occurred during execution, please contact Quest support if the error persists. Error Message: There is an error in XML document (823, 223).

This error is commonly caused by a corruption or issue with the manifest file as detailed here.

To fix it, go into the manifest file for the repository (GlobalManifest.metadata), make sure that all trailing brackets are closed, and add </vRangerGlobalManifestFile> at the end of the document.

Save and test.

------

Dustin Shaw
VCP

Installing PowerCLI on Server 2008

In attempting to install VMware vSphere PowerCLI on Windows Server 2008 x64, the following error comes up. This also happens on Server 2008R2.

Error 1406. Could not write value InstallPath to key \Software\VMware, Inc.\VMware vSphere PSDK Runtime. Verify tha tyou have sufficient access to that key, or contact your support personnel.

The prescribed fix for this is to remove the following registry key (and subkeys):
HKLM\Software\Wow6432Node\VMware, Inc.

*** Please make sure you know what you are doing in the registry before you do anything in there!!!

After that, installation proceeds successfully.

The only thing under the VMware, Inc. registry key is a "volatile" key with a UUIDHost DWord under it. Since I don't like to mess with other programs installed on the server (the one I was using has Quest vRanger installed), I exported the key, removed it, installed PowerCLI, and the imported my volatile key back in.

------
Dustin Shaw
VCP

Move vRanger Repository

I needed to move a vRanger Repository, but didn't want to start all my backups over, so I found a handy article from Quest that identifies how to do it.

FYI, I believe it requires vRanger 5.3 and up.
The basics are you go into SQL Management Studio, expand down to dbo.Repository to identify the repository, and change the host or target directory. Then go to dbo.RepositoryCIFS to change the sharename.

Make sure you restart the vRanger Services after making the changes.

------
Dustin Shaw
VCP

vRanger "host could not be identified"

We had our vCenter sever hiccup on us yesterday, so (naturally) our vRanger backups failed last night.

The error we received was:

An internal error occurred during execution, please contact Quest support if the error persists.  Error Message: VMSERVER's host could not be identified

Essentially what causes this issue is a communication sync error between vRanger and vCenter. I ran across this error a year and a half ago running on vRanger 4.x (you can read my post here). This time it was on Quest vRanger version 5.3.0, so I did a quick search to see if there were any newer solutions for verison 5. It appears that Quest now has a KB article about it.

Basically, their solution is to restart the services - stop the API and FLR Services, restart the vRanger service, then start up the API and FLR services.

I always prefer to go above, so I also refreshed the jobs (like my previous experience taught me) by opening them up and click next and finish all the way through (without changing anything).

Next time I'll remind my guys to restart vRanger services whenever vCenter bounces.

------
Dustin Shaw
VCP

vRanger 5.0 is released!

As of January 31, 2011, Quest vRanger 5.0 is now publicly available and out of beta.

There are now two versions of vRanger available: vRanger Standard Edition 5.0 and vRanger Pro 5.0. Pro buys you these additional features that people have been looking forward to: Replication, LAN Free, Catalog, and Linux FLR. Here's a look at them:

vReplicator is integrated into vRanger Pro
You no longer have to buy two separate products to do backups and replication if you choose to go the Quest route. You can use one single product for both.

Support for NFS and FTP Repositories
We already had support for CIFS and SFTP targets. The addition of NFS and FTP gives more flexibility on what you can use as a backup target. They didn't integrate credentials into the NFS support, however, so you will need to make sure that you lock down your NFS shares in other ways.

Linux File Level Restores
I know a number of people who are excited about this one. You've always been able to do FLR on Windows VMs, and now you can do the same with LinuxVMs.

Catalogs
You now have the option to catalog your file and directory structure during the backup process, rather than having to wait for it to propagate during your restore process. Awesome thing if you are always doing the type of restore where someone is sitting over your shoulder because they needed the file (that they deleted, coincidentally) yesterday; now they can't scream at you for it taking too long. My concern is that it might increase the backup window (one of the big selling points for vRanger for me is the speed) - I'm going to have to test it out and see.

LAN Free Restores over Fibre
Allows you do restores on your fibre network without touching the bandwidth on your lan.


What's Missing

Unfortunately, I notice several key features missing that were promised, that we will probably have to wait for vRanger 6.0 (current due date 2012) until they are released:

Direct-to-Target backups from ESXi (it has to backup using the vRanger box as a proxy) via a Virtual Appliance. You can currently do Direct-to-Target from ESX, but due to the lack of service console on ESXi, it will need a VA.

FlashRestore, which allows you to start running a VM before it finishes restoring. Very cool for those who are under a time crunch (wait, I believe that's the whole IT industry).


If you want to read more, or download a trial, you can from the Quest vRanger Website.

------
Dustin Shaw
VCP

Speed, Speed, Speed

Who would've ever thought you'd see network performance like this on an Windows box:

This is the networking graph I got off of a physical box running the inital push for Quest vConverter's Continuous Protection option. The blip in the middle was it switching from the C: volume to the D: volume. It just about knocked my socks off - I've never seen anything eat up the entire NIC on Windows.

I found out some really cool stuff with vConverter. With the Continuous Protection option, you can keep a full-image copy of your physical server on an NTFS Share, ESX(i) Server, or Hyper-V server. After the first push, it just copies block (or file level) changes (whichever option you want) to the target store. This way, if you ever lose your physical box, you can either immediately pop it online in your ESX/Hyper-V box, or do a V2P back to your original hardware once it's back up and running. Very cool stuff.

Even cooler - let's say you're looking at Offsite DR. You already have Virtual Servers (VMware or Microsoft) in a hardened data center, and are currently replicating your VMs at your main office offsite to there. Good for you! Now how do you push your physical boxes out there?

With vConverter, you can do your initial copy locally (say to an NTFS Share on a NAS device), then you can take it to your Datacenter, and upload it to your SAN that your ESX or Hyper-V boxes use. Then go into vConverter, do a "change target" and select your ESX or Hyper-V box, tell it where the data is, and your good to go. Now all you have to worry about getting across the WAN is your block changes. Now you really do have an Offsite DR solution.

------
Dustin Shaw

VCP

vFoglight 6.5 is out

Quest vFoglight 6.5 was released today and is available for download from Quest's site.

For those who aren't familiar, vFoglight is Performance Management software for Virtual Environments (both VMware and Hyper-V).

Here's a breakdown of the new features available in vFoglight 6.5, as highlighed by Ben Scheerer:

Release Highlights
With Version 6.5, Quest vFoglight is focused on addressing specific virtual management needs by:

• Reducing time and effort required to manage and fix problems in the virtual infrastructure through built-in automation, workflow based alert remediation and in-context administration
• Easily presented user centric views “perspectives” that provides a single view giving a complete understanding of the infrastructure status 
• Leveraging additional hypervisor technologies to more seamlessly fit into existing virtualized environments

vFoglight v6.5 includes new and improved capabilities across performance monitoring, administration, remediation, capacity planning, chargeback, and service management in a single, downloadable solution.

• Only to offer a unique visual paradigm using color coding and movement to direct administrators to problem areas
• Reduces time and effort required to manage the virtual infrastructure through built-in automation, workflow based alert remediation and in context administration
• Capacity planning capabilities quickly identifies bottlenecks, predict capacity failures, provide scenario models and identifies candidates for optimization
• Understand how the infrastructure is supporting key applications and business services including the ability to associate costs for the use of infrastructure
• Supports heterogeneous hypervisor platforms including VMware and Hyper-V
• New support for Exchange Server 2010 (Foglight for Exchange)

------
Dustin Shaw
VCP

vConverter 5.1 is out!

Yesterday Quest released vConverter 5.1 onto their website for download. You can check out the product here.

For those of you familiar with their product, you'll know that vConverter 4.2 was the previous version that was out. They recently released vConverter 5.0 as an "early release preview" (AKA - Beta), but it had numerous issues. One tech that I spoke with said that the marketing team put version 5 online for download and issued a press release before they consulted with tech or engineering - sounds like many marketing guys that I know.

If you were one of the lucky ones that installed the "early release preview," then you'll have to uninstall it before you install vConverter 5.1.

The new features to look for in version 5.1 are:

P2V for Linux Servers: this was a big absent in the previous releases. I know many penguin fans who are excited about this one. It's a Cold Convert, so you have to be down when doing the conversion and have physical access to the box so that you can boot off a CD or PXE server (unless of course you have ILO or DRAC or other console access). This is unfortunate since VMware's vCenter Converter 4.x supports several Linux builds doing Hot Convert, but there are a number of other features that vConvert does buy you.

ESXi Support: This is a good thing since VMware is pushing everyone hard away from ESX onto ESXi. They are doing this by using a Virtual Appliance.

Hidden Partitions Support: This has recently become an issue with Windows 7 and Server 2008r2, keeping most converter programs from being able to work. I'm glad to see this added, as if you've been reading my blog you know that I recently ran into an issue caused from someone doing a conversion of a 2008r2 box using Acronis.

vConverter 5.1 download comes with a 30 day trial license that allows you to use Continuous Protection on up to 5 source machines, and limits you to 5 conversions (not including the Continuous Protection conversions).

------
Dustin Shaw
VCP

vRanger Error

Ran across this error on a client running vRanger:

Message: Failure creating VM datastore lock Host: ESXHostNameha-host VM: MVName

After investigating, the root of the issue was that the ESX host had run out of memory - there were too many VMs running on this host. This was keeping the VMs from quiescing long enough to get a good snapshot. When they would try, the memory would spike and balloon all over the board. After thinning out some memory, the problem disappeared.

This particular client is growing faster than they were paying attention to, hence the increase in memory utilization across the board.The client is ordering more memory for all the hosts in their cluster and looking at ordering another host or two so that they can get back to N+1 (where they actually started at).

------
Dustin Shaw
VCP
------
Here's some books that should help out:
VMware vSphere 4 Administration Instant Reference
VMware Fusion 3
VMware Cookbook: A Real-World Guide to Effective VMware Use

vRanger Error

Here's a random vRanger Error:

Message: An internal error occurred during execution, please contact Vizioncore support if the error persists.  Error Message: VM's host could not be identified

Got this on one of my client's install's today. vRanger 4.x.

The solution? Go refresh the inventory. Go edit your backup job and keep all the same settings.

Easy fix.

-Update-

I ran into the error again on vRanger 5.3.x, and posted an update here.

------
Dustin Shaw
VCP

VM Shuts Off When Taking Snapshot

Here's a really cool one :-)

Have a client that just did a P2V - they took all of their 5 servers and did a P2V of them into a new vSphere implementation. Naturally, they also purchased vRanger so that they could get the best bang for their buck.

They had been continuing to backup using their good old Symantec method after the P2V project, and just now got around to the point of switching over to vRanger. They're a small shop, so they can take a while to move from Point A to Point B.

Well, vRanger ran for the first time last night, and kicked out the following errors on two VMs:

Message: An internal error occurred during execution, please contact Vizioncore support if the error persists.  Error Message: API Call failed with message: Error encountered while restarting virtual machine after taking snapshot.  The virtual machine will be powered off.

Took a quick glance, and remembered that these two boxes were the ones that were already running 2008r2 before the P2V. Because of this, we couldn't use the standard VMware Standalone Converter that we typically use for small shops - it doesn't support 2008r2. Instead we used Acronis to do the P2V using their True Image Echo software. 

Not my first choice, and not my decision either. I didn't even get to play on this project... One of my other techs kept saying "Can I do one? Can I do one? You get to have all the fun! Can I do it?" Obviously I got tired, and said sure; plus I wanted a weekend off to spend with the family. I found out after words what happened, but they were up and running and happy, so I didn't bother following up.

Think it's a coincidence? Not so, according to Quest's tech support - the second call my guys made to get the VMs back up (the first was to me: "HELP!" "I'm at a VMware View Presentation in front of 10 prospective companies. Call Quest or VMware; they have support on both." "Right!")

Apparently, when the Acronis P2V was done, it split up the vmdk files into 2GB chunks, rather than one big one. Not sure if this was an option they picked or if it was default - if the guy who did it was still working for us I would ask... He's gone for different performance reasons.

When the vmdk file is like this on vSphere, snapshotting doesn't work right, and kicks the box off line.

So the fix is to run vmkfstools to convert the multiple vmdk files down to one, then you're good to go. Alternatively, you can just Clone the VM to let it consolidate them for you, then remove the old one.

Ain't life fun!

Next time I'm going to verify the work done a little closer...

------
Dustin Shaw
VCP
------
Here's some books that should help out:
Maximum vSphere: Tips, How-Tos, and Best Practices for Working with VMware vSphere 4
VCP VMware Certified Professional on vSphere 4 Study Guide: Exam VCP-410
VMware vSphere 4 Implementation

We'll miss you ESX!

For those of you who haven't been paying attention in VMware, with the advent of vSphere (over a year ago), the official recommendation is to no longer use ESX, but now to use ESXi.

Why would I do something silly like that?

I'm glad you asked :-)

ESXi is lighter - not so heavy on the host box
No security console - aka, less security risk and things to patch
Less configuration options - easier to setup a new host

What are the downsides?


Oh, those... Well, if you believe VMware, none! If you listen to third party vendors, then you will hear a long list of agents that they've had to scramble to figure out new methods of doing. Some (like Quest's vRanger Pro) are switching to a VA (Virtual Appliance) format. Others are just pulling from the remote API calls. And others (like HP Systems Insight Manager or Dell OpenManage) are switching their management tools to use SNMP and CIM based calls for managing the hardware.

Why can't I just keep using ESX?

Oh, that's because ESX is going to disappear in "future major releases," so why wait until you are forced into the situation. Try to be a little forward looking; after all, isn't that why you're Virtualized in the first place? :-)

If you want to read more from VMware, you can go here:
http://www.vmware.com/products/vsphere/esxi-and-esx/overview.html
http://www.vmware.com/products/vsphere/esxi-and-esx/faqs.html

------

UPDATE 3/29/11

VMware made it official that as of vSphere 5 (coming out July-August 2011), ESX will no longer be included. ESXi will be the only way to go. Better start migrating!

Read more about it here.

------
Dustin Shaw
VCP

vRanger Recommended Settings

I had the pleasure of sitting through one of Quest's (formerly Vizioncore) webinars today, and had a chance to review all of the best practices that they have with vRanger 4.5.

One thing that always stands out is the settings in vSphere 4.x for the Service Console. You should set 1500Mhz CPU reservation and 800Mb memory reservation. This will give you optimal performance on your hosts for the best/quickest backups. Then you also need to make sure you adjust your settings inside vRanger to match what your equipment can handle (how many jobs one host can handle, how much throughput your data store can take), so that you can do what vRanger does best and push data as fast and hard as you can to your backup store. That is, unless you are a 24 hour shop; then you might want to throttle it, depending on how cheap your equipment is :-)

This obviously only applies to ESX. ESXi doesn't allow vRanger to get in and work the same way, so they have some other fancy tricks for ESXi using the vStorage API. You can read about it here:
http://vcommunity.vizioncore.com/dataprotection/vrangerpro/b/backup20/archive/2010/09/02/scalable-and-high-performance-data-protection-on-esxi-backup-replication-and-recovery.aspx

Cool stuff. And it all works awesome. I ran across a comparison chart the other day in Quest's blogs that show that vRanger outperforms competitors by 3-4x on LAN-based backups, and by 1.75x on LAN-free backups. Check it out:
http://vcommunity.vizioncore.com/dataprotection/vrangerpro/b/backup20/archive/2010/07/28/more-performance-data-for-vranger-pro-4-5-unrivaled-backup-speed-in-real-world-tests.aspx

And in the interest of full disclosure, I don't work for Quest. I'm actually an agnostic when it comes to this (as I am with most software; I like what works, don't like what doesn't), and have implemented and managed multiple different VM backup software - including Acronis and Symantec. They all have their good sides and their bads (some more than others) but if I have a client that prefers a particular product due to loyalty, existing service contract, or the classic "I am already familiar with their console," I'm happy to help them go with whatever they want. Ultimately, it's their job, not mine, that's on the line if the product doesn't work as advertised by the manufacturer - and I always list the caveats so that they are aware going in.

------
Dustin Shaw
VCP
------
Here's some books that should help out:
Mastering VMware vSphere 4 (Computer/Tech)
VMware vSphere 4 Administration Instant Reference
VMware vSphere 4 Implementation