vRanger 5 and ESXi Performance Settings

Everyone should know that the previous recommendations from Quest were to increase the CPU Reservations to 1500 on ESX servers for the best performance from vRanger.

I just recently ran across an article (here) from Sept 2011 detailing that this is NOT the case for ESXi. There is no performance gain on ESXi by increasing the CPU reservations from the default 293.

------
Dustin Shaw
VCP

vRanger error "There is an error in XML document"

I ran into an issue with one of our repositories - every job in the particular repository was failing with the error:

An internal error occurred during execution, please contact Quest support if the error persists. Error Message: There is an error in XML document (823, 223).

This error is commonly caused by a corruption or issue with the manifest file as detailed here.

To fix it, go into the manifest file for the repository (GlobalManifest.metadata), make sure that all trailing brackets are closed, and add </vRangerGlobalManifestFile> at the end of the document.

Save and test.

------

Dustin Shaw
VCP

Installing PowerCLI on Server 2008

In attempting to install VMware vSphere PowerCLI on Windows Server 2008 x64, the following error comes up. This also happens on Server 2008R2.

Error 1406. Could not write value InstallPath to key \Software\VMware, Inc.\VMware vSphere PSDK Runtime. Verify tha tyou have sufficient access to that key, or contact your support personnel.

The prescribed fix for this is to remove the following registry key (and subkeys):
HKLM\Software\Wow6432Node\VMware, Inc.

*** Please make sure you know what you are doing in the registry before you do anything in there!!!

After that, installation proceeds successfully.

The only thing under the VMware, Inc. registry key is a "volatile" key with a UUIDHost DWord under it. Since I don't like to mess with other programs installed on the server (the one I was using has Quest vRanger installed), I exported the key, removed it, installed PowerCLI, and the imported my volatile key back in.

------
Dustin Shaw
VCP

Move vRanger Repository

I needed to move a vRanger Repository, but didn't want to start all my backups over, so I found a handy article from Quest that identifies how to do it.

FYI, I believe it requires vRanger 5.3 and up.
The basics are you go into SQL Management Studio, expand down to dbo.Repository to identify the repository, and change the host or target directory. Then go to dbo.RepositoryCIFS to change the sharename.

Make sure you restart the vRanger Services after making the changes.

------
Dustin Shaw
VCP

vRanger "host could not be identified"

We had our vCenter sever hiccup on us yesterday, so (naturally) our vRanger backups failed last night.

The error we received was:

An internal error occurred during execution, please contact Quest support if the error persists.  Error Message: VMSERVER's host could not be identified

Essentially what causes this issue is a communication sync error between vRanger and vCenter. I ran across this error a year and a half ago running on vRanger 4.x (you can read my post here). This time it was on Quest vRanger version 5.3.0, so I did a quick search to see if there were any newer solutions for verison 5. It appears that Quest now has a KB article about it.

Basically, their solution is to restart the services - stop the API and FLR Services, restart the vRanger service, then start up the API and FLR services.

I always prefer to go above, so I also refreshed the jobs (like my previous experience taught me) by opening them up and click next and finish all the way through (without changing anything).

Next time I'll remind my guys to restart vRanger services whenever vCenter bounces.

------
Dustin Shaw
VCP

vRanger 5.0 is released!

As of January 31, 2011, Quest vRanger 5.0 is now publicly available and out of beta.

There are now two versions of vRanger available: vRanger Standard Edition 5.0 and vRanger Pro 5.0. Pro buys you these additional features that people have been looking forward to: Replication, LAN Free, Catalog, and Linux FLR. Here's a look at them:

vReplicator is integrated into vRanger Pro
You no longer have to buy two separate products to do backups and replication if you choose to go the Quest route. You can use one single product for both.

Support for NFS and FTP Repositories
We already had support for CIFS and SFTP targets. The addition of NFS and FTP gives more flexibility on what you can use as a backup target. They didn't integrate credentials into the NFS support, however, so you will need to make sure that you lock down your NFS shares in other ways.

Linux File Level Restores
I know a number of people who are excited about this one. You've always been able to do FLR on Windows VMs, and now you can do the same with LinuxVMs.

Catalogs
You now have the option to catalog your file and directory structure during the backup process, rather than having to wait for it to propagate during your restore process. Awesome thing if you are always doing the type of restore where someone is sitting over your shoulder because they needed the file (that they deleted, coincidentally) yesterday; now they can't scream at you for it taking too long. My concern is that it might increase the backup window (one of the big selling points for vRanger for me is the speed) - I'm going to have to test it out and see.

LAN Free Restores over Fibre
Allows you do restores on your fibre network without touching the bandwidth on your lan.


What's Missing

Unfortunately, I notice several key features missing that were promised, that we will probably have to wait for vRanger 6.0 (current due date 2012) until they are released:

Direct-to-Target backups from ESXi (it has to backup using the vRanger box as a proxy) via a Virtual Appliance. You can currently do Direct-to-Target from ESX, but due to the lack of service console on ESXi, it will need a VA.

FlashRestore, which allows you to start running a VM before it finishes restoring. Very cool for those who are under a time crunch (wait, I believe that's the whole IT industry).


If you want to read more, or download a trial, you can from the Quest vRanger Website.

------
Dustin Shaw
VCP

vRanger Error

Ran across this error on a client running vRanger:

Message: Failure creating VM datastore lock Host: ESXHostNameha-host VM: MVName

After investigating, the root of the issue was that the ESX host had run out of memory - there were too many VMs running on this host. This was keeping the VMs from quiescing long enough to get a good snapshot. When they would try, the memory would spike and balloon all over the board. After thinning out some memory, the problem disappeared.

This particular client is growing faster than they were paying attention to, hence the increase in memory utilization across the board.The client is ordering more memory for all the hosts in their cluster and looking at ordering another host or two so that they can get back to N+1 (where they actually started at).

------
Dustin Shaw
VCP
------
Here's some books that should help out:
VMware vSphere 4 Administration Instant Reference
VMware Fusion 3
VMware Cookbook: A Real-World Guide to Effective VMware Use

vRanger Error

Here's a random vRanger Error:

Message: An internal error occurred during execution, please contact Vizioncore support if the error persists.  Error Message: VM's host could not be identified

Got this on one of my client's install's today. vRanger 4.x.

The solution? Go refresh the inventory. Go edit your backup job and keep all the same settings.

Easy fix.

-Update-

I ran into the error again on vRanger 5.3.x, and posted an update here.

------
Dustin Shaw
VCP

VM Shuts Off When Taking Snapshot

Here's a really cool one :-)

Have a client that just did a P2V - they took all of their 5 servers and did a P2V of them into a new vSphere implementation. Naturally, they also purchased vRanger so that they could get the best bang for their buck.

They had been continuing to backup using their good old Symantec method after the P2V project, and just now got around to the point of switching over to vRanger. They're a small shop, so they can take a while to move from Point A to Point B.

Well, vRanger ran for the first time last night, and kicked out the following errors on two VMs:

Message: An internal error occurred during execution, please contact Vizioncore support if the error persists.  Error Message: API Call failed with message: Error encountered while restarting virtual machine after taking snapshot.  The virtual machine will be powered off.

Took a quick glance, and remembered that these two boxes were the ones that were already running 2008r2 before the P2V. Because of this, we couldn't use the standard VMware Standalone Converter that we typically use for small shops - it doesn't support 2008r2. Instead we used Acronis to do the P2V using their True Image Echo software. 

Not my first choice, and not my decision either. I didn't even get to play on this project... One of my other techs kept saying "Can I do one? Can I do one? You get to have all the fun! Can I do it?" Obviously I got tired, and said sure; plus I wanted a weekend off to spend with the family. I found out after words what happened, but they were up and running and happy, so I didn't bother following up.

Think it's a coincidence? Not so, according to Quest's tech support - the second call my guys made to get the VMs back up (the first was to me: "HELP!" "I'm at a VMware View Presentation in front of 10 prospective companies. Call Quest or VMware; they have support on both." "Right!")

Apparently, when the Acronis P2V was done, it split up the vmdk files into 2GB chunks, rather than one big one. Not sure if this was an option they picked or if it was default - if the guy who did it was still working for us I would ask... He's gone for different performance reasons.

When the vmdk file is like this on vSphere, snapshotting doesn't work right, and kicks the box off line.

So the fix is to run vmkfstools to convert the multiple vmdk files down to one, then you're good to go. Alternatively, you can just Clone the VM to let it consolidate them for you, then remove the old one.

Ain't life fun!

Next time I'm going to verify the work done a little closer...

------
Dustin Shaw
VCP
------
Here's some books that should help out:
Maximum vSphere: Tips, How-Tos, and Best Practices for Working with VMware vSphere 4
VCP VMware Certified Professional on vSphere 4 Study Guide: Exam VCP-410
VMware vSphere 4 Implementation

We'll miss you ESX!

For those of you who haven't been paying attention in VMware, with the advent of vSphere (over a year ago), the official recommendation is to no longer use ESX, but now to use ESXi.

Why would I do something silly like that?

I'm glad you asked :-)

ESXi is lighter - not so heavy on the host box
No security console - aka, less security risk and things to patch
Less configuration options - easier to setup a new host

What are the downsides?


Oh, those... Well, if you believe VMware, none! If you listen to third party vendors, then you will hear a long list of agents that they've had to scramble to figure out new methods of doing. Some (like Quest's vRanger Pro) are switching to a VA (Virtual Appliance) format. Others are just pulling from the remote API calls. And others (like HP Systems Insight Manager or Dell OpenManage) are switching their management tools to use SNMP and CIM based calls for managing the hardware.

Why can't I just keep using ESX?

Oh, that's because ESX is going to disappear in "future major releases," so why wait until you are forced into the situation. Try to be a little forward looking; after all, isn't that why you're Virtualized in the first place? :-)

If you want to read more from VMware, you can go here:
http://www.vmware.com/products/vsphere/esxi-and-esx/overview.html
http://www.vmware.com/products/vsphere/esxi-and-esx/faqs.html

------

UPDATE 3/29/11

VMware made it official that as of vSphere 5 (coming out July-August 2011), ESX will no longer be included. ESXi will be the only way to go. Better start migrating!

Read more about it here.

------
Dustin Shaw
VCP

vRanger Recommended Settings

I had the pleasure of sitting through one of Quest's (formerly Vizioncore) webinars today, and had a chance to review all of the best practices that they have with vRanger 4.5.

One thing that always stands out is the settings in vSphere 4.x for the Service Console. You should set 1500Mhz CPU reservation and 800Mb memory reservation. This will give you optimal performance on your hosts for the best/quickest backups. Then you also need to make sure you adjust your settings inside vRanger to match what your equipment can handle (how many jobs one host can handle, how much throughput your data store can take), so that you can do what vRanger does best and push data as fast and hard as you can to your backup store. That is, unless you are a 24 hour shop; then you might want to throttle it, depending on how cheap your equipment is :-)

This obviously only applies to ESX. ESXi doesn't allow vRanger to get in and work the same way, so they have some other fancy tricks for ESXi using the vStorage API. You can read about it here:
http://vcommunity.vizioncore.com/dataprotection/vrangerpro/b/backup20/archive/2010/09/02/scalable-and-high-performance-data-protection-on-esxi-backup-replication-and-recovery.aspx

Cool stuff. And it all works awesome. I ran across a comparison chart the other day in Quest's blogs that show that vRanger outperforms competitors by 3-4x on LAN-based backups, and by 1.75x on LAN-free backups. Check it out:
http://vcommunity.vizioncore.com/dataprotection/vrangerpro/b/backup20/archive/2010/07/28/more-performance-data-for-vranger-pro-4-5-unrivaled-backup-speed-in-real-world-tests.aspx

And in the interest of full disclosure, I don't work for Quest. I'm actually an agnostic when it comes to this (as I am with most software; I like what works, don't like what doesn't), and have implemented and managed multiple different VM backup software - including Acronis and Symantec. They all have their good sides and their bads (some more than others) but if I have a client that prefers a particular product due to loyalty, existing service contract, or the classic "I am already familiar with their console," I'm happy to help them go with whatever they want. Ultimately, it's their job, not mine, that's on the line if the product doesn't work as advertised by the manufacturer - and I always list the caveats so that they are aware going in.

------
Dustin Shaw
VCP
------
Here's some books that should help out:
Mastering VMware vSphere 4 (Computer/Tech)
VMware vSphere 4 Administration Instant Reference
VMware vSphere 4 Implementation