The wizard was interrupted before VMware Tools could be completely installed.

Upon attempting to install VMware Tools on a Windows Server 2008R2 server, I received an error stating "The wizard was interrupted before VMware Tools could be completely installed."

After looking around on the internet, the closest thing I could find was this post by David Homer, detailing a similar issue with VMware Tools on VMware Workstation.

I tried his fixes (remove VMware Tools registry key, remove the vmtools service), but none of them allowed me to get past the problem.

What eventually worked was doing a search in the registry for all references to "vmware" and removing all keys having to do with VMware Tools - make sure you don't remove any of the keys referring to your SCSI devices/other hardware; just the VMware Tools entries.

I believe it was most likely the Installer registry entries that were mucking it up.

After a quick reboot to refresh the registry, VMware Tools installed with no issues.

------
Dustin Shaw
VCP

Install VMware Tools on RHEL 6 or CentOS 6

Below is the process to install VMware Tools on RHEL 6 or CentOS 6. This guide is more here for me than anyone else, but I hope that you can benefit from it.


Install the Pre-Requisites:
yum install make gcc kernel-devel kernel-headers glibc-headers perl


Start the VMware Tools installation process on your VM:





Mount the VMware Tools installation media:
mkdir /mnt/cd
mount /dev/cdrom /mnt/cd
Expected warning:
mount: block device /dev/sr0 is write-protected, mounting read-only


Extract the installer:
cp /mnt/cd/VMwareTools-9.0.10-1481436.tar.gz /tmp/
umount /mnt/cd
cd /tmp
tar xvf VMwareTools-9.0.10-1481436.tar.gz
cd vmware-tools-distrib/


Install tools (accepting all defaults):
sudo ./vmware-install.pl -d

Reboot the VM to verify that the service starts up automatically as expected.
shutdown -r now


------
Dustin Shaw
VCP

Extend a volume in RedHat 6 or CentOS 6

To extend a partition on a VM on CentOS 6 or RHEL 6, do the following:

Extend the Partition in vSphere to the size that you need:

in the VM, refresh the Partition Tables by the following command:

echo 1 > /sys/block/sda/device/rescan
Run fdisk to create a new primary partition:

fdisk /dev/sda

Press p to print the number of partitions
Press n to create a new primary patition
Press p for primary
The partition number should automatically be selected, and press enter twice to accept the beginning and ending blocks on the free space.
Press w to write the changes.

Reboot the system: shutdown -r now

Verify that the new partition exists:
fdisk -l

Convert the new partition to a physical volume:
pvcreate /dev/sda3
Extend the Physical Volume (run vgdisplay to obtain the name of the Volume Group; default is VolGroup00):
vgextend VolGroup00 /dev/sda3

Extend the Logical Volume (run vgdisplay to obtain the free space and replace # below, and lvdisplay to obtain the name of the Logical Volume; default is /dev/VolGroup00/LogVol00)
lvextend -L+#G /dev/VolGroup00/LogVol00

Expand the ext3 online using the following command (substituting your proper Volume Group and Logical Volume names):
resize2fs /dev/VolGroup00/LogVol00

Confirm that your new space is available:
df -h



This was written using VMware's KB article on the subject as a reference.

------
Dustin Shaw
VCP

vRanger 5 and ESXi Performance Settings

Everyone should know that the previous recommendations from Quest were to increase the CPU Reservations to 1500 on ESX servers for the best performance from vRanger.

I just recently ran across an article (here) from Sept 2011 detailing that this is NOT the case for ESXi. There is no performance gain on ESXi by increasing the CPU reservations from the default 293.

------
Dustin Shaw
VCP

vCenter Server 5 Service Fails

We had an issue with the vCenter Server 5 Service failing recently. Basically what happened was the VMware VirtualCenter Server service failed (out of the blue) with the following Informational Event ID 1000 logged in the Application log:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Starting VMware VirtualCenter 5.0.0 build-623373

the message resource is present but the message is not found in the string/message table

Followed by another Info Event 1000:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Log directory: C:\ProgramData\VMware\VMware VirtualCenter\Logs.

the message resource is present but the message is not found in the string/message table

And followed by an Error Event 1000 (upon it attempting to auto-restart the service):

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Failed to intialize VMware VirtualCenter. Shutting down...

the message resource is present but the message is not found in the string/message table

First try was to restart all the vCenter services - only the VMware VirtualCenter Server service was offline; all other services (like VMware VirtualCenter Management Webservices) were still online. This obviously failed with the same error as before.

All of the SQL services were verified online (these are kept on a separate server due to the size), and accessible. I ran across a similar Discussion thread in VMware Communities, which pointed to issues inside of the SQL Database. Based off of this, I decided the first order was to look at my SQL server to see what was going on there. I was able to log in, and verified that everything was up. Then I looked at my logs, and saw Event ID 17053:

C:\VCDB.ldf: Operating system error 112(There is not enough space on the disk.) encountered.

 And instantly I knew my problem. Some yahoo (namely the yahoo writing this article) must not've been paying attention when installing the VCDB database, and stuck it in the root of the C drive. Naturally, that yahoo had to fix his own problem...

So I went into Microsoft SQL Server Management Studio, and ran the following command:

alter database VCDB modify file ( name = vcdb , filename = 'E:\SQL\MDF\VCDB.mdf' )
alter database VCDB modify file ( name = vcdb_log , filename = 'E:\SQL\LDF\VCDB.ldf')
go

Then I Offlined the database (Right click, Tasks, Take Offline), and moved the files to their new homes. Then I Onlined the database, and verified that the path was correct for it. You can see the full process of how to move a SQL database here.


Once this was done, I went back to my vCenter Server, and was able to bring all my services online without incident, and was able to again go in and manage my vCenter 5 Server via vSphere Client.

------
Dustin Shaw
VCP

WNLB and VMware

So I believe I've found an (known) issue with 2003 Windows Network Load Balancing and VMware.
VMware reports that WNLB on Windows 2003 Servers does not behave as expected here. Basically the article says the the NLB will point to one of the servers, not all of them, when running in unicast. They give you two fixes: use multicast; or reconfigure your Port Groups (or vSwitches) to prevent RARP Packet Transmissions. Interesting thing, with the current environment these servers are hosted in, I am unable to either run multicast or disable Switch Notify. I'll have to take that up with the Network Team, or perhaps investigate some NLB hardware.

So here's where I'm left with - these servers are not supposed to go down (hence the NLB), but they are everyday at 4AM (fun call). This is when backups are running, so I'm adjusting the time to see if the issue follows.

What appears to be happening is that when snapshots are taken for backups, the NLB seems to freak out at the one dropped ping. Currently the backups all run at once, which makes them all hiccup at the same time, killing the NLB for a good reported 45 minutes (no idea why so long). If the issue follows the backups, perhaps staggering the backups might solve the problem (let the NLB roll from one server to another).

I'll keep you posted on what I find.
------
Dustin Shaw
VCP

Host update fails after updating to 4.1u2

After updating vCenter Update Manager (right after updating my vCenter Server) from 4.1u1 to 4.1u2, I received the following error when trying to update on of my hosts:

Remediation did not succeed for esxihost: SingleHostRemediate: esxupdate error, version: 1.30, operation: 7: ('http://esxihost:9084/vci/hostupdates/hostupdate/vmw/vibs/cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1OEM.vib','/var/tmp/cache/-1699692350','[Errno 14] HTTP Error 404: Not Found')

After doing some research, I discovered what happened is that Update Manager 4.1u2 is case sensitive, whereas 4.1u1 was not. Any patches that were downloaded previous to the 4.1u2 update that contain upper case letters will fail with this error. VMware has a KB article about it here.

The patches affected are the ones below. I've put the correct capitalization on them for you. If you go to your patch repository, you can rename the files that you have to the below, and you should then be able to update your hosts. The repositories are located here:

• Windows2008: C:\ProgramData\VMware\VMware Update Manager\Data\Hostupdate\vmw\vib\
• Windows2003: C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Data\Hostupdate\vmw\vib

bind-libs-9.3.6-4.P1.el5_5.3.i386.vib
bind-libs-9.3.6-4.P1.el5_5.3.x86_64.vib
bind-utils-9.3.6-4.P1.el5_5.3.x86_64.vib
bind-libs-9.3.6-4.P1.el5_5.3.i386.vib
cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1OEM.vib
cross_oem-vmware-esx-drivers-scsi-3w-9xxx_400.2.26.08.036vm40-1OEM.vib
vmware-esx_swMgmt_provider-4x.1.0.1-1.4.348481.vib


------
Dustin Shaw
VCP

Syslog not configured on ESXi 4.1u2

When I updated my ESXi host from 4.1u1 to 4.1u2, I got the following error message:

Configuration Issues
Issue detected on esxihost in datacenter: Warning: Syslog not configured. Please check Syslog options under Configuration.Software.Advanced Settings in vSphere client.

I thought it was odd since the host previously never complained about the Syslog before. I compared the settings between it and the other 4.1u2 hosts that I had, and indeed, it was missing the Syslog.Local.DatastorePath setting.

The setting on my hosts was:
[] /scratch/log/messages

Once I copied this into my Syslog.Local.DatastorePath setting on the server, it was happy. I went ahead and copied the setting to my remaining 4.1u1 servers so that they will be happy when updated as well.

So apparently ESXi 4.1u2 has issues with the Syslog running on ramdisk, but ESXi 4.1u1 doesn't.

I found the following VMware KB Article that explains why it complains about it:
Syslog not configured messages on ESXi host console or in logs

------
Dustin Shaw
VCP

Installing PowerCLI on Server 2008

In attempting to install VMware vSphere PowerCLI on Windows Server 2008 x64, the following error comes up. This also happens on Server 2008R2.

Error 1406. Could not write value InstallPath to key \Software\VMware, Inc.\VMware vSphere PSDK Runtime. Verify tha tyou have sufficient access to that key, or contact your support personnel.

The prescribed fix for this is to remove the following registry key (and subkeys):
HKLM\Software\Wow6432Node\VMware, Inc.

*** Please make sure you know what you are doing in the registry before you do anything in there!!!

After that, installation proceeds successfully.

The only thing under the VMware, Inc. registry key is a "volatile" key with a UUIDHost DWord under it. Since I don't like to mess with other programs installed on the server (the one I was using has Quest vRanger installed), I exported the key, removed it, installed PowerCLI, and the imported my volatile key back in.

------
Dustin Shaw
VCP

ESXi Host Unable to Update

I had a VMware ESXi 4.1 host that I was unable to update/patch using Update Manager the today.

It was coming up with the following errors when I tried:

VMware vCenter Update Manager had an unknown error. Check the Tasks and Events tab and log files for details.

And when I did that, I found this:

Could not install patches on esxihostname
Remediation did not succeed for esxihostname: SingleHostRemediate: Install error on host: esxihostname, error details: vim.fault.NoHost.

So I did some quick googling, and ran across this website that had the answer I was looking for.

The particular host that I was trying to update did indeed have an Unknown (inaccessible) VM on it (left over from yanking the storage and being sloppy...). I removed the VM from inventory, then was able to successfully patch the host.

------
Dustin Shaw
VCP

Move vRanger Repository

I needed to move a vRanger Repository, but didn't want to start all my backups over, so I found a handy article from Quest that identifies how to do it.

FYI, I believe it requires vRanger 5.3 and up.
The basics are you go into SQL Management Studio, expand down to dbo.Repository to identify the repository, and change the host or target directory. Then go to dbo.RepositoryCIFS to change the sharename.

Make sure you restart the vRanger Services after making the changes.

------
Dustin Shaw
VCP

Datastore Alarm Not Accurate

I had the unfortunate mishap of a VM dropping offline because it outgrew the amount of free space on the datastore it existed on. On the plus side, it happened in the middle of the night and I wasn't on call :-) The on call guy simply had to reboot the box, and it came back online.
When I got in and looked, the datastore that it existed on had 0.00 B of free space. Not very comforting, considering the fact that there were no alarms on that datastore, and there were several other Production VMs on that LUN.
To fix the alarm state, it was easy. Go into the alarm definition and disable and re-enable it. This will force it to refresh it's states and you will get to see all the pretty yellow and red icons that you've been missing.

This also will work if it is reporting a Usage Above Threshold before it should too.

------
Dustin Shaw
VCP

vRanger "host could not be identified"

We had our vCenter sever hiccup on us yesterday, so (naturally) our vRanger backups failed last night.

The error we received was:

An internal error occurred during execution, please contact Quest support if the error persists.  Error Message: VMSERVER's host could not be identified

Essentially what causes this issue is a communication sync error between vRanger and vCenter. I ran across this error a year and a half ago running on vRanger 4.x (you can read my post here). This time it was on Quest vRanger version 5.3.0, so I did a quick search to see if there were any newer solutions for verison 5. It appears that Quest now has a KB article about it.

Basically, their solution is to restart the services - stop the API and FLR Services, restart the vRanger service, then start up the API and FLR services.

I always prefer to go above, so I also refreshed the jobs (like my previous experience taught me) by opening them up and click next and finish all the way through (without changing anything).

Next time I'll remind my guys to restart vRanger services whenever vCenter bounces.

------
Dustin Shaw
VCP

V2V from Hyper-V to VMware ESXi

So you finally came to the realization that Microsoft Hyper-V really isn't as mature as you were lead to believe. Good! You can attend the Gullible Anonymous meetings later. First, let's get you over to something that can keep your servers up and running and performing like the $10k you spent on your hardware should provide you.

Assuming that you have a free server available, doing a V2V from Hyper-V to VMware ESXi is extremely easy. Just think of this as doing a P2V, and ignore the wasted resources in between...

Take one of your spare servers, and load up VMware ESXi on it (you can do 4.1 or 5, or whatever strikes your fancy). Get it all ready for the VM to run on it, and get it on the same network and gig switch as your Hyper-V box.

Login to the VM that you want to convert, and download and install VMware's Standalone Converter on it (current version is 5.0). Current supported platforms are:

Windows XP Professional (32-bit and 64-bit)

Windows Server 2003 SP2, R2 (32-bit and 64-bit)

Windows Vista (32-bit and 64-bit)

Windows Server 2008 (32-bit and 64-bit)

Windows Server 2008 R2 (64-bit)

Windows 7 (32-bit and 64-bit)

Next, you will want to run the Converter against the local machine, and convert it to ESXi server that you just created. There are several settings to change to verify that it's done properly:

• Prep your computer for a P2V by stopping ALL unneccessary services. You only need core services to make sure that it's successful.
• Select Powered-on machine; local machine

• Put in the IP/Hostname of your new ESXi server and root/password

• Give it a destination Virtual Machine name
• Select a Resource Pool (if used), datastore, and Virtual Machine hardware version (unless you have a driving reason, go with Version 7)
• Set your disk drives the way that you want them to be on the new environment - I prefer Thin on all my disks, but it's up to you and your application. To change this, Edit the "Data to copy," Advanced, and Destination Layout tab. Change the type to Thin

• Change the Disk controller to SCSI. To change this, Edit the "Devices" and go to the Other tab. I typically select SCSI LSI Logic SAS for newer OSes and SCSI LSI Logic for older OSes.

• Set all services that say Hyper-V to disabled. To do this, Edit the "Services" and go to the Destination services tab. The ones I came across on my most recent migration were (your mileage may vary based on the version/options you have):
• Hyper-V Heartbeat Service
• Hyper-V Data Exchange Service
• Hyper-V Guest Shutdown Service
• Hyper-V Time Synchronization Service
• Hyper-V Volume Shadow Copy Requestor
• Edit Advanced options and go to the Post-conversion tab and check "Install VMware Tools on the destination virtual machine"

Off it goes converting. Once it's done, power down your Hyper-V VM, power up the VMware VM, and wait for the VMware Tools to install. Once that's done, uninstall the VMware Converter, and your off and running.

Now you can wipe your Hyper-V server, load up ESXi on it, and build yourself a nice little redundant cluster.

------
Dustin Shaw
VCP

vSphere 5 is now available!

According to Yellow Bricks, it is now available for download. I'll update with links as soon as I get to a computer, but here is Yellow Bricks post.

------
Dustin Shaw
VCP

vSphere 5

A number of vSphere 5 official announcements came out today. You can check them out on VMware's official site.


------
Dustin Shaw
VCP

vSphere 5 Around the Corner!

VMware recently announced vSphere 5 will be coming July-August time frame 2011.

Three big new features will be included when it does come:

Storage DRS

Just like you use DRS to help you manage the workload on your hosts, Storage DRS will help you manage the workload on your SANs. The essential workup is that you'll be able to group your storage into storage pods that you can allow Storage DRS to manage what goes where based on capacity. Pretty cool, if you are dealing with some complicated storage situations.

SRM Host-Based Replication
This buys you the ability to use SRM even if you have disparing storage at your different locations.

Network I/O Control for VMs

This allows you the option to reserve bandwidth for high-priority VMs when you've got a congested network. Very useful control to have, depending on your network layout and configuration.


Another anouncement that VMware has made about vSphere 5 - it's ESXi all the way. That's right, ESX will be discontinued in favor of it's lighter, tigher cousin. I mentioned that this was coming before, but now it's actually here. Hope you are ready for migration!

VMware won't leave you figuring it out on your own, though. They recommend that you start migrating off ESX to ESXi now in preparation for the move to vSphere 5. Here is their Info Center about it, and here is their Migration Guide.
------
Dustin Shaw
VCP

vRanger 5.0 is released!

As of January 31, 2011, Quest vRanger 5.0 is now publicly available and out of beta.

There are now two versions of vRanger available: vRanger Standard Edition 5.0 and vRanger Pro 5.0. Pro buys you these additional features that people have been looking forward to: Replication, LAN Free, Catalog, and Linux FLR. Here's a look at them:

vReplicator is integrated into vRanger Pro
You no longer have to buy two separate products to do backups and replication if you choose to go the Quest route. You can use one single product for both.

Support for NFS and FTP Repositories
We already had support for CIFS and SFTP targets. The addition of NFS and FTP gives more flexibility on what you can use as a backup target. They didn't integrate credentials into the NFS support, however, so you will need to make sure that you lock down your NFS shares in other ways.

Linux File Level Restores
I know a number of people who are excited about this one. You've always been able to do FLR on Windows VMs, and now you can do the same with LinuxVMs.

Catalogs
You now have the option to catalog your file and directory structure during the backup process, rather than having to wait for it to propagate during your restore process. Awesome thing if you are always doing the type of restore where someone is sitting over your shoulder because they needed the file (that they deleted, coincidentally) yesterday; now they can't scream at you for it taking too long. My concern is that it might increase the backup window (one of the big selling points for vRanger for me is the speed) - I'm going to have to test it out and see.

LAN Free Restores over Fibre
Allows you do restores on your fibre network without touching the bandwidth on your lan.


What's Missing

Unfortunately, I notice several key features missing that were promised, that we will probably have to wait for vRanger 6.0 (current due date 2012) until they are released:

Direct-to-Target backups from ESXi (it has to backup using the vRanger box as a proxy) via a Virtual Appliance. You can currently do Direct-to-Target from ESX, but due to the lack of service console on ESXi, it will need a VA.

FlashRestore, which allows you to start running a VM before it finishes restoring. Very cool for those who are under a time crunch (wait, I believe that's the whole IT industry).


If you want to read more, or download a trial, you can from the Quest vRanger Website.

------
Dustin Shaw
VCP

Speed, Speed, Speed

Who would've ever thought you'd see network performance like this on an Windows box:

This is the networking graph I got off of a physical box running the inital push for Quest vConverter's Continuous Protection option. The blip in the middle was it switching from the C: volume to the D: volume. It just about knocked my socks off - I've never seen anything eat up the entire NIC on Windows.

I found out some really cool stuff with vConverter. With the Continuous Protection option, you can keep a full-image copy of your physical server on an NTFS Share, ESX(i) Server, or Hyper-V server. After the first push, it just copies block (or file level) changes (whichever option you want) to the target store. This way, if you ever lose your physical box, you can either immediately pop it online in your ESX/Hyper-V box, or do a V2P back to your original hardware once it's back up and running. Very cool stuff.

Even cooler - let's say you're looking at Offsite DR. You already have Virtual Servers (VMware or Microsoft) in a hardened data center, and are currently replicating your VMs at your main office offsite to there. Good for you! Now how do you push your physical boxes out there?

With vConverter, you can do your initial copy locally (say to an NTFS Share on a NAS device), then you can take it to your Datacenter, and upload it to your SAN that your ESX or Hyper-V boxes use. Then go into vConverter, do a "change target" and select your ESX or Hyper-V box, tell it where the data is, and your good to go. Now all you have to worry about getting across the WAN is your block changes. Now you really do have an Offsite DR solution.

------
Dustin Shaw

VCP

ESX boot issue

Ran into this on one of our ESX 4.1 boxes. After it was shutdown, it failed to come up. When I looked at the console, it showed: vsd-mount failed.

There's a good VMware knowledge base article on this.

The only caveat that I found for the knowledge base article base was that you need to follow the instructions verbatim.

Also, you need to pay attention to step 10. When you edit the kernel line, you need to understand how it's presenting the line. It's scrolled sideways. If you really want to see the beginning of the line, move your cursor to the left. Where you want to insert the text is at the end of the line, though.

For your convenience, here's the resolution from the KB article:

Resolution

If the ESX host has detected the VMFS volume containing the esxconsole.vmdk file as a snapshot LUN, the ESX host drops into Troubleshooting (busy box) mode during boot. 

To allow your ESX host to boot successfully:

1. Provide the necessary credentials to access the busy box.
2. Run this command to enable resignaturing on the VMware ESX machine:
   
   esxcfg-advcfg -s 1 /LVM/EnableResignature
   
   You must get an output similar to:
   Value of EnableResignature is 1.
   
   Note: If the root is mounted as read only, run the command  mount -o remount / to remount the volumes so that they are in a writable state.
    
3. Run this command to unload the VMFS drivers:
   
   vmkload_mod -u vmfs3 
4. Run this command to load the VMFS drivers:
   
   vmkload_mod vmfs3
    
5. Run this command to detect new VMFS volumes and resignature the volume:
   
   vmkfstools -V
6. Run this command to identify the full path of the esxconsole.vmdk file:
   
   find /vmfs/volumes/ -name esxconsole.vmdk
   
   The output appears similar to:
   
   /vmfs/volumes/4a14d968-88bf7161-700f-00145ef48f76/esxconsole-4a14d906-2f96-7956-7284-00145ef48f74/esxconsole.vmdk
    Note: Make a note of this full path.
   
7. Restart the VMware ESX machine. You see a menu provided by the grub boot loader. 
8. Press e to edit the grub entries manually.
9. Scroll down to the line that starts with kernel /vmlinuz (it is indented under the VMware ESX 4.0 heading).
10. Go to the end of the line and include the following entry after a space:
    
    /boot/cosvmdk=<path>/esxconsole.vmdk
    
    Where <path> is the full path identified in step 6.
     
11. Press Enter to accept the changes.
12. Press b to boot using the modified settings. The ESX host successfully boots.
    
    Note: The changes made to the boot options are not saved. They only apply to the current boot process. The changes need to be made to the boot configuration files as described in the following steps.
     
13. Log into the console as root.
14. Edit the /etc/vmware/esx.conf file with a text editor and modify the following lines:
    
    /adv/Misc/CosCorefile = "/vmfs/volumes/<path>/core-dumps/cos-core"
    /boot/cosvmdk = "/vmfs/volumes/<path>/esxconsole.vmdk" 
    
    Where <path> is the full path identified in step 6.
     
15. Run this command to update the boot configuration files:
    
    esxcfg-boot -b

------
Dustin Shaw
VCP