Deploying vCOPS 5.x in vSphere Essentials Plus

VMware vCenter Operations Manager 5.x has a requirement for DRS (Distributed Resource Schedule) to be able to be deployed in a cluster - this is because it is deployed as a vApp, and vApps require DRS.

If you are attempting to install vCOPS on vSphere Essentials, Essentials Plus, or Standard, you will discover that you don't have license for DRS. VMware released this article as the resolution to the problem. Here are the steps that they detailed:

---

To deploy vCenter Operations Manager in a vCenter Server environment with an Essentials Plus or Standard cluster of three ESX hosts:

1. Remove one of the ESX hosts from the cluster so that it resides directly under the parent datacenter.
2. On that ESX host, deploy the vCenter Operations Manager vApp, specifying static IP addresses.
3. Power on the vApp before moving the host back into the cluser to ensure IP settings are picked up.
4. License the solution in vCenter.
5. Move the ESX host with vCenter Operations back into the cluster.

Note: Static IP addresses are required.

Warning: The steps above dissolve the vApp container and an error is displayed. Disregard the error message and continue moving the host into the cluster.

Moving the ESX host with the vCenter Operations vApp back into the cluster results in the addition of the two virtual machines (the UI virtual machine and the Analytics virtual machine) to the cluster, without the vApp container. vCenter Operations Manager 5.x continues to function normally when this happens.
---

Also, in case you need it, the default logins for vCOPS are:
Default Administration:
admin
admin

Default Root Login:
root
vmware

------
Dustin Shaw
VCP

The wizard was interrupted before VMware Tools could be completely installed.

Upon attempting to install VMware Tools on a Windows Server 2008R2 server, I received an error stating "The wizard was interrupted before VMware Tools could be completely installed."

After looking around on the internet, the closest thing I could find was this post by David Homer, detailing a similar issue with VMware Tools on VMware Workstation.

I tried his fixes (remove VMware Tools registry key, remove the vmtools service), but none of them allowed me to get past the problem.

What eventually worked was doing a search in the registry for all references to "vmware" and removing all keys having to do with VMware Tools - make sure you don't remove any of the keys referring to your SCSI devices/other hardware; just the VMware Tools entries.

I believe it was most likely the Installer registry entries that were mucking it up.

After a quick reboot to refresh the registry, VMware Tools installed with no issues.

------
Dustin Shaw
VCP

Install VMware Tools on RHEL 6 or CentOS 6

Below is the process to install VMware Tools on RHEL 6 or CentOS 6. This guide is more here for me than anyone else, but I hope that you can benefit from it.


Install the Pre-Requisites:
yum install make gcc kernel-devel kernel-headers glibc-headers perl


Start the VMware Tools installation process on your VM:





Mount the VMware Tools installation media:
mkdir /mnt/cd
mount /dev/cdrom /mnt/cd
Expected warning:
mount: block device /dev/sr0 is write-protected, mounting read-only


Extract the installer:
cp /mnt/cd/VMwareTools-9.0.10-1481436.tar.gz /tmp/
umount /mnt/cd
cd /tmp
tar xvf VMwareTools-9.0.10-1481436.tar.gz
cd vmware-tools-distrib/


Install tools (accepting all defaults):
sudo ./vmware-install.pl -d

Reboot the VM to verify that the service starts up automatically as expected.
shutdown -r now


------
Dustin Shaw
VCP

Extend a volume in RedHat 6 or CentOS 6

To extend a partition on a VM on CentOS 6 or RHEL 6, do the following:

Extend the Partition in vSphere to the size that you need:

in the VM, refresh the Partition Tables by the following command:

echo 1 > /sys/block/sda/device/rescan
Run fdisk to create a new primary partition:

fdisk /dev/sda

Press p to print the number of partitions
Press n to create a new primary patition
Press p for primary
The partition number should automatically be selected, and press enter twice to accept the beginning and ending blocks on the free space.
Press w to write the changes.

Reboot the system: shutdown -r now

Verify that the new partition exists:
fdisk -l

Convert the new partition to a physical volume:
pvcreate /dev/sda3
Extend the Physical Volume (run vgdisplay to obtain the name of the Volume Group; default is VolGroup00):
vgextend VolGroup00 /dev/sda3

Extend the Logical Volume (run vgdisplay to obtain the free space and replace # below, and lvdisplay to obtain the name of the Logical Volume; default is /dev/VolGroup00/LogVol00)
lvextend -L+#G /dev/VolGroup00/LogVol00

Expand the ext3 online using the following command (substituting your proper Volume Group and Logical Volume names):
resize2fs /dev/VolGroup00/LogVol00

Confirm that your new space is available:
df -h



This was written using VMware's KB article on the subject as a reference.

------
Dustin Shaw
VCP

vRanger 5 and ESXi Performance Settings

Everyone should know that the previous recommendations from Quest were to increase the CPU Reservations to 1500 on ESX servers for the best performance from vRanger.

I just recently ran across an article (here) from Sept 2011 detailing that this is NOT the case for ESXi. There is no performance gain on ESXi by increasing the CPU reservations from the default 293.

------
Dustin Shaw
VCP

vRanger error "There is an error in XML document"

I ran into an issue with one of our repositories - every job in the particular repository was failing with the error:

An internal error occurred during execution, please contact Quest support if the error persists. Error Message: There is an error in XML document (823, 223).

This error is commonly caused by a corruption or issue with the manifest file as detailed here.

To fix it, go into the manifest file for the repository (GlobalManifest.metadata), make sure that all trailing brackets are closed, and add </vRangerGlobalManifestFile> at the end of the document.

Save and test.

------

Dustin Shaw
VCP

vCenter Server 5 Service Fails

We had an issue with the vCenter Server 5 Service failing recently. Basically what happened was the VMware VirtualCenter Server service failed (out of the blue) with the following Informational Event ID 1000 logged in the Application log:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Starting VMware VirtualCenter 5.0.0 build-623373

the message resource is present but the message is not found in the string/message table

Followed by another Info Event 1000:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Log directory: C:\ProgramData\VMware\VMware VirtualCenter\Logs.

the message resource is present but the message is not found in the string/message table

And followed by an Error Event 1000 (upon it attempting to auto-restart the service):

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Failed to intialize VMware VirtualCenter. Shutting down...

the message resource is present but the message is not found in the string/message table

First try was to restart all the vCenter services - only the VMware VirtualCenter Server service was offline; all other services (like VMware VirtualCenter Management Webservices) were still online. This obviously failed with the same error as before.

All of the SQL services were verified online (these are kept on a separate server due to the size), and accessible. I ran across a similar Discussion thread in VMware Communities, which pointed to issues inside of the SQL Database. Based off of this, I decided the first order was to look at my SQL server to see what was going on there. I was able to log in, and verified that everything was up. Then I looked at my logs, and saw Event ID 17053:

C:\VCDB.ldf: Operating system error 112(There is not enough space on the disk.) encountered.

 And instantly I knew my problem. Some yahoo (namely the yahoo writing this article) must not've been paying attention when installing the VCDB database, and stuck it in the root of the C drive. Naturally, that yahoo had to fix his own problem...

So I went into Microsoft SQL Server Management Studio, and ran the following command:

alter database VCDB modify file ( name = vcdb , filename = 'E:\SQL\MDF\VCDB.mdf' )
alter database VCDB modify file ( name = vcdb_log , filename = 'E:\SQL\LDF\VCDB.ldf')
go

Then I Offlined the database (Right click, Tasks, Take Offline), and moved the files to their new homes. Then I Onlined the database, and verified that the path was correct for it. You can see the full process of how to move a SQL database here.


Once this was done, I went back to my vCenter Server, and was able to bring all my services online without incident, and was able to again go in and manage my vCenter 5 Server via vSphere Client.

------
Dustin Shaw
VCP

Host update fails after updating to 4.1u2

After updating vCenter Update Manager (right after updating my vCenter Server) from 4.1u1 to 4.1u2, I received the following error when trying to update on of my hosts:

Remediation did not succeed for esxihost: SingleHostRemediate: esxupdate error, version: 1.30, operation: 7: ('http://esxihost:9084/vci/hostupdates/hostupdate/vmw/vibs/cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1OEM.vib','/var/tmp/cache/-1699692350','[Errno 14] HTTP Error 404: Not Found')

After doing some research, I discovered what happened is that Update Manager 4.1u2 is case sensitive, whereas 4.1u1 was not. Any patches that were downloaded previous to the 4.1u2 update that contain upper case letters will fail with this error. VMware has a KB article about it here.

The patches affected are the ones below. I've put the correct capitalization on them for you. If you go to your patch repository, you can rename the files that you have to the below, and you should then be able to update your hosts. The repositories are located here:

• Windows2008: C:\ProgramData\VMware\VMware Update Manager\Data\Hostupdate\vmw\vib\
• Windows2003: C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Data\Hostupdate\vmw\vib

bind-libs-9.3.6-4.P1.el5_5.3.i386.vib
bind-libs-9.3.6-4.P1.el5_5.3.x86_64.vib
bind-utils-9.3.6-4.P1.el5_5.3.x86_64.vib
bind-libs-9.3.6-4.P1.el5_5.3.i386.vib
cross_oem-vmware-esx-drivers-net-vxge_400.2.0.28.21239-1OEM.vib
cross_oem-vmware-esx-drivers-scsi-3w-9xxx_400.2.26.08.036vm40-1OEM.vib
vmware-esx_swMgmt_provider-4x.1.0.1-1.4.348481.vib


------
Dustin Shaw
VCP

ESXi Host Unable to Update

I had a VMware ESXi 4.1 host that I was unable to update/patch using Update Manager the today.

It was coming up with the following errors when I tried:

VMware vCenter Update Manager had an unknown error. Check the Tasks and Events tab and log files for details.

And when I did that, I found this:

Could not install patches on esxihostname
Remediation did not succeed for esxihostname: SingleHostRemediate: Install error on host: esxihostname, error details: vim.fault.NoHost.

So I did some quick googling, and ran across this website that had the answer I was looking for.

The particular host that I was trying to update did indeed have an Unknown (inaccessible) VM on it (left over from yanking the storage and being sloppy...). I removed the VM from inventory, then was able to successfully patch the host.

------
Dustin Shaw
VCP

Datastore Alarm Not Accurate

I had the unfortunate mishap of a VM dropping offline because it outgrew the amount of free space on the datastore it existed on. On the plus side, it happened in the middle of the night and I wasn't on call :-) The on call guy simply had to reboot the box, and it came back online.
When I got in and looked, the datastore that it existed on had 0.00 B of free space. Not very comforting, considering the fact that there were no alarms on that datastore, and there were several other Production VMs on that LUN.
To fix the alarm state, it was easy. Go into the alarm definition and disable and re-enable it. This will force it to refresh it's states and you will get to see all the pretty yellow and red icons that you've been missing.

This also will work if it is reporting a Usage Above Threshold before it should too.

------
Dustin Shaw
VCP

vRanger "host could not be identified"

We had our vCenter sever hiccup on us yesterday, so (naturally) our vRanger backups failed last night.

The error we received was:

An internal error occurred during execution, please contact Quest support if the error persists.  Error Message: VMSERVER's host could not be identified

Essentially what causes this issue is a communication sync error between vRanger and vCenter. I ran across this error a year and a half ago running on vRanger 4.x (you can read my post here). This time it was on Quest vRanger version 5.3.0, so I did a quick search to see if there were any newer solutions for verison 5. It appears that Quest now has a KB article about it.

Basically, their solution is to restart the services - stop the API and FLR Services, restart the vRanger service, then start up the API and FLR services.

I always prefer to go above, so I also refreshed the jobs (like my previous experience taught me) by opening them up and click next and finish all the way through (without changing anything).

Next time I'll remind my guys to restart vRanger services whenever vCenter bounces.

------
Dustin Shaw
VCP

V2V from Hyper-V to VMware ESXi

So you finally came to the realization that Microsoft Hyper-V really isn't as mature as you were lead to believe. Good! You can attend the Gullible Anonymous meetings later. First, let's get you over to something that can keep your servers up and running and performing like the $10k you spent on your hardware should provide you.

Assuming that you have a free server available, doing a V2V from Hyper-V to VMware ESXi is extremely easy. Just think of this as doing a P2V, and ignore the wasted resources in between...

Take one of your spare servers, and load up VMware ESXi on it (you can do 4.1 or 5, or whatever strikes your fancy). Get it all ready for the VM to run on it, and get it on the same network and gig switch as your Hyper-V box.

Login to the VM that you want to convert, and download and install VMware's Standalone Converter on it (current version is 5.0). Current supported platforms are:

Windows XP Professional (32-bit and 64-bit)

Windows Server 2003 SP2, R2 (32-bit and 64-bit)

Windows Vista (32-bit and 64-bit)

Windows Server 2008 (32-bit and 64-bit)

Windows Server 2008 R2 (64-bit)

Windows 7 (32-bit and 64-bit)

Next, you will want to run the Converter against the local machine, and convert it to ESXi server that you just created. There are several settings to change to verify that it's done properly:

• Prep your computer for a P2V by stopping ALL unneccessary services. You only need core services to make sure that it's successful.
• Select Powered-on machine; local machine

• Put in the IP/Hostname of your new ESXi server and root/password

• Give it a destination Virtual Machine name
• Select a Resource Pool (if used), datastore, and Virtual Machine hardware version (unless you have a driving reason, go with Version 7)
• Set your disk drives the way that you want them to be on the new environment - I prefer Thin on all my disks, but it's up to you and your application. To change this, Edit the "Data to copy," Advanced, and Destination Layout tab. Change the type to Thin

• Change the Disk controller to SCSI. To change this, Edit the "Devices" and go to the Other tab. I typically select SCSI LSI Logic SAS for newer OSes and SCSI LSI Logic for older OSes.

• Set all services that say Hyper-V to disabled. To do this, Edit the "Services" and go to the Destination services tab. The ones I came across on my most recent migration were (your mileage may vary based on the version/options you have):
• Hyper-V Heartbeat Service
• Hyper-V Data Exchange Service
• Hyper-V Guest Shutdown Service
• Hyper-V Time Synchronization Service
• Hyper-V Volume Shadow Copy Requestor
• Edit Advanced options and go to the Post-conversion tab and check "Install VMware Tools on the destination virtual machine"

Off it goes converting. Once it's done, power down your Hyper-V VM, power up the VMware VM, and wait for the VMware Tools to install. Once that's done, uninstall the VMware Converter, and your off and running.

Now you can wipe your Hyper-V server, load up ESXi on it, and build yourself a nice little redundant cluster.

------
Dustin Shaw
VCP

vRanger Error

Ran across this error on a client running vRanger:

Message: Failure creating VM datastore lock Host: ESXHostNameha-host VM: MVName

After investigating, the root of the issue was that the ESX host had run out of memory - there were too many VMs running on this host. This was keeping the VMs from quiescing long enough to get a good snapshot. When they would try, the memory would spike and balloon all over the board. After thinning out some memory, the problem disappeared.

This particular client is growing faster than they were paying attention to, hence the increase in memory utilization across the board.The client is ordering more memory for all the hosts in their cluster and looking at ordering another host or two so that they can get back to N+1 (where they actually started at).

------
Dustin Shaw
VCP
------
Here's some books that should help out:
VMware vSphere 4 Administration Instant Reference
VMware Fusion 3
VMware Cookbook: A Real-World Guide to Effective VMware Use

vRanger Error

Here's a random vRanger Error:

Message: An internal error occurred during execution, please contact Vizioncore support if the error persists.  Error Message: VM's host could not be identified

Got this on one of my client's install's today. vRanger 4.x.

The solution? Go refresh the inventory. Go edit your backup job and keep all the same settings.

Easy fix.

-Update-

I ran into the error again on vRanger 5.3.x, and posted an update here.

------
Dustin Shaw
VCP

View 4.5 is Alive!

So I finally had to make time to put in View 4.5 on my 2 test Datacenters (one is a Demo environment, the other for play).

Went in like a breeze on both. Run the installer for View Manager, update the View Composer, and boom. I'm up to 4.5.

The only real issue I ran into was with my existing XP Linked Clone pool on my test Datacenter (the Demo datacenter was fine...). I went in and removed and re-added the View Agent and updated my Snap and pointed the Pool to it. No joy. I got a blank screen.

Did some research, and saw some notes about ESX 4.0u2 with View 4.0 causing this issue, so I thought I'd start there.
Here's two links I found:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1016753
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022830

Besides, I felt like updating the datacenters to ESX 4.1 anyway, since I had held off because View 4.0 doesn't like ESX 4.1.

So I went through and upgraded vCenter, vCenter Update Manger, then used Update Manager to update my hosts. If you've not used Update Manager for it, you should - it's cool to watch. Apply your baseline with 4.1, tell it to remediate, and sit back and enjoy. That is, assuming of course, that all of your vms can migrate on their own (aka, you don't have them attached to CD drives, etc).

So once I had them updated, I once again updated my vmTools on the XP image, removed and readded the View Agent for good measure, then updated my Snap and Linked Clones. Still no joy.

So I though, ok, maybe something is wrong with my pool.

I created a new desktop pool (based off the same Snap) and it worked!

Since these were just Demo/Test datacenters, I just blew away my old pools and created new ones. If this was a live environment, then I'd have to worry about keeping the persistent disks and reattaching them.

After XP was successful, I built myself a Win 7 image, created a pool off of it, and awesome! It works great.

I love the dual monitors! I have two different resolution monitors at home, and two different aspect ratio monitors at work, so I can now take full advantage of them with View 4.5.

Now I'm going to proceed with testing out the ThinApp stuff integrated into View 4.5, and some performance testing, but so far it all looks awesome!

------
Dustin Shaw
VCP
------
Here's some books that should help out:

VM Shuts Off When Taking Snapshot

Here's a really cool one :-)

Have a client that just did a P2V - they took all of their 5 servers and did a P2V of them into a new vSphere implementation. Naturally, they also purchased vRanger so that they could get the best bang for their buck.

They had been continuing to backup using their good old Symantec method after the P2V project, and just now got around to the point of switching over to vRanger. They're a small shop, so they can take a while to move from Point A to Point B.

Well, vRanger ran for the first time last night, and kicked out the following errors on two VMs:

Message: An internal error occurred during execution, please contact Vizioncore support if the error persists.  Error Message: API Call failed with message: Error encountered while restarting virtual machine after taking snapshot.  The virtual machine will be powered off.

Took a quick glance, and remembered that these two boxes were the ones that were already running 2008r2 before the P2V. Because of this, we couldn't use the standard VMware Standalone Converter that we typically use for small shops - it doesn't support 2008r2. Instead we used Acronis to do the P2V using their True Image Echo software. 

Not my first choice, and not my decision either. I didn't even get to play on this project... One of my other techs kept saying "Can I do one? Can I do one? You get to have all the fun! Can I do it?" Obviously I got tired, and said sure; plus I wanted a weekend off to spend with the family. I found out after words what happened, but they were up and running and happy, so I didn't bother following up.

Think it's a coincidence? Not so, according to Quest's tech support - the second call my guys made to get the VMs back up (the first was to me: "HELP!" "I'm at a VMware View Presentation in front of 10 prospective companies. Call Quest or VMware; they have support on both." "Right!")

Apparently, when the Acronis P2V was done, it split up the vmdk files into 2GB chunks, rather than one big one. Not sure if this was an option they picked or if it was default - if the guy who did it was still working for us I would ask... He's gone for different performance reasons.

When the vmdk file is like this on vSphere, snapshotting doesn't work right, and kicks the box off line.

So the fix is to run vmkfstools to convert the multiple vmdk files down to one, then you're good to go. Alternatively, you can just Clone the VM to let it consolidate them for you, then remove the old one.

Ain't life fun!

Next time I'm going to verify the work done a little closer...

------
Dustin Shaw
VCP
------
Here's some books that should help out:
Maximum vSphere: Tips, How-Tos, and Best Practices for Working with VMware vSphere 4
VCP VMware Certified Professional on vSphere 4 Study Guide: Exam VCP-410
VMware vSphere 4 Implementation

ESX File Systems Error on boot

Ran into this recently, though it might help someone else:

We have a Demo vSphere environment that gets banged around a lot, including getting shut down dirty often ("Look! I can yank the plug and it keeps running!"). One of our ESX hosts in the cluster came up with this fun error:

fsck.ext3: Unable to resolve 'UUID=####'
An error ocurred during the file system check. Dropping you to a shell; the system will reboot when you leave the shell.

The rest of the details in the message will tell you exactly which volume you've got issue with.

Usually, this is an easy enough fix; just force fsck.ext3 to clean your problem volume. I actually like to use it as an opportunity to check all my volumes. Here's what I do:

df     ---  get a list of all your volumes, they will as /dev/sdxx where xx is a hex number
fsck.ext3 -f /dev/sdxx          ---- run this command on each volume on the df list

You'll have the problem volume prompt you about "Do you want to fix this?" "Do you want to fix that?" Just answer yes (after reviewing what you're about to do, of course), and you'll be good to go.

Reboot your host, and you should be up and running.

------
Dustin Shaw
VCP
------
Here's some books that should help out:
Computer File Systems: Computer File, File Archiver, Fsck, 8.3 Filename, Root Directory, Working Directory, Comparison of File Systems
VMware vSphere 4 Administration Instant Reference
Mastering VMware vSphere 4 (Computer/Tech)