EventID 9385 on Exchange 2010 After Demoting DC

I recently demoted an old Domain Controller in an effort to move forward in my domain - it was a 32-bit 2008 Server, and all the rest of the DCs are 2008R2 or 2012R2. I don't have any needs today to move up AD functionality today (we are already on 2008 Forest and Domain Functionality), but it never hurts to be ready.

After demoting an old Domain Controller, I recently started receiving Error 9385 on one of my Exchange 2010 Mailbox servers:

Log Name:      Application
Source:        MSExchangeSA
Date:          6/2/2015 8:43:34 AM
Event ID:      9385
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MailboxSVR.internal.domain
Description:
Microsoft Exchange System Attendant failed to read the membership of the universal security group '/dc=domain/dc=internal/ou=Microsoft Exchange Security Groups/cn=Exchange Servers'; the error code was '8007203a'. The problem might be that the Microsoft Exchange System Attendant does not have permission to read the membership of the group. 

If this computer is not a member of the group '/dc=domain/dc=internal/ou=Microsoft Exchange Security Groups/cn=Exchange Servers', you should manually stop all Microsoft Exchange services, run the task 'add-ExchangeServerGroupMember,' and then restart all Microsoft Exchange services. 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeSA" />
    <EventID Qualifiers="49152">9385</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-06-02T13:43:34.000000000Z" />
    <EventRecordID>2199039</EventRecordID>
    <Channel>Application</Channel>
    <Computer>MailboxSVR.internal.domain</Computer>
    <Security />
  </System>
  <EventData>
    <Data>/dc=com/dc=wmfingrp/dc=internal/ou=Microsoft Exchange Security Groups/cn=Exchange Servers</Data>
    <Data>8007203a</Data>
  </EventData>
</Event>

After doing some research, most articles said that you need to make sure that it's a member of the group, etc, but all of that was correct. There weren't any references to the old DC in anything I checked (DNS was pointed elsewhere, Domain controllers and Global catalog servers that this Exchange server used were pointed elsewhere, etc). But, I knew it had to do with my demoted DC. For some reason this particular Exchange server was really hoping that the DC would answer his requests. I didn't notice any other performance or user issues during this time, so it looks like the Exchange server was able to get his answer elsewhere after checking here.

Once I was able to take a maintenance window, I rebooted the affected Exchange server, and all was well. It just needed to clear it's head after loosing it's good friend.

------
Dustin Shaw
VCP

The symbolic link cannot be followed

When you setup symbolic links on a server that point to another server, you will by default run into the inability for a client computer to follow the links with the following error:


The symbolic link cannot be followed because its type is disabled.

This is because the ability to traverse from one remote system to another across the symbolic link is disabled by default. You can see what is disabled and what is enabled on a computer by running the fsutil command:

>fsutil behavior query eymlinkevaluation
Local to local symbolic links are enabled.
Local to remote symbolic links are enabled.
Remote to local symbolic links are disabled.
Remote to remote symbolic links are disabled.

You have two methods to enable this - enable it locally on each machine, or enable it via Group Policy.

Local

The downsides to enabling it locally are obvious, but sometimes you just need it on one stubborn computer *right now* and can't wait for GP. To enable Remote to Remote symbolic links, run the following command:
fsutil behavior set symlinkevaluation R2R:1

Similarly, you can change the settings for Local to Local (L2L), Local to Remote (L2R), and Remote to Local (R2L) by using 1 for enabled and 0 for disabled.

Group Policy

To enable (or disable) Remote to Remote symbolic links in Group Policy, create a new GPO Policy (or edit a current one), and edit it. Navigate to:
Computer Configuration -> Administrative Templates -> System -> Filesystem
You can then set the settings how you want in Selectively allow the evaluation of a symbolic link

Once you've created your new GPO, test it and validate that it is successfully applied using gpresult /R and rsop.

The wizard was interrupted before VMware Tools could be completely installed.

Upon attempting to install VMware Tools on a Windows Server 2008R2 server, I received an error stating "The wizard was interrupted before VMware Tools could be completely installed."

After looking around on the internet, the closest thing I could find was this post by David Homer, detailing a similar issue with VMware Tools on VMware Workstation.

I tried his fixes (remove VMware Tools registry key, remove the vmtools service), but none of them allowed me to get past the problem.

What eventually worked was doing a search in the registry for all references to "vmware" and removing all keys having to do with VMware Tools - make sure you don't remove any of the keys referring to your SCSI devices/other hardware; just the VMware Tools entries.

I believe it was most likely the Installer registry entries that were mucking it up.

After a quick reboot to refresh the registry, VMware Tools installed with no issues.

------
Dustin Shaw
VCP

Installing PowerCLI on Server 2008

In attempting to install VMware vSphere PowerCLI on Windows Server 2008 x64, the following error comes up. This also happens on Server 2008R2.

Error 1406. Could not write value InstallPath to key \Software\VMware, Inc.\VMware vSphere PSDK Runtime. Verify tha tyou have sufficient access to that key, or contact your support personnel.

The prescribed fix for this is to remove the following registry key (and subkeys):
HKLM\Software\Wow6432Node\VMware, Inc.

*** Please make sure you know what you are doing in the registry before you do anything in there!!!

After that, installation proceeds successfully.

The only thing under the VMware, Inc. registry key is a "volatile" key with a UUIDHost DWord under it. Since I don't like to mess with other programs installed on the server (the one I was using has Quest vRanger installed), I exported the key, removed it, installed PowerCLI, and the imported my volatile key back in.

------
Dustin Shaw
VCP

Change DAG Witness Server - Exchange 2010 Hosted

I recently had to change a DAG on Exchange 2010 Hosted. This only uses the Exchange Management Shell, and I couldn't find instructions specific to this, so I thought I'd share my experience.

Requirements

Witness Server has the following Requirements:

-          Can’t be part of the DAG

-          Same AD forest as DAG

-          Windows Server 2008 R2, Windows Server 2008, Windows Server 2003 R2, or Windows Server 2003

-          Each DAG requires a different witness directory, but they can all be run on the same server

Permissions

Set permissions on target server –

Local Administrors group – add Exchange Trusted Subsystem

Reboot server to apply permissions

Commands

Go to Exchange Management Shell, and type in:

Get-DatabaseAvailabilityGroup | fl

There are two lines to pay attention to: WitnessServer and WitnessDirectory

To change the Witness server, type the following:

Get-DatabaseAvailabilityGroup | Set-DatabaseAvailabilityGroup –WitnessServer witness.serverdomain.local –WitnessDirectory C:\DAG1

------
Dustin Shaw
VCP

Exchange 2010 Prereq Script

I was looking for an easy way to install all the Prerequisites on a new Exchange 2010 install this last week, and ran across a good script that does it all for you.

The script was written by Dejan Foro over at ExchangeMaster.net. You can find it here.

He has several variations of the script: one that lets you choose what server roles you are installing, or one that installs all prerequisites needed if you are loading down a box with all Exchange Roles.

It's also got variations for whether you are running Server 2008 SP2 or Server 2008 R2.

It's available in either ISO or ZIP format.

------
Dustin Shaw
VCP

All About the Design

It seems to me that you can always tell if an IT guy is worth his water by the way that he architects/designs solutions.

We picked up a small shop recently that their previous guy built them a heavy-built Dell T710 with Microsoft Hyper-V. They've got 5 VMs running on it, so nothing wrong with the basic idea behind it, but the execution is where I have an issue.

The machine was originally built (from factory) with the onboard SAS 6/iR RAID card with a set of four 450GB drives in a RAID 5 config. No hot spares. Issue number one.

He realized that he must have under built, so he then went off the deep end. He went out to NewEgg and bought four 2TB SATA drives, bought the caddies off EBay, and bought a PERC 6/i RAID controller directly from Dell (must've not found it cheaper somewhere else). This means the factory warranty from Dell (3 year NBD onsite) won't extend to anything except the RAID card. What's the most likely thing to die on a server again?

He then went in and reconfigured the machine to run off the PERC with a second RAID 5 set with the 2TB drives (again no hot spare), and directly attached one of the VMs to it for their main data storage. This is all fine and dandy, except for the fact that there is 800GB free on the first RAID set, and they are only (at this point) using 40GB on the second RAID set (leaving roughly ALL of it free).

And here comes the second part of the issue - much bigger than the previous config issues.

The PERC 6/i controller has issues running on drives 1.5TB and up. The latest firmware revision runs better than the previous one, but the controller is regularly (it's happened 3 times in a month, hence why the old guy went bye-bye) dropping a drive offline. Part of the issue is that when it drops the drive, it drops the RAID set offline at the same time.

When I got called in, the RAID set had dropped two drives offline (meaning bad, bad things could've happened to their data). I reseated the "bad" drive, imported the foreign config, and let the RAID set rebuild itself. I then proceeded to move their data off of that RAID set and remove the config from the environment.

They still have 800GB to grow into before we even need to consider doing anything. My first plan of order is to tell them that they need to get another 450GB drive and put it in as a hot spare. As I mentioned, this is a small shop, and the server sits on the floor in a closet. They only open the door when there is a problem, so they could easily loose a drive and not know it until someone asks "What's that red light for? It's been doing that for a month." Hence the driving need for a hot spare.

------
Dustin Shaw
VCP

vConverter 5.1 is out!

Yesterday Quest released vConverter 5.1 onto their website for download. You can check out the product here.

For those of you familiar with their product, you'll know that vConverter 4.2 was the previous version that was out. They recently released vConverter 5.0 as an "early release preview" (AKA - Beta), but it had numerous issues. One tech that I spoke with said that the marketing team put version 5 online for download and issued a press release before they consulted with tech or engineering - sounds like many marketing guys that I know.

If you were one of the lucky ones that installed the "early release preview," then you'll have to uninstall it before you install vConverter 5.1.

The new features to look for in version 5.1 are:

P2V for Linux Servers: this was a big absent in the previous releases. I know many penguin fans who are excited about this one. It's a Cold Convert, so you have to be down when doing the conversion and have physical access to the box so that you can boot off a CD or PXE server (unless of course you have ILO or DRAC or other console access). This is unfortunate since VMware's vCenter Converter 4.x supports several Linux builds doing Hot Convert, but there are a number of other features that vConvert does buy you.

ESXi Support: This is a good thing since VMware is pushing everyone hard away from ESX onto ESXi. They are doing this by using a Virtual Appliance.

Hidden Partitions Support: This has recently become an issue with Windows 7 and Server 2008r2, keeping most converter programs from being able to work. I'm glad to see this added, as if you've been reading my blog you know that I recently ran into an issue caused from someone doing a conversion of a 2008r2 box using Acronis.

vConverter 5.1 download comes with a 30 day trial license that allows you to use Continuous Protection on up to 5 source machines, and limits you to 5 conversions (not including the Continuous Protection conversions).

------
Dustin Shaw
VCP

Disabling IPv6 on Server 2008/2008r2

As an outsource IT professional, there is one thing that I've run into over and over again in SMBs - IPv6 doesn't play well unless FULLY integrated. That means if you try to simply "uncheck" the box for IPv6 to disable it, you've just hosed yourself. Similarly, if you just left it checked because you don't know what your doing, you've just hosed yourself.

There are a number of issues that I've seen because of the idiocracy above:

- Slow XP performance on every task when joined to a domain with a 2008 server with IPv6

- Broken SBS 2008 server when you uncheck the box (things that break are "Applying Computer Settings" for a long time, Exchange won't start, Network Icon shows offline in error)

- Exchange 2010 won't install when IPv6 is unchecked (but not disabled)

- Exchange 2007 and Exchange 2010 won't start when unchecked (but not disabled)

- Other "random" issues that can't be pinned down

So basically you have two solutions:

1. Implement IPv6 on EVERYTHING - aka get a new switch, router, firewall, etc. that supports IPv6, configure it and everything else with IPv6 (that includes any legacy XP or 2003 boxes in the network), and get it all working. Then you just translate to the IPv4 world for the rest of your internet until everyone jumps on that bandwagon. Oh, and by the way, you'll probably need to replace all your gear before it gets to that point.
   
2. Go the cheap route and completely disable IPv6. I say cheap because it's not only cheaper in hardware (you don't have to replace all your infrastructure just because of IPv6 compatibility - granted, it can give you a good bullet point to show the boss if you want to anyway...), but it's cheaper in labor because you don't have to teach all your geeks (umm, I mean IT professionals...) how to read and understand the new subnets.

Since I'm betting that most people like to go the way of "cheap," I'll detail that one for you :-) If you'd rather go balls-to-the-walls, go for it. Just make sure you understand the animal first.

--

On your new, shiney Server 2008 / 2008r2, first you want to Uncheck the box that says "IPv6" (DUHH!)

Once that's done, go to Regedit***. You'll want to navigate here:

HKEY_Local_Machine\System\CurrentControlSet\Services\Tcpip6\Parameters

Create a DWord (32-bit) value named "DisabledComponents" and Click OK. Note: There is no space in the DWord name - and no quotes either ;-)

Double Click you're new "DisabledComponents" DWord and put in the following value to completely disable ALL of IPv6 (except IPv6 loopback interface):
ffffffff --- Hex
or
4294967295 --- Dec

Click OK.

There are also other options to do with this DWord, as detailed perfectly by MS here:
http://support.microsoft.com/kb/929852

Once done, restart your server to make sure that everything takes full effect. I've seen it take effect without restart, but restart is always best practice when playing with the registry.

If you have additional servers running 2008 or 2008r2: rinse, lather, repeat.

Now you should have smooth sailing with just IPv4 on the network. And if it's not smooth, at least you know that IPv6 wasn't your problem :-)


------
Dustin Shaw
VCP

~~~~***~~~~
Danger! Danger!

Don't go playing with the registry unless you actually know what you're doing.
If you don't know what you're doing in the registry, bad bad things ensue.
If you insist on playing in the registry anyway, you should also go update your resume and notify your boss that they will soon need a replacement that can rebuild a server.

Danger! Danger!
~~~~***~~~~


Here's some books that should help out:
IPv6 Essentials
Understanding IPv6, Second Edition