Friday, October 10, 2014

Deploying vCOPS 5.x in vSphere Essentials Plus

VMware vCenter Operations Manager 5.x has a requirement for DRS (Distributed Resource Schedule) to be able to be deployed in a cluster - this is because it is deployed as a vApp, and vApps require DRS.

If you are attempting to install vCOPS on vSphere Essentials, Essentials Plus, or Standard, you will discover that you don't have license for DRS. VMware released this article as the resolution to the problem. Here are the steps that they detailed:


To deploy vCenter Operations Manager in a vCenter Server environment with an Essentials Plus or Standard cluster of three ESX hosts:
  1. Remove one of the ESX hosts from the cluster so that it resides directly under the parent datacenter.
  2. On that ESX host, deploy the vCenter Operations Manager vApp, specifying static IP addresses.
  3. Power on the vApp before moving the host back into the cluser to ensure IP settings are picked up.
  4. License the solution in vCenter.
  5. Move the ESX host with vCenter Operations back into the cluster.
Note: Static IP addresses are required.

Warning: The steps above dissolve the vApp container and an error is displayed. Disregard the error message and continue moving the host into the cluster.

Moving the ESX host with the vCenter Operations vApp back into the cluster results in the addition of the two virtual machines (the UI virtual machine and the Analytics virtual machine) to the cluster, without the vApp container. vCenter Operations Manager 5.x continues to function normally when this happens.

Also, in case you need it, the default logins for vCOPS are:
Default Administration:

Default Root Login:

Dustin Shaw

Monday, September 8, 2014

User exceeded the maximum of 250 objects of type "objtMessage".

I have users that are regularly exceeding 250 objtMessage objects in Exchange 2010. What this really means is that the user has over 250 Messages open at once. While this seems high, when you consider third party tools and other such things, 250 is really not that much. I actually have a couple of users that regularly spike over 500 on a daily basis.

Whenever this behavior exhibits itself, it's in the form of ERROR in the Application log with EventID 9646. The text reads:

Mapi session "64807ed0-b1b3-4938-92e3-b9dbd9cfe67b: /o=Company/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=User Name" exceeded the maximum of 250 objects of type "objtMessage".

The default setting for this (and other open item limits in Exchange 2010) is detailed in Microsoft's Exchange Store Limits article.

To change from the default settings, open the registry on your Mailbox Server, and navigate to the following key:

Update/create the DWORD named objtMessage and give it the decimal value that you need (up it to 500).

Don't forget if you are in a DAG to repeat the registry entries on all Mailbox Servers.

Dustin Shaw

Wednesday, July 30, 2014

The symbolic link cannot be followed

When you setup symbolic links on a server that point to another server, you will by default run into the inability for a client computer to follow the links with the following error:

The symbolic link cannot be followed because its type is disabled.

This is because the ability to traverse from one remote system to another across the symbolic link is disabled by default. You can see what is disabled and what is enabled on a computer by running the fsutil command:

>fsutil behavior query eymlinkevaluation
Local to local symbolic links are enabled.
Local to remote symbolic links are enabled.
Remote to local symbolic links are disabled.
Remote to remote symbolic links are disabled.

You have two methods to enable this - enable it locally on each machine, or enable it via Group Policy.


The downsides to enabling it locally are obvious, but sometimes you just need it on one stubborn computer *right now* and can't wait for GP. To enable Remote to Remote symbolic links, run the following command:
fsutil behavior set symlinkevaluation R2R:1

Similarly, you can change the settings for Local to Local (L2L), Local to Remote (L2R), and Remote to Local (R2L) by using 1 for enabled and 0 for disabled.

Group Policy

To enable (or disable) Remote to Remote symbolic links in Group Policy, create a new GPO Policy (or edit a current one), and edit it. Navigate to:
Computer Configuration -> Administrative Templates -> System -> Filesystem
You can then set the settings how you want in Selectively allow the evaluation of a symbolic link

Once you've created your new GPO, test it and validate that it is successfully applied using gpresult /R and rsop.

Monday, July 28, 2014

How to use Group Policy to allow the users to chose any screensaver except (None)

I just found one of the most beautiful Group Policies that I've ever come across:

How to use Group Policy to allow the users to chose any screensaver except (None)

This post is from Group Policy Central, and is 4 years old, but I've verified that it works properly with Windows 7 and 8, and is just a beautifully done Group Policy. Thanks Kevin for creating it and thank Alan for sharing.

The below is excerpts from the posting:

Step 1. Edit a Group Policy Object (GPO) that is targeted to the users accounts you wan to apply this policy
Step 2. Navigate to User Configuration > Preferences > Windows Settings > Registry then from the menu click on Action > New > Registry Item

Step 3. Select “Update” from the Action then type “Control Panel\Desktop” in the Key Path: text field then type “SCRNSAVE.EXE”  in the Value Name text field and “C:\Windows\System32\scrnsave.scr” in the Value data: text field.

Step 4. Click on the Common tab and then tick “Item-level targeting” and then click the “Targeting…” button.

Now we will target the screen saver to apply only when the “HKCU\Control Panel\Desktop\SCRNSAVE.EXE” registry key does NOT exist as this means the screen saver has been configured to “(None)”.
Step 5. Click on “New Item” then the “Registry Match” option.

Step 6. Select the “Value exists” Match type” then type “Control Panel\Desktop” in the key path field and then type “SCRNSAVE.EXE” in the value name field

Step 7. Click back on the targeting setting in the top pane and press “F8” which changes the option to “does not exist” then click OK and OK.

This policy will now apply the blank screen saver on the next group policy refresh to all targeted users whenever they select the “(None)”.

Saturday, July 19, 2014

Installing Exchange Server 2010 SP3 Rollup 6

To get the permissions correct for installing Rollups on Exchange 2010 SP3, you will need to either disable UAC (not recommended) or you will need to launch the Rollup installer from an elevated command prompt (Right click and Run as Administrator) with the following command:
msiexec /update Exchange2010-KB2936871-x64-en.msp

This will allow the rollup to install properly. Other words, it will Roll Back and say that it Ended Prematurely.

Another note on Rollup 6 for SP3 is that it takes (at least in my environment) an extremely long time to generate native images for .NET assemblies. One of my servers took 45 minutes for this process. Wait it out and you'll be able to get it installed, just plan your windows accordingly.

Dustin Shaw

Thursday, May 15, 2014

The wizard was interrupted before VMware Tools could be completely installed.

Upon attempting to install VMware Tools on a Windows Server 2008R2 server, I received an error stating "The wizard was interrupted before VMware Tools could be completely installed."

After looking around on the internet, the closest thing I could find was this post by David Homer, detailing a similar issue with VMware Tools on VMware Workstation.

I tried his fixes (remove VMware Tools registry key, remove the vmtools service), but none of them allowed me to get past the problem.

What eventually worked was doing a search in the registry for all references to "vmware" and removing all keys having to do with VMware Tools - make sure you don't remove any of the keys referring to your SCSI devices/other hardware; just the VMware Tools entries.

I believe it was most likely the Installer registry entries that were mucking it up.

After a quick reboot to refresh the registry, VMware Tools installed with no issues.

Dustin Shaw

Wednesday, April 30, 2014

Graylog2 Version Check Errors

I noticed periodic warnings in my Graylog2 0.20.1 instance saying:
WARN : org.graylog2.periodical.VersionCheckThread - Could not perform version check

The fix for this (as detailed here in Google Groups by Lennart Koopmann) is to set an undocumented flag in your /etc/graylog2.conf as follows:
versionchecks = false

Dustin Shaw

Tuesday, April 29, 2014

Graylog2 Extractors for FortiGate

UPDATE: I've created the JSON version of this for Graylog 1.0 in a new post here.

Graylog2 is a really powerful log monitor, but it needs some customization when it comes to specific devices.

Pointing a Fortinet FortiGate firewall to Graylog2 results in mediocre usage, due to the syslog field not getting extracted properly. Essentially you end up with all of the data just dumped into the message field.

To point the FortiGate to Graylog2, open the FortiGate console and config the syslog settings:
config log syslogd setting

Then enable and point the logging to your server:
set status enable
set server
set port 514
set facility syslog

Once Graylog2 is successfully receiving the logs from the FortiGate, you will need to use extractors for custom processing of the specific syslog messages using the Drools Rule File in Graylog2. You can read more in Graylog2's help file "Custom message rewriting/processing."

To implement the drl file, edit your graylog2.conf file and uncomment the following line, making sure that it points to your drl file:rules_file = /etc/graylog2.drl

Then create your drl file with the rules that you need. Below are the extractors that I use for FortiGate. These are most of the fields that I've run across in the messages that seemed to be of value to me. If there are others, then follow the template and add them. Please notice that some of the fields (like action) don't have quotes around the variable, but others (like app) do, and there are different patterns to match them. There are also a couple of exceptionones (service, status, and vd) that sometimes have quotes and sometimes don't have quotes, so I've got rules in there to take both into account.

The only one that I can't seem to get working correctly is msg one that is supposed to replace the "message" field with the contents of "msg." I'll update here if I can get it working.

To implement the drl file, restart your graylog2-server service, and watch the logfile to make sure that there are no errors on start up.

# Graylog2 Extractors
import org.graylog2.plugin.Message
import java.util.regex.Matcher
import java.util.regex.Pattern
import java.text.DateFormat
import java.text.ParseException

# Fortigate
rule "Fortigate source rewrite"
    m : Message ( message matches ".+devname=.+\\sdevid=.+" )
    Matcher matcher = Pattern.compile("^.+\\sdevname=(\\S+)\\s").matcher(m.getMessage());
    if (matcher.find()) {

    Matcher action = Pattern.compile("^.+\\saction=(\\S+)\\s").matcher(m.getMessage());
    if (action.find()) {

    Matcher app = Pattern.compile("^.+\\sapp=\"(\\S+)\"").matcher(m.getMessage());
    if (app.find()) {

    Matcher appact = Pattern.compile("^.+\\sappact=(\\S+)\\s").matcher(m.getMessage());
    if (appact.find()) {

    Matcher appcat = Pattern.compile("^.+\\sappcat=\"(\\S+)\"").matcher(m.getMessage());
    if (appcat.find()) {

    Matcher applist = Pattern.compile("^.+\\sapplist=\"(\\S+)\"").matcher(m.getMessage());
    if (applist.find()) {

    Matcher attack = Pattern.compile("^.+\\sattack=\"(\\S+)\"").matcher(m.getMessage());
    if (attack.find()) {

    Matcher devid = Pattern.compile("^.+\\sdevid=(\\S+)\\s").matcher(m.getMessage());
    if (devid.find()) {

    Matcher dir = Pattern.compile("^.+\\sdir=(\\S+)\\s").matcher(m.getMessage());
    if (dir.find()) {

    Matcher dstcountry = Pattern.compile("^.+\\sdstcountry=\"(\\S+)\"").matcher(m.getMessage());
    if (dstcountry.find()) {

    Matcher dstintf = Pattern.compile("^.+\\sdstintf=\"(\\S+)\"").matcher(m.getMessage());
    if (dstintf.find()) {

    Matcher dstip = Pattern.compile("^.+\\sdstip=(\\S+)\\s").matcher(m.getMessage());
    if (dstip.find()) {

     Matcher dtype = Pattern.compile("^.+\\sdtype=\"(\\S+)\"").matcher(m.getMessage());
    if (dtype.find()) {

    Matcher duration = Pattern.compile("^.+\\sduration=(\\S+)\\s").matcher(m.getMessage());
    if (duration.find()) {

    Matcher error_reason = Pattern.compile("^.+\\serror_reason=\"(\\S+)\"").matcher(m.getMessage());
    if (error_reason.find()) {

    Matcher eventtype = Pattern.compile("^.+\\seventtype=(\\S+)\\s").matcher(m.getMessage());
    if (eventtype.find()) {

     Matcher file = Pattern.compile("^.+\\sfile=\"(\\S+)\"").matcher(m.getMessage());
    if (file.find()) {

    Matcher group = Pattern.compile("^.+\\sgroup=\"(\\S+)\"").matcher(m.getMessage());
    if (group.find()) {

    Matcher hostname = Pattern.compile("^.+\\shostname=\"(\\S+)\"").matcher(m.getMessage());
    if (hostname.find()) {

    Matcher identidx = Pattern.compile("^.+\\sidentidx=(\\S+)\\s").matcher(m.getMessage());
    if (identidx.find()) {

    Matcher init = Pattern.compile("^.+\\sinit=(\\S+)\\s").matcher(m.getMessage());
    if (init.find()) {

    Matcher locip = Pattern.compile("^.+\\slocip=(\\S+)\\s").matcher(m.getMessage());
    if (locip.find()) {

    Matcher locport = Pattern.compile("^.+\\slocport=(\\S+)\\s").matcher(m.getMessage());
    if (locport.find()) {

    Matcher logid = Pattern.compile("^.+\\slogid=(\\S+)\\s").matcher(m.getMessage());
    if (logid.find()) {

    Matcher mode = Pattern.compile("^.+\\smode=(\\S+)\\s").matcher(m.getMessage());
    if (mode.find()) {

    Matcher msg = Pattern.compile("^.+\\smsg=\"(\\S+)\"").matcher(m.getMessage());
    if (msg.find()) {

    Matcher outintf = Pattern.compile("^.+\\soutintf=\"(\\S+)\"").matcher(m.getMessage());
    if (outintf.find()) {

    Matcher peer_notif = Pattern.compile("^.+\\speer_notif=\"(\\S+)\"").matcher(m.getMessage());
    if (peer_notif.find()) {

    Matcher policyid = Pattern.compile("^.+\\spolicyid=(\\S+)\\s").matcher(m.getMessage());
    if (policyid.find()) {

    Matcher profile = Pattern.compile("^.+\\sprofile=\"(\\S+)\"").matcher(m.getMessage());
    if (profile.find()) {

    Matcher profiletype = Pattern.compile("^.+\\sprofiletype=\"(\\S+)\"").matcher(m.getMessage());
    if (profiletype.find()) {

    Matcher proto = Pattern.compile("^.+\\sproto=(\\S+)\\s").matcher(m.getMessage());
    if (proto.find()) {

     Matcher quarskip = Pattern.compile("^.+\\squarskip=\"(\\S+)\"").matcher(m.getMessage());
    if (quarskip.find()) {

    Matcher rcvdbyte = Pattern.compile("^.+\\srcvdbyte=(\\S+)\\s").matcher(m.getMessage());
    if (rcvdbyte.find()) {

    Matcher rcvdpkt = Pattern.compile("^.+\\srcvdpkt=(\\S+)\\s").matcher(m.getMessage());
    if (rcvdpkt.find()) {

    Matcher ref = Pattern.compile("^.+\\sref=\"(\\S+)\"").matcher(m.getMessage());
    if (ref.find()) {

    Matcher remip = Pattern.compile("^.+\\sremip=(\\S+)\\s").matcher(m.getMessage());
    if (remip.find()) {

    Matcher remport = Pattern.compile("^.+\\sremport=(\\S+)\\s").matcher(m.getMessage());
    if (remport.find()) {

    Matcher result = Pattern.compile("^.+\\sresult=(\\S+)\\s").matcher(m.getMessage());
    if (result.find()) {

    Matcher role = Pattern.compile("^.+\\srole=(\\S+)\\s").matcher(m.getMessage());
    if (role.find()) {

    Matcher sentbyte = Pattern.compile("^.+\\ssentbyte=(\\S+)\\s").matcher(m.getMessage());
    if (sentbyte.find()) {

    Matcher sentpkt = Pattern.compile("^.+\\ssentpkt=(\\S+)\\s").matcher(m.getMessage());
    if (sentpkt.find()) {

    Matcher service = Pattern.compile("^.+\\sservice=\"(\\S+)\"").matcher(m.getMessage());
    if (service.find()) {
    } else {
      Matcher servicenq = Pattern.compile("^.+\\sservice=(\\S+)\\s").matcher(m.getMessage());
      if (servicenq.find()) {

    Matcher srccountry = Pattern.compile("^.+\\ssrccountry=\"(\\S+)\"").matcher(m.getMessage());
    if (srccountry.find()) {

    Matcher srcintf = Pattern.compile("^.+\\ssrcintf=\"(\\S+)\"").matcher(m.getMessage());
    if (srcintf.find()) {

    Matcher srcip = Pattern.compile("^.+\\ssrcip=(\\S+)\\s").matcher(m.getMessage());
    if (srcip.find()) {

    Matcher srcport = Pattern.compile("^.+\\ssrcport=(\\S+)\\s").matcher(m.getMessage());
    if (srcport.find()) {

    Matcher stage = Pattern.compile("^.+\\sstage=(\\S+)\\s").matcher(m.getMessage());
    if (stage.find()) {

    Matcher status = Pattern.compile("^.+\\sstatus=\"(\\S+)\"").matcher(m.getMessage());
    if (status.find()) {
    } else {
      Matcher statusnq = Pattern.compile("^.+\\sstatus=(\\S+)\\s").matcher(m.getMessage());
      if (statusnq.find()) {

    Matcher subtype = Pattern.compile("^.+\\ssubtype=(\\S+)\\s").matcher(m.getMessage());
    if (subtype.find()) {

    Matcher transport = Pattern.compile("^.+\\stransport=(\\S+)\\s").matcher(m.getMessage());
    if (transport.find()) {

    Matcher type = Pattern.compile("^.+\\stype=(\\S+)\\s").matcher(m.getMessage());
    if (type.find()) {

    Matcher trandisp = Pattern.compile("^.+\\strandisp=(\\S+)\\s").matcher(m.getMessage());
    if (trandisp.find()) {

    Matcher transip = Pattern.compile("^.+\\stransip=(\\S+)\\s").matcher(m.getMessage());
    if (transip.find()) {

    Matcher user = Pattern.compile("^.+\\suser=\"(\\S+)\"").matcher(m.getMessage());
    if (user.find()) {

    Matcher utmaction = Pattern.compile("^.+\\sutmaction=(\\S+)\\s").matcher(m.getMessage());
    if (utmaction.find()) {

    Matcher utmevent = Pattern.compile("^.+\\sutmevent=(\\S+)\\s").matcher(m.getMessage());
    if (utmevent.find()) {

    Matcher vd = Pattern.compile("^.+\\svd=\"(\\S+)\"").matcher(m.getMessage());
    if (vd.find()) {
    } else {
      Matcher vdnq = Pattern.compile("^.+\\svd=(\\S+)\\s").matcher(m.getMessage());
      if (vdnq.find()) {

    Matcher virus = Pattern.compile("^.+\\svirus=\"(\\S+)\"").matcher(m.getMessage());
    if (virus.find()) {

    Matcher vpntunnel = Pattern.compile("^.+\\svpntunnel=\"(\\S+)\"").matcher(m.getMessage());
    if (vpntunnel.find()) {
    Matcher xauthgroup = Pattern.compile("^.+\\sxauthgroup=\"(\\S+)\"").matcher(m.getMessage());
    if (xauthgroup.find()) {

    Matcher xauthuser = Pattern.compile("^.+\\sxauthuser=\"(\\S+)\"").matcher(m.getMessage());
    if (xauthuser.find()) {


Dustin Shaw

Wednesday, April 23, 2014

Install ElasticSearch 0.90.10 on CentOS 6

Graylog2 0.20.x requires ElasticSearch 0.90.10. To fullfil this requirement, you will need to manually download and install the RPM for ElasticSearch.

Download ElasticSearch 0.90.10 from the ElasticSearch Downloads page here.

Save the file and upload it to your CentOS 6 server.

Install Java 1.7:
#yum install java-1.7.0-openjdk.x86_64

Install the RPM:
#rpm -ivh elasticsearch-0.90.10.noarch.rpm

Stop the elasticsearch service so that we can update the cluster name:
#service elasticsearch stop

Edit the /etc/elasticsearch/elasticsearch.yml file to update your variable. Ex: graylog2_production

Update any additional settings needed and save the file. I recommend updating the and path.logs to custom directories.

Start the elasticsearch service and set it to run on startup:
#service elasticsearch start
#chkconfig elasticsearch on

Check your logs to make sure that it started properly and joined the cluster (if there is an existing one).

For Graylog2, the recommended settings are also to increase the open file limit to at least 64000 as seen in the Configuring and tuning ElasticSearch for Graylog2 >v0.20.0 documentation. I did this by increasing the max number of ulimit open file below.

Edit /etc/sysctl.conf and add the following line at the end:
fs.file-max = 65536

Save the file. Next edit /etc/security/limits.conf and add the following lines:
*               soft    nproc           65535
*               hard    nproc           65535
*               soft    nofile          65535
*               hard    nofile          65535

Save the file and restart the server.
#shutdown -r now

Once restarted, verify that the max open file ulimit has been increased.
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30507
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Additional recommended settings are to increase the ES_HEAP_SIZE. I did this by editing /etc/init.d/elasticsearch and adding the following line after checkJava under start():

They recommend that you leave 50% of your memory for other system functions, and I had 4 Gig of RAM, hence the 2g setting.

Dustin Shaw

Ship RHEL 6 or CentOS6 syslogs

Reference more for myself than for y'all, but you get the shared benefit.

To ship the logs from RHEL6 or Centos 6 to a remote syslog server, edit the following file: /etc/rsyslog.conf

At the bottom of the file, remove the comment from the remote-host entry, and update with your server name or IP. Example:
*.* @@

Restart the rsyslog service:
service rsyslog restart

Make sure you are receiving the logs at your syslog server.

Dustin Shaw

Install VMware Tools on RHEL 6 or CentOS 6

Below is the process to install VMware Tools on RHEL 6 or CentOS 6. This guide is more here for me than anyone else, but I hope that you can benefit from it.

Install the Pre-Requisites:
yum install make gcc kernel-devel kernel-headers glibc-headers perl

Start the VMware Tools installation process on your VM:

Mount the VMware Tools installation media:
mkdir /mnt/cd
mount /dev/cdrom /mnt/cd
Expected warning:
mount: block device /dev/sr0 is write-protected, mounting read-only

Extract the installer:
cp /mnt/cd/VMwareTools-9.0.10-1481436.tar.gz /tmp/
umount /mnt/cd
cd /tmp
tar xvf VMwareTools-9.0.10-1481436.tar.gz
cd vmware-tools-distrib/

Install tools (accepting all defaults):
sudo ./ -d

Reboot the VM to verify that the service starts up automatically as expected.
shutdown -r now

Dustin Shaw