Graylog2 Version Check Errors

I noticed periodic warnings in my Graylog2 0.20.1 instance saying:
WARN : org.graylog2.periodical.VersionCheckThread - Could not perform version check

The fix for this (as detailed here in Google Groups by Lennart Koopmann) is to set an undocumented flag in your /etc/graylog2.conf as follows:
versionchecks = false


------
Dustin Shaw
VCP

Graylog2 Extractors for FortiGate

UPDATE: I've created the JSON version of this for Graylog 1.0 in a new post here.

Graylog2 is a really powerful log monitor, but it needs some customization when it comes to specific devices.

Pointing a Fortinet FortiGate firewall to Graylog2 results in mediocre usage, due to the syslog field not getting extracted properly. Essentially you end up with all of the data just dumped into the message field.

To point the FortiGate to Graylog2, open the FortiGate console and config the syslog settings:
config log syslogd setting

Then enable and point the logging to your server:
set status enable
set server 192.168.40.50
set port 514
set facility syslog

Once Graylog2 is successfully receiving the logs from the FortiGate, you will need to use extractors for custom processing of the specific syslog messages using the Drools Rule File in Graylog2. You can read more in Graylog2's help file "Custom message rewriting/processing."


To implement the drl file, edit your graylog2.conf file and uncomment the following line, making sure that it points to your drl file:rules_file = /etc/graylog2.drl

Then create your drl file with the rules that you need. Below are the extractors that I use for FortiGate. These are most of the fields that I've run across in the messages that seemed to be of value to me. If there are others, then follow the template and add them. Please notice that some of the fields (like action) don't have quotes around the variable, but others (like app) do, and there are different patterns to match them. There are also a couple of exceptionones (service, status, and vd) that sometimes have quotes and sometimes don't have quotes, so I've got rules in there to take both into account.

The only one that I can't seem to get working correctly is msg one that is supposed to replace the "message" field with the contents of "msg." I'll update here if I can get it working.

To implement the drl file, restart your graylog2-server service, and watch the logfile to make sure that there are no errors on start up.

# Graylog2 Extractors
import org.graylog2.plugin.Message
import java.util.regex.Matcher
import java.util.regex.Pattern
import java.text.DateFormat
import java.text.ParseException

# Fortigate
rule "Fortigate source rewrite"
  when
    m : Message ( message matches ".+devname=.+\\sdevid=.+" )
  then
    Matcher matcher = Pattern.compile("^.+\\sdevname=(\\S+)\\s").matcher(m.getMessage());
    if (matcher.find()) {
      m.addField("source", matcher.group(1));
    }

    Matcher action = Pattern.compile("^.+\\saction=(\\S+)\\s").matcher(m.getMessage());
    if (action.find()) {
      m.addField("action", action.group(1));
    }

    Matcher app = Pattern.compile("^.+\\sapp=\"(\\S+)\"").matcher(m.getMessage());
    if (app.find()) {
      m.addField("app", app.group(1));
    }

    Matcher appact = Pattern.compile("^.+\\sappact=(\\S+)\\s").matcher(m.getMessage());
    if (appact.find()) {
      m.addField("appact", appact.group(1));
    }

    Matcher appcat = Pattern.compile("^.+\\sappcat=\"(\\S+)\"").matcher(m.getMessage());
    if (appcat.find()) {
      m.addField("appcat", appcat.group(1));
    }

    Matcher applist = Pattern.compile("^.+\\sapplist=\"(\\S+)\"").matcher(m.getMessage());
    if (applist.find()) {
      m.addField("applist", applist.group(1));
    }

    Matcher attack = Pattern.compile("^.+\\sattack=\"(\\S+)\"").matcher(m.getMessage());
    if (attack.find()) {
      m.addField("attack", attack.group(1));
    }

    Matcher devid = Pattern.compile("^.+\\sdevid=(\\S+)\\s").matcher(m.getMessage());
    if (devid.find()) {
      m.addField("devid", devid.group(1));
    }

    Matcher dir = Pattern.compile("^.+\\sdir=(\\S+)\\s").matcher(m.getMessage());
    if (dir.find()) {
      m.addField("dir", dir.group(1));
    }

    Matcher dstcountry = Pattern.compile("^.+\\sdstcountry=\"(\\S+)\"").matcher(m.getMessage());
    if (dstcountry.find()) {
      m.addField("dstcountry", dstcountry.group(1));
    }

    Matcher dstintf = Pattern.compile("^.+\\sdstintf=\"(\\S+)\"").matcher(m.getMessage());
    if (dstintf.find()) {
      m.addField("dstintf", dstintf.group(1));
    }

    Matcher dstip = Pattern.compile("^.+\\sdstip=(\\S+)\\s").matcher(m.getMessage());
    if (dstip.find()) {
      m.addField("dstip", dstip.group(1));
    }

     Matcher dtype = Pattern.compile("^.+\\sdtype=\"(\\S+)\"").matcher(m.getMessage());
    if (dtype.find()) {
      m.addField("dtype", dtype.group(1));
    }

    Matcher duration = Pattern.compile("^.+\\sduration=(\\S+)\\s").matcher(m.getMessage());
    if (duration.find()) {
      m.addField("duration", duration.group(1));
    }

    Matcher error_reason = Pattern.compile("^.+\\serror_reason=\"(\\S+)\"").matcher(m.getMessage());
    if (error_reason.find()) {
      m.addField("error_reason", error_reason.group(1));
    }

    Matcher eventtype = Pattern.compile("^.+\\seventtype=(\\S+)\\s").matcher(m.getMessage());
    if (eventtype.find()) {
      m.addField("eventtype", eventtype.group(1));
    }

     Matcher file = Pattern.compile("^.+\\sfile=\"(\\S+)\"").matcher(m.getMessage());
    if (file.find()) {
      m.addField("file", file.group(1));
    }

    Matcher group = Pattern.compile("^.+\\sgroup=\"(\\S+)\"").matcher(m.getMessage());
    if (group.find()) {
      m.addField("group", group.group(1));
    }

    Matcher hostname = Pattern.compile("^.+\\shostname=\"(\\S+)\"").matcher(m.getMessage());
    if (hostname.find()) {
      m.addField("hostname", hostname.group(1));
    }

    Matcher identidx = Pattern.compile("^.+\\sidentidx=(\\S+)\\s").matcher(m.getMessage());
    if (identidx.find()) {
      m.addField("identidx", identidx.group(1));
    }

    Matcher init = Pattern.compile("^.+\\sinit=(\\S+)\\s").matcher(m.getMessage());
    if (init.find()) {
      m.addField("init", init.group(1));
    }

    Matcher locip = Pattern.compile("^.+\\slocip=(\\S+)\\s").matcher(m.getMessage());
    if (locip.find()) {
      m.addField("locip", locip.group(1));
    }

    Matcher locport = Pattern.compile("^.+\\slocport=(\\S+)\\s").matcher(m.getMessage());
    if (locport.find()) {
      m.addField("locport", locport.group(1));
    }

    Matcher logid = Pattern.compile("^.+\\slogid=(\\S+)\\s").matcher(m.getMessage());
    if (logid.find()) {
      m.addField("logid", logid.group(1));
    }

    Matcher mode = Pattern.compile("^.+\\smode=(\\S+)\\s").matcher(m.getMessage());
    if (mode.find()) {
      m.addField("mode", mode.group(1));
    }

    Matcher msg = Pattern.compile("^.+\\smsg=\"(\\S+)\"").matcher(m.getMessage());
    if (msg.find()) {
      m.addField("message", msg.group(1));
    }

    Matcher outintf = Pattern.compile("^.+\\soutintf=\"(\\S+)\"").matcher(m.getMessage());
    if (outintf.find()) {
      m.addField("outintf", outintf.group(1));
    }

    Matcher peer_notif = Pattern.compile("^.+\\speer_notif=\"(\\S+)\"").matcher(m.getMessage());
    if (peer_notif.find()) {
      m.addField("peer_notif", peer_notif.group(1));
    }

    Matcher policyid = Pattern.compile("^.+\\spolicyid=(\\S+)\\s").matcher(m.getMessage());
    if (policyid.find()) {
      m.addField("policyid", policyid.group(1));
    }

    Matcher profile = Pattern.compile("^.+\\sprofile=\"(\\S+)\"").matcher(m.getMessage());
    if (profile.find()) {
      m.addField("profile", profile.group(1));
    }

    Matcher profiletype = Pattern.compile("^.+\\sprofiletype=\"(\\S+)\"").matcher(m.getMessage());
    if (profiletype.find()) {
      m.addField("profiletype", profiletype.group(1));
    }

    Matcher proto = Pattern.compile("^.+\\sproto=(\\S+)\\s").matcher(m.getMessage());
    if (proto.find()) {
      m.addField("proto", proto.group(1));
    }

     Matcher quarskip = Pattern.compile("^.+\\squarskip=\"(\\S+)\"").matcher(m.getMessage());
    if (quarskip.find()) {
      m.addField("quarskip", quarskip.group(1));
    }

    Matcher rcvdbyte = Pattern.compile("^.+\\srcvdbyte=(\\S+)\\s").matcher(m.getMessage());
    if (rcvdbyte.find()) {
      m.addField("rcvdbyte", rcvdbyte.group(1));
    }

    Matcher rcvdpkt = Pattern.compile("^.+\\srcvdpkt=(\\S+)\\s").matcher(m.getMessage());
    if (rcvdpkt.find()) {
      m.addField("rcvdpkt", rcvdpkt.group(1));
    }

    Matcher ref = Pattern.compile("^.+\\sref=\"(\\S+)\"").matcher(m.getMessage());
    if (ref.find()) {
      m.addField("ref", ref.group(1));
    }

    Matcher remip = Pattern.compile("^.+\\sremip=(\\S+)\\s").matcher(m.getMessage());
    if (remip.find()) {
      m.addField("remip", remip.group(1));
    }

    Matcher remport = Pattern.compile("^.+\\sremport=(\\S+)\\s").matcher(m.getMessage());
    if (remport.find()) {
      m.addField("remport", remport.group(1));
    }

    Matcher result = Pattern.compile("^.+\\sresult=(\\S+)\\s").matcher(m.getMessage());
    if (result.find()) {
      m.addField("result", result.group(1));
    }

    Matcher role = Pattern.compile("^.+\\srole=(\\S+)\\s").matcher(m.getMessage());
    if (role.find()) {
      m.addField("role", role.group(1));
    }

    Matcher sentbyte = Pattern.compile("^.+\\ssentbyte=(\\S+)\\s").matcher(m.getMessage());
    if (sentbyte.find()) {
      m.addField("sentbyte", sentbyte.group(1));
    }

    Matcher sentpkt = Pattern.compile("^.+\\ssentpkt=(\\S+)\\s").matcher(m.getMessage());
    if (sentpkt.find()) {
      m.addField("sentpkt", sentpkt.group(1));
    }

    Matcher service = Pattern.compile("^.+\\sservice=\"(\\S+)\"").matcher(m.getMessage());
    if (service.find()) {
      m.addField("service", service.group(1));
    } else {
      Matcher servicenq = Pattern.compile("^.+\\sservice=(\\S+)\\s").matcher(m.getMessage());
      if (servicenq.find()) {
        m.addField("service", servicenq.group(1));
      }
    }

    Matcher srccountry = Pattern.compile("^.+\\ssrccountry=\"(\\S+)\"").matcher(m.getMessage());
    if (srccountry.find()) {
      m.addField("srccountry", srccountry.group(1));
    }

    Matcher srcintf = Pattern.compile("^.+\\ssrcintf=\"(\\S+)\"").matcher(m.getMessage());
    if (srcintf.find()) {
      m.addField("srcintf", srcintf.group(1));
    }

    Matcher srcip = Pattern.compile("^.+\\ssrcip=(\\S+)\\s").matcher(m.getMessage());
    if (srcip.find()) {
      m.addField("srcip", srcip.group(1));
    }

    Matcher srcport = Pattern.compile("^.+\\ssrcport=(\\S+)\\s").matcher(m.getMessage());
    if (srcport.find()) {
      m.addField("srcport", srcport.group(1));
    }

    Matcher stage = Pattern.compile("^.+\\sstage=(\\S+)\\s").matcher(m.getMessage());
    if (stage.find()) {
      m.addField("stage", stage.group(1));
    }

    Matcher status = Pattern.compile("^.+\\sstatus=\"(\\S+)\"").matcher(m.getMessage());
    if (status.find()) {
      m.addField("status", status.group(1));
    } else {
      Matcher statusnq = Pattern.compile("^.+\\sstatus=(\\S+)\\s").matcher(m.getMessage());
      if (statusnq.find()) {
        m.addField("status", statusnq.group(1));
      }
    }

    Matcher subtype = Pattern.compile("^.+\\ssubtype=(\\S+)\\s").matcher(m.getMessage());
    if (subtype.find()) {
      m.addField("subtype", subtype.group(1));
    }

    Matcher transport = Pattern.compile("^.+\\stransport=(\\S+)\\s").matcher(m.getMessage());
    if (transport.find()) {
      m.addField("transport", transport.group(1));
    }

    Matcher type = Pattern.compile("^.+\\stype=(\\S+)\\s").matcher(m.getMessage());
    if (type.find()) {
      m.addField("type", type.group(1));
    }

    Matcher trandisp = Pattern.compile("^.+\\strandisp=(\\S+)\\s").matcher(m.getMessage());
    if (trandisp.find()) {
      m.addField("trandisp", trandisp.group(1));
    }

    Matcher transip = Pattern.compile("^.+\\stransip=(\\S+)\\s").matcher(m.getMessage());
    if (transip.find()) {
      m.addField("transip", transip.group(1));
    }

    Matcher user = Pattern.compile("^.+\\suser=\"(\\S+)\"").matcher(m.getMessage());
    if (user.find()) {
      m.addField("user", user.group(1));
    }

    Matcher utmaction = Pattern.compile("^.+\\sutmaction=(\\S+)\\s").matcher(m.getMessage());
    if (utmaction.find()) {
      m.addField("utmaction", utmaction.group(1));
    }

    Matcher utmevent = Pattern.compile("^.+\\sutmevent=(\\S+)\\s").matcher(m.getMessage());
    if (utmevent.find()) {
      m.addField("utmevent", utmevent.group(1));
    }

    Matcher vd = Pattern.compile("^.+\\svd=\"(\\S+)\"").matcher(m.getMessage());
    if (vd.find()) {
      m.addField("vd", vd.group(1));
    } else {
      Matcher vdnq = Pattern.compile("^.+\\svd=(\\S+)\\s").matcher(m.getMessage());
      if (vdnq.find()) {
        m.addField("vd", vdnq.group(1));
      }
    }

    Matcher virus = Pattern.compile("^.+\\svirus=\"(\\S+)\"").matcher(m.getMessage());
    if (virus.find()) {
      m.addField("virus", virus.group(1));
    }

    Matcher vpntunnel = Pattern.compile("^.+\\svpntunnel=\"(\\S+)\"").matcher(m.getMessage());
    if (vpntunnel.find()) {
      m.addField("vpntunnel", vpntunnel.group(1));
    }
   
    Matcher xauthgroup = Pattern.compile("^.+\\sxauthgroup=\"(\\S+)\"").matcher(m.getMessage());
    if (xauthgroup.find()) {
      m.addField("xauthgroup", xauthgroup.group(1));
    }

    Matcher xauthuser = Pattern.compile("^.+\\sxauthuser=\"(\\S+)\"").matcher(m.getMessage());
    if (xauthuser.find()) {
      m.addField("xauthuser", xauthuser.group(1));
    }

  end

------
Dustin Shaw
VCP

Install ElasticSearch 0.90.10 on CentOS 6

Graylog2 0.20.x requires ElasticSearch 0.90.10. To fullfil this requirement, you will need to manually download and install the RPM for ElasticSearch.

Download ElasticSearch 0.90.10 from the ElasticSearch Downloads page here.

Save the file and upload it to your CentOS 6 server.

Install Java 1.7:
#yum install java-1.7.0-openjdk.x86_64

Install the RPM:
#rpm -ivh elasticsearch-0.90.10.noarch.rpm

Stop the elasticsearch service so that we can update the cluster name:
#service elasticsearch stop

Edit the /etc/elasticsearch/elasticsearch.yml file to update your cluster.name variable. Ex:
cluster.name: graylog2_production

Update any additional settings needed and save the file. I recommend updating the path.data and path.logs to custom directories.

Start the elasticsearch service and set it to run on startup:
#service elasticsearch start
#chkconfig elasticsearch on

Check your logs to make sure that it started properly and joined the cluster (if there is an existing one).

For Graylog2, the recommended settings are also to increase the open file limit to at least 64000 as seen in the Configuring and tuning ElasticSearch for Graylog2 >v0.20.0 documentation. I did this by increasing the max number of ulimit open file below.

Edit /etc/sysctl.conf and add the following line at the end:
fs.file-max = 65536

Save the file. Next edit /etc/security/limits.conf and add the following lines:
*               soft    nproc           65535
*               hard    nproc           65535
*               soft    nofile          65535
*               hard    nofile          65535

Save the file and restart the server.
#shutdown -r now

Once restarted, verify that the max open file ulimit has been increased.
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30507
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Additional recommended settings are to increase the ES_HEAP_SIZE. I did this by editing /etc/init.d/elasticsearch and adding the following line after checkJava under start():
ES_HEAP_SIZE=2g

They recommend that you leave 50% of your memory for other system functions, and I had 4 Gig of RAM, hence the 2g setting.

------
Dustin Shaw
VCP

Ship RHEL 6 or CentOS6 syslogs

Reference more for myself than for y'all, but you get the shared benefit.

To ship the logs from RHEL6 or Centos 6 to a remote syslog server, edit the following file: /etc/rsyslog.conf


At the bottom of the file, remove the comment from the remote-host entry, and update with your server name or IP. Example:
*.* @@192.168.40.15:514

Restart the rsyslog service:
service rsyslog restart

Make sure you are receiving the logs at your syslog server.

------
Dustin Shaw
VCP

Install VMware Tools on RHEL 6 or CentOS 6

Below is the process to install VMware Tools on RHEL 6 or CentOS 6. This guide is more here for me than anyone else, but I hope that you can benefit from it.


Install the Pre-Requisites:
yum install make gcc kernel-devel kernel-headers glibc-headers perl


Start the VMware Tools installation process on your VM:





Mount the VMware Tools installation media:
mkdir /mnt/cd
mount /dev/cdrom /mnt/cd
Expected warning:
mount: block device /dev/sr0 is write-protected, mounting read-only


Extract the installer:
cp /mnt/cd/VMwareTools-9.0.10-1481436.tar.gz /tmp/
umount /mnt/cd
cd /tmp
tar xvf VMwareTools-9.0.10-1481436.tar.gz
cd vmware-tools-distrib/


Install tools (accepting all defaults):
sudo ./vmware-install.pl -d

Reboot the VM to verify that the service starts up automatically as expected.
shutdown -r now


------
Dustin Shaw
VCP