Graylog and stream alerts on message terms

In a stream, you might be wondering why you have search hits looking for a basic string in message when the query string is built for you as you click the little magnifying glass “+” next to a record’s message field, yet none when you actually copy/paste the query or typing it out — and hence no alert triggering.

This is due to terms.

Our devoted folks at Graylog actually warn us about it: http://docs.graylog.org/en/2.1/pages/streams/alerts.html#alerts

Here is an example. I want to trigger an alert when value condition message:”has not been migrated”. The screenshot below shows the associated terms. In this particular example, I am getting values from a file via graylog-sidecar and nxlog from a Windows machine.

graylog_terms_example_1

If I just create my stream alert with “Field content value condition”, with “message” contains “has not been migrated” (i.e. as in just typing it out or copy-pasting the text), it won’t work. But there is a trick. A bit ugly, but well, it works.

Given the screenshot above, build your query simply by clicking on the magnifying glass and search. There, you can (need to) actually adjust it. It will retain its “magic”. And it will match. Yet, nothing stands out in that query compared to when you type it out, right?

Now, let’s look at the Elasticsearch query to see how it was built. Click on Show query.

graylog_terms_showquery

The query string is probably not the one you would have expected. This is because of terms. Each arrow points to a letter of what I am looking for: “h”, “a”, “s” and so on.

graylog_terms_esquery
(the whole query doesn’t show obviously)

You need that odd-looking query into your alerting criteria instead of your “plain text” query. It will now trigger as expected.

Know your event size in Graylog2

It may be interesting to know your event size in Graylog — or as a matter of fact, the size of any document you store in your Elasticsearch backend. You can use it for capacity planning based on your average EPS (Event Per Second), monitor its fluctuation, etc…

This specific example will be Graylog2 centric. It works on Graylog 1.3 and Elasticsearch 1.7.x.

1. Create a file containing the json content to create a new template — let’s call it enable_size.json, such as:

{
  "template": "graylog*",
  "mappings": {
    "message": {
      "_size" : { "enabled" : true, "store" : true }
    }
  }
}

2. Using a simple curl command from your shell, send that JSON content to your elasticsearch, such as:

curl -XPUT http://$ES_HOST:9200/_template/custom-graylog -d @enable_size.json

You should get a {“acknowledged”:”true”} in response. All your newly created graylog indices will now store the document’s size.

3. You can then manually cycle your deflector in the UI (System > Indices) or use the POST hook in graylog’s API, for example:

curl -uadmin:admin -XPOST http://192.168.1.155:12900/system/deflector/cycle

4. In order to query for the average size of documents for the current index, as pointed to by the graylog deflector alias:

curl -s -XPOST http://$ES_HOST:9200/graylog2_deflector/_search?pretty -d '
{
  "query" : { "match_all": {}},
  "aggs": {
    "avg_doc_size": {
      "avg": {
        "field": "_size"
      }
    }
  }
}'


NB: The deflector alias name could be graylog_deflector and not graylog2_deflector as above.

5. If you want to use this within Zabbix for example (and a shameless plug for my Zabbix template on Graylog’s marketplace), you could embed the above into a small bash script as a Zabbix externalscript, leveraging the excellent JQ tool and start plotting the data.

#!/bin/bash

[[ -z "$1" ]] && echo "Hostname needed" && exit -1

curl -s -XPOST http://$1:9200/graylog2_deflector/_search?pretty -d '
{
  "query" : { "match_all": {}},
  "aggs": {
    "avg_doc_size": {
      "avg": {
        "field": "_size"
       }
     }
    }
}' |jq '.aggregations.avg_doc_size.value'