Graylog and stream alerts on message terms

In a stream, you might be wondering why you have search hits looking for a basic string in message when the query string is built for you as you click the little magnifying glass “+” next to a record’s message field, yet none when you actually copy/paste the query or typing it out — and hence no alert triggering.

This is due to terms.

Our devoted folks at Graylog actually warn us about it: http://docs.graylog.org/en/2.1/pages/streams/alerts.html#alerts

Here is an example. I want to trigger an alert when value condition message:”has not been migrated”. The screenshot below shows the associated terms. In this particular example, I am getting values from a file via graylog-sidecar and nxlog from a Windows machine.

graylog_terms_example_1

If I just create my stream alert with “Field content value condition”, with “message” contains “has not been migrated” (i.e. as in just typing it out or copy-pasting the text), it won’t work. But there is a trick. A bit ugly, but well, it works.

Given the screenshot above, build your query simply by clicking on the magnifying glass and search. There, you can (need to) actually adjust it. It will retain its “magic”. And it will match. Yet, nothing stands out in that query compared to when you type it out, right?

Now, let’s look at the Elasticsearch query to see how it was built. Click on Show query.

graylog_terms_showquery

The query string is probably not the one you would have expected. This is because of terms. Each arrow points to a letter of what I am looking for: “h”, “a”, “s” and so on.

graylog_terms_esquery
(the whole query doesn’t show obviously)

You need that odd-looking query into your alerting criteria instead of your “plain text” query. It will now trigger as expected.

Leave a Reply