Logstash grok pattern for nginx

Do you use ELK stack in your env ? if so have you noticed that there is no real nice way to integrate nginx logs into it?Last time I was trying to submit a ticket even going to IRC channel to question a proven way of parsing nginx logs into it. While so, I need to mention that if otherwise your log will be a string of unrecognized and patterned data that later you can not manipulate. There is also a fully customization for creating your own grok pattern using online grok pattern testing tools as well.
The short howto
Install filebeat and configure it to ship the data to the logstash server. There is a new method of segregating and labeling logs but I prefer the good old method being:
document_type: nginx_access
Try to enable and start the filebeat service and make sure that the logs state successful connection. Now on the nginx side we are dealing with combined log format which is the default if nothing mentioned as in example below in your config directives:
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

Time now to head to logstash server.
Here is the tricky bit since the new method tidy up the logstash into input>filter>output sort of discipline. We need to follow them and basically it boils to an input.conf that universally listens for any data on logstash port, the output that sends the filtered data to an elasticsearch server. Now the filter segment is where all our interests will reside, and here is the sample of my nginx that does the trick:
[root@elk 0]# cat /etc/logstash/conf.d/11-nginx-filter.conf
filter {
if [type] == "nginx_access" {
grok {
match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => [ "message" ]
}
}
}

If you happen to use the geoip then the geoip section needs to be added to the filter as:
geoip {
source => "clientip"
}

Remember the field name is the actual source field holds the info required to geolocate. Note also that the geoip plugin needs to be installed in your elk stack:
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.