In previous post we installed elasticsearch and kibana. Locally stuff works fine but to consume logs from outside world we need to add more love to that setup.

The next step is to decide how to feed logs to ELK. The first thing that comes to our mind is using Elasticsearch target that most logging libraries provide. Even if that option is not supported out of the box there should be the 3rd party extension / plugin. This is quite simple and usually well documented.

However it might be not the best option. In case of connectivity issues log event may not reach ELK. To be honest I never tried such setup and agree that some implementations MAY provide some workaround like queing, buffering etc. But there is another option that will work with any logger and even with other logs – not just ones generated by your applications. Meet log shipping.

The idea behind log shipping is dead simple. Logs are stored in their default place – file system. And there is small service / daemon that monitors folders, detects changes in files and sends those changes to centralized log storage – Elasticsearch in our case.

Why log shipping is better that direct sending logs to ELK? First of all it is more reliable. In case of network issues logs still be be stored on the disk. Local file system is one of the most reliable storage, by the way. Anyone remember when their hard disk got broken? Even if you do, I bet that network issues happen more frequently. And log shipping mitigates both risks. So, once log is stored on the disk, shipper will try to send it to ELK retrying until it is there. Nothing gets lost. ELK contains lightweight file based shipper named Filebeat.

One could say that adding one more process causes performance degradation of the whole system. But that should not be the case. When sending logs to ELK is done by the application, its performance suffers from network latency, ELK processing time etc. Even if sending is done in asynchronous manner, long delays would cause queueing of requests thus increasing the possibility of losing the data. Imagine the situation when application gets crushed or restarted with long buffer of yet unsent log messages. It will be lost and no-one will notice it.

Another important feature of Filebeat is a backpressure handling. It can detect that receiving party is under load and slows down, preventing queueing and bottlenecks.

Despite the fact that Filebeat can forward logs directly to Elastisearch, more common setup is to put Logstash between them. One of the reason is that Logstash has rich transformation and enriching capabilities. It parses incoming log messages extracting metadata like log level, thread id, environment details and so on. All these will be indexed by Elasticsearch adding more context to the data.

OK, so we’ve defined all participants of our setup. In the next posts we’ll see how they should be configured to provide nice logging experience.

Stay tuned!