The ELK Stack is three tools, Elasticsearch, Logstash and Kibana, that collect the logs from every part of a system, store them in one searchable place and draw them as charts.
When something breaks, you search one place instead of logging in to ten servers.
Why raw logs aren't enough
A log is a line of text an application writes each time something happens, such as an order placed or an error thrown. A busy system writes thousands of them a minute, and they pile up in the wrong shape for answering questions.
- They're scattered. The web app, the payment service and the delivery service each write to their own files, on their own machines. A single customer's failed order leaves a trail across all three.
- They're inconsistent. One service writes plain text, another writes JSON, a third uses its own date format. Searching across them with
grepmeans knowing every format in advance. - They disappear. Containers restart and servers get replaced. If the logs only lived on that machine, they go with it.
What you want is every log line in one place, in one shape, searchable by its fields: 'every error from the payment service since 9am'. The three letters of ELK each handle one part of that: collect, store, understand.
Logstash: collect and clean
Logstash is a processing pipeline. Every pipeline has three stages:
- Inputs read events from somewhere: a log file, a network port, a message queue.
- Filters reshape each event: split a text line into named fields, parse the timestamp, drop noise, remove a field that shouldn't be stored.
- Outputs send the finished event on, usually to Elasticsearch.
The filter stage does the most useful work. A plain text line such as ERROR orders order=4812 means nothing to a search engine; once Logstash has turned it into , and fields, you can filter and count by each of them. Common filters include and for pulling fields out of text, for reading timestamps and for renaming or removing fields.
Wondering how this was made? How we make these