Logstash - Introduction
Logstash is a tool based on the filter/pipes patterns for gathering, processing and generating the logs or events. It helps in centralizing and making real time analysis of logs and events from different sources.
Logstash is written on JRuby programming language that runs on the JVM, hence you can run Logstash on different platforms. It collects different types of data like Logs, Packets, Events, Transactions, Timestamp Data, etc., from almost every type of source. The data source can be Social data, E-commerce, News articles, CRM, Game data, Web trends, Financial data, Internet of Things, Mobile devices, etc.
Logstash General Features
The general features of Logstash are as follows −
Logstash can collect data from different sources and send to multiple destinations.
Logstash can handle all types of logging data like Apache Logs, Windows Event Logs, Data over Network Protocols, Data from Standard Input and many more.
Logstash can also handle http requests and response data.
Logstash provides a variety of filters, which helps the user to find more meaning in the data by parsing and transforming it.
Logstash can also be used for handling sensors data in internet of things.
Logstash is open source and available under the Apache license version 2.0.
Logstash Key Concepts
The key concepts of Logstash are as follows −
Event Object
It is the main object in Logstash, which encapsulates the data flow in the Logstash pipeline. Logstash uses this object to store the input data and add extra fields created during the filter stage.
Logstash offers an Event API to developers to manipulate events. In this tutorial, this event is referred with various names like Logging Data Event, Log Event, Log Data, Input Log Data, Output Log Data, etc.
Pipeline
It comprises of data flow stages in Logstash from input to output. The input data is entered in the pipeline and is processed in the form of an event. Then sends to an output destination in the user or end system’s desirable format.
Input
This is the first stage in the Logstash pipeline, which is used to get the data in Logstash for further processing. Logstash offers various plugins to get data from different platforms. Some of the most commonly used plugins are – File, Syslog, Redis and Beats.
Filter
This is the middle stage of Logstash, where the actual processing of events take place. A developer can use pre-defined Regex Patterns by Logstash to create sequences for differentiating between the fields in the events and criteria for accepted input events.
Logstash offers various plugins to help the developer to parse and transform the events into a desirable structure. Some of the most commonly used filter plugins are – Grok, Mutate, Drop, Clone and Geoip.
Output
This is the last stage in the Logstash pipeline, where the output events can be formatted into the structure required by the destination systems. Lastly, it sends the output event after complete processing to the destination by using plugins. Some of the most commonly used plugins are – Elasticsearch, File, Graphite, Statsd, etc.
Logstash Advantages
The following points explain the various advantages of Logstash.
Logstash offers regex pattern sequences to identify and parse the various fields in any input event.
Logstash supports a variety of web servers and data sources for extracting logging data.
Logstash provides multiple plugins to parse and transform the logging data into any user desirable format.
Logstash is centralized, which makes it easy to process and collect data from different servers.
Logstash supports many databases, network protocols and other services as a destination source for the logging events.
Logstash uses the HTTP protocol, which enables the user to upgrade Elasticsearch versions without having to upgrade Logstash in a lock step.
Logstash Disadvantages
The following points explain the various disadvantages of Logstash.
Logstash uses http, which negatively affects the processing of the logging data.
Working with Logstash can sometimes be a little complex, as it needs a good understanding and analysis of the input logging data.
Filter plugins are not generic, so, the user may need to find the correct sequence of patterns to avoid error in parsing.
In the next chapter, we will understand what the ELK Stack is and how it helps Logstash.
