Logstash - Plugins

Logstash offers various plugins for all three stages of its pipeline (Input, Filter and Output). These plugins help the user to capture logs from various sources like Web Servers, Databases, Over Network Protocols, etc.

After capturing, Logstash can parse and transform the data into meaningful information as required by the user. Lastly, Logstash can send or store that meaningful information to various destination sources like Elasticsearch, AWS Cloudwatch, etc.

Input Plugins

Input plugins in Logstash helps the user to extract and receive logs from various sources. The syntax for using the input plugin is as follows −

Input {
   Plugin name {
      Setting 1……
      Setting 2……..
   }
}

You can download input plugin by using the following command −

>Logstash-plugin install Logstash-input-<plugin name>

The Logstash-plugin utility is present in the bin folder of the Logstash installation directory. The following table has a list of the input plugins offered by Logstash.

Plugin Settings

All the plugins have their specific settings, which helps to specify the important fields like Port, Path, etc., in a plugin. We will discuss the settings of some of the input plugins.

File

This input plugin is used to extract events directly from log or text files present in the input source. It works similar to the tail command in UNIX and save the last read cursor and read only the new appended data from the input file, but it can be changed by using star_position setting. Following are the settings of this input plugin.

Elasticsearch

This particular plugin is used to read the search queries results in an Elasticsearch cluster. The following has the settings used in this plugin −

eventlog

This input plugin reads data from win32 API of windows servers. Followings are the settings of this plugin −

Twitter

This input plugin is used to collect the feed of twitter from its Streaming API. The following table describes the settings of this plugin.

TCP

TCP is used to get the events over the TCP socket; it can read from the user connections or server, which is specified in mode setting. The following table describes the settings of this plugin −

Logstash – Output Plugins

Logstash supports various output sources and in different technologies like Database, File, Email, Standard Output, etc.

The syntax for using the output plugin is as follows −

output {
   Plugin name {
      Setting 1……
      Setting 2……..
   }
}

You can download the output plugin by using the following command −

>logstash-plugin install logstash-output-<plugin name>

The Logstash-plugin utility is present in the bin folder of Logstash installation directory. The following table describes the output plugins offered by Logstash.

All the plugins have their specific settings, which helps to specify the important fields like Port, Path, etc., in a plugin. We will discuss the settings of some of the output plugins.

Elasticsearch

Elasticsearch output plugin enables Logstash to store the output in the specific clusters of Elasticsearch engine. This is one of the famous choices of users because it comes in the package of ELK Stack and therefore, provides end-to-end solutions for Devops. The following table describes the settings of this output plugin.

Email

The email output plugin is used to notify the user, when Logstash generates output. The following table describes the settings for this plugin.

Http

This setting is used to send the output events over http to the destination. This plugin has following settings −

stdout

The stdout output plugin is used to write the output events on the standard output of the command line interface. It is command prompt in windows and terminal in UNIX. This plugin has the following settings −

statsd

It is a network daemon used to send the matrices data over UDP to the destination backend services. It is command prompt in windows and terminal in UNIX. This plugin has following settings −

Filter Plugins

Logstash supports various filter plugins to parse and transform input logs to a more structured and easy to query format.

The syntax for using the filter plugin is as follows −

filter {
   Plugin name {
      Setting 1……
      Setting 2……..
   }
}

You can download the filter plugin by using the following command −

>logstash-plugin install logstash-filter-<plugin name>

The Logstash-plugin utility is present in the bin folder of Logstash installation directory. The following table describes the output plugins offered by Logstash.

Codec plugins

Codec Plugins can be a part of input or output plugins. These Plugins are used to change or format the logging data presentation. Logstash offers multiple codec Plugins and those are as follows −

Build Your Own Plugin

You can also create your own Plugins in Logstash, which suites your requirements. The Logstash-plugin utility is used to create custom Plugins. Here, we will create a filter plugin, which will add a custom message in the events.

Generate the Base Structure

A user can generate the necessary files by using the generate option of the logstash-plugin utility or it is also available on the GitHub.

>logstash-plugin generate --type filter --name myfilter --path c:/tpwork/logstash/lib

Here, type option is used to specify the plugin is either Input, Output or Filter. In this example, we are creating a filter plugin named myfilter. The path option is used to specify the path, where you want your plugin directory to be created. After executing the above mentioned command, you will see that a directory structure is created.

Develop the Plugin

You can find the code file of the plugin in the \lib\logstash\filters folder in the plugin directory. The file extension will be .rb.

In our case, the code file was located inside the following path −

C:\tpwork\logstash\lib\logstash-filter-myfilter\lib\logstash\filters\myfilter.rb

We change the message to − default ⇒ "Hi, You are learning this on howcodex.com" and save the file.

Install the Plugin

To install this plugin, the Gemfile of Logstash need to be modified. You can find this file in the installation directory of Logstash. In our case, it will be in C:\tpwork\logstash. Edit this file using any text editor and add the following text in it.

gem "logstash-filter-myfilter",:path => "C:/tpwork/logstash/lib/logstash-filter-myfilter"

In the above command, we specify the name of the plugin along with where we can find it for installation. Then, run the Logstash-plugin utility to install this plugin.

>logstash-plugin install --no-verify

Testing

Here, we are adding myfilter in one of the previous examples −

logstash.conf

This Logstash config file contains myfilter in the filter section after the grok filter plugin.

input {
   file {
      path => "C:/tpwork/logstash/bin/log/input1.log"
   } 
}
filter {
   grok {
      match => [
         "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} -
            %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?" ]
   }
   myfilter{}
}
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output1.log"
      codec => rubydebug
   }
}

Run logstash

We can run Logstash by using the following command.

>logstash –f logsatsh.conf

input.log

The following code block shows the input log data.

INFO - 48566 - TRANSACTION_START - start

output.log

The following code block shows the output log data.

{
   "path" => "C:/tpwork/logstash/bin/log/input.log",
   "@timestamp" => 2017-01-07T06:25:25.484Z,
   "loglevel" => "INFO",
   "logger" => "TRANSACTION_END",
   "@version" => "1",
   "host" => "Dell-PC",
   "label" => "end",
   "message" => "Hi, You are learning this on howcodex.com",
   "taskid" => "48566",
   "tags" => []
}

Publish it on Logstash

A developer can also publish his/her custom plugin to Logstash by uploading it on the github and following the standardized steps defined by the Elasticsearch Company.

Please refer the following URL for more information on publishing −

https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html