Elastic Stack Terminologies - Technologies for Elasticsearch Logging

Blog:   / 1f6cd948d4ec  
Free Newsletter:   / subscribe  

This week, I spent time in implementing logging in my Spring boot application using elasticsearch, but I felt a lot of the common technologies involved were confusing. So here we are understanding what each of the popular tech in the Elastic Stack means.

Apache Lucene
Apache Lucene is a high-performance, open-source text search engine library written in Java. A search engine library is not a search engine itself, but rather, a toolkit that can be used to build a search engine.

Search engine library vs Search engine
A toolkit comes with a lot of important methods that enable search ranking, tokenization, scoring, query searching, etc, which are crucial for any search engine’s working. However, a library such as Lucene is not responsible for the UI interface, APIs required, custom logic, data storage, and pre and post-processing of results. A library is like an engine, and a search engine is a complete car.

Elasticsearch
Elasticsearch is a search engine built on top of Apache Lucene. On top of Lucene’s capabilities, it comes with a UI dashboard, an “out-of-the-box” implementation, and distributed architecture.

Kibana
Kibana is an open-source data exploration and visualization tool, very often used with Elasticsearch to visualize the data collected inside it.

Logstash
Logstash is an opensource data pipeline tool that dynamically collects, parses, transforms data and forwards it to a location from where the tool such as Elasticsearch can operate on it. It’s best used for heavy processing tasks before Elasticsearch can consume the data. This is ideal for the server side.

Filebeat
Filebeat is an open-source and lightweight log shipper. Unlike Logstash, Filebeat does not process the data but rather just ships it to Logstash or Elasticsearch directly. In scenarios where no heavy processing is involved, logs can be directly sent to Elasticsearch for consumption using Filebeat without needing Logstash. Filebeat monitors logs continuously and ships them as soon as they are generated. This is ideal for client machines.

This sums up all the basics you need to get started with logging to Elasticsearch!

#elasticsearch #springboot #java #backend #devops