Reroute processor for Elasticsearch

Ep #1: Explore the innovative reroute processor for Elasticsearch. Learn how it effectively routes container logs into multiple data streams, enhancing log management.

Recently Elastic has published a blog post about a new processor called reroute processor that is available from Elastic 8.8.0 version.

Disclaimer: the reroute processor is currently in technical preview.

A customer can use this processor in a custom ingest pipeline to split a single data stream into multiple data streams based on one more fields in the document being ingested.

More information about processor can either be found in the blog post or in the official documentation. I'll provide here a quick glossary for some of the concepts necessary to fully understand the article or the documentation and a brief personal note.

Glossary

• data stream

  • Let you store append-only data across multiple hidden backing indices

  • you write to the latest backing index

  • you read from all backing indices

  • requires an index template

  • used in Elastic integrations

  • every document in a data stream needs to have a @timestamp date field

• index template

  • contains: settings and mappings

  • can be made of multiple component templates which are reusable component for settings and mappings.

  • map to one or more indices using an index pattern

• ingest pipeline

  • allow you to process documents ingested into Elasticsearch

  • made of multiple tasks called processors.

Personal note

My team is involved in developing the Kubernetes integration for Elastic.

I've been working on using this processor to enable routing container logs from Kubernetes to different data streams.

Imagine running a Kubernetes cluster with two containers respectively for Nginx and Prometheus. You want to collect logs from those two containers with the Kubernetes integration and then reroute each log entry to two different data streams.

Since you can have ingest pipelines attached to a data stream, you can use custom ingest pipelines for each different application.

Work on the reroute processor is still in progress but it looks very promising.

Resources

• reroute processor - github PR

---

Want to connect?

👉 Follow me on LinkedIn and Twitter.

Giuseppe Santoro

If you need 1-1 mentoring sessions, feel free to check my Mentorcruise profile.