Dev environment with Elastic Cloud on Kubernetes (ECK)

Ep #11: Discover the easiest way to setup a development environment for the Elasticsearch stack that can easily scale to a production cluster.

Disclaimer

The views and opinions expressed in this article are solely my own and do not necessarily reflect the views of Elastic, where I am currently employed. This article is for informational purposes only and should not be understood as an endorsement of any particular company, product, or policy. It should not be mistaken for official communication from Elastic. I share my perspectives based on my personal experiences and learning, and these are subject to change over time.

Introduction

I'll show you how to set up a scalable development environment for the Elastic stack, whether you plan to run it on your laptop or in production.

If you use the same automation, experience gained locally on your laptop benefits production

To achieve this, I'm going to assume that you are already familiar with the Elastic stack and with Kubernetes. For more information about either of those, please refer to their official documentation.

We are going to use an orchestration solution called ECK or Elastic Cloud on Kubernetes that allows you to run an Elastic stack anywhere you have a Kubernetes cluster running. Either on your laptop or on any cloud provider.

The provided Kubernetes manifests are intended for a basic setup with Elasticsearch and Kibana, but can be easily extended for more complex configurations.

It's worth noting that the manifests provided are not meant for production environments. However, they do serve as an excellent starting point for those who want to explore Elasticsearch without incurring cloud resource costs or the complexities of setting up a cluster from the binaries.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

Giuseppe Santoro

If you need 1-1 mentoring sessions, feel free to check my Mentorcruise profile.

Evolution of the Elastic stack

I remember the first time that I tried Elasticsearch.

It was very early in my career. Almost 10 years ago now.

I just downloaded the Elasticsearch binary on my laptop, provided a basic configuration, started the binary and that was it. Elasticsearch was ready to be queried at port 9200.

I remember, a while after that, installing a piece of software called Head, to manage the state of the cluster. You could see the number of shards, their state, and the number of nodes and get a basic interactive client to run Elasticsearch queries. You didn't need much more than that.

Only later, I discovered a more complex web UI called Kopf (Kopf is German for Head) with some more functionality. Things were still quite simple.

I don't even remember Kibana being there at the time.

Now in 2023, after a year working at Elastic (the company behind Elasticsearch), I can say that Elasticsearch is much more than a search engine.

The Elastic ecosystem has now three major solutions:

• Observability

• Security

• Search

and a lot more moving parts to install, configure and manage:

• Kibana

• Beats

• Elastic-agent

• Logstash

• Fleet server

• Map Server

• APM Server

The point here is not to name all those parts or to explain what those components are responsible for. The official documentation is a much better place for that.

The point here is to explain that, given the current complexity of this ecosystem, it can take some time to configure and manage all those components.

In the next sections, we are going to describe some options to configure the Elastic stack.

Alternatives to install the Elastic stack

There are currently many ways to deploy a cluster for the Elastic stack.

Some are simpler but cost money (like the Elastic Cloud), and some are free but complex (like the raw binaries). We are going to explore here, in more detail, a solution that stands in the middle of this spectrum. A solution that is both simple and free.

But first, let me briefly enumerate what are the possible ways to install an Elastic stack today.

You can either go with the fully automated one-click solution called Elastic cloud, where the stack is automatically installed on a choice of 3 major cloud providers (namely AWS, GCP and Azure).

Alternatively, you can download the binaries and configure every component yourself. You can achieve this either on-premise on bare metal machines, on the cloud, or your laptop. This is a much more convoluted way that is only suited for those of you who want to go "the hard way".

In the past, there was another way to set up a cluster that used Elastic helm charts. The linked GitHub repository was archived on May 16, 2023, and it is no longer the suggested way to install the Elastic stack.

If you have contributed to the Elastic stack code in the past (either as an Elastic employee or an external contributor), you might have used a single binary tool called elastic-package to easily set up a Elastic stack on your laptop via docker-compose. This tool provides much more than a development environment but that is outside the scope of this article.

I believe that using the elastic-package solution is great for development purposes since it is free and easy. With a single command, you get a ready-to-use cluster that is already configured with all the major components.

Unfortunately, this solution has two major downsides:

• Not flexible enough. It doesn't have the degree of flexibility to let me choose which components to run and it doesn't scale up to a full production cluster with multiple nodes.

• It depends on Docker and Docker-compose. It doesn't work with other container runtimes like Containerd.

What I believe is the best way to spin up a development environment, that solves all those issues, is ECK (Elastic Cloud on Kubernetes).

ECK run the Elastic stack on a Kubernetes cluster by providing a Kubernetes operator and Kubernetes CRDs (Custom Resource Definitions) for each of the Elastic stack components.

The Kubernetes cluster can either be running on the cloud (in any cloud provider) or on your laptop (if you have a beefy machine like I do with enough memory).

My development environment setup

In this section, we will explore how to set up a development environment for the Elastic Stack using ECK.

The first step is to create a Kubernetes cluster to run the stack.

k3d cluster create elastic

We have chosen to use K3d because it requires fewer resources than other solutions such as Minikube or Kind.

You can learn more about K3d in my article Kubernetes development environments.

After setting up the Kubernetes cluster, you can use Helm to install the ECK operator. Execute the following commands to install the operator and the necessary CRDs. These CRDs will be referenced in the Kubernetes manifests for the next step.

helm repo add elastic https://helm.elastic.co
helm repo update
helm install elastic-operator elastic/eck-operator --namespace elastic-system --create-namespace

Now you can finally provide the Kubernetes manifests to install the Elastic stack. Here we are using Kustomize so that we can specify a list of Kubernetes manifests to apply in order.

kubectl apply -k manifests

Kubernetes manifests and the kustomization.yaml to configure them are provided under the directory manifests. Here is the content of the kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - es.yml
  - kibana.yml

And here is the manifest for Elasticsearch from the file es.yml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: elastic-system
spec:
  version: 8.10.2
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 2Gi
              cpu: 1
            limits:
              memory: 2Gi

and here is the one for Kibana from kibana.yml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic-system
spec:
  version: 8.10.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2.5Gi
            cpu: 2

As you can see the setup is quite minimal but it has potential to grow.

You will need to wait a few minutes for the manifests to apply and for all components to be operational.

The ECK operator creates by default a elastic user with some random credentials. You can get the password for this user by using the following command

kubectl -n elastic-system get secret elasticsearch-es-elastic-user -o jsonpath="{.data.elastic }" | base64 -d

Finally, you need to port forward Kibana to a local port with the command

kubectl port-forward -n elastic-system svc/kibana-kb-http 5601:5601

You need to have the previous command running in a shell for the entire time you want to interact with Kibana.

You can access your Elastic Stack by opening a web browser and navigating to https://localhost:5601. Then, enter your credentials obtained from the previous step.

Conclusion

This setup might feel a bit convoluted but it is by far the easiest way to have a scalable setup that can run both on your laptop and in production.

In the future, I plan to expand on this setup to add more resources like a Fleet server, elastic-agent, and a few other components.

I'm also planning to write another article where I combine those commands in a Taskfile to make it easier to automate.

Stay tuned for more content on this topic.