Confluent Kafka Kubernetes Operator — Setup and Playing!

Confluent released their Kafka Kubernetes Operator this week (as part of their CP 5.3 release). The TL;DR summary is this gives you a straight forward way to deploy your own clustered & secured Apache Kafka on Kubernetes (ZooKeeper, Apache Kafka) along with the cool bits (Kafka Connect, KSQL, Schema Registry, REST Proxy, Control Center).

This is pretty cool as it opens the way for Kafka deployments to on premises (eg., OpenShift or Rancher) and eases the way for clouds deploys (EKS, AKS, GKE). It also automates security configuration setup (setting up kerberos authentication in Kafka is a job for the patient!).

So, let’s see how to see this up.

Machine Setup

I recently got a new Mac — so this was a good opportunity to document the steps necessary to start from a clean machine. This demonstration uses the Google (GCP) Kubernetes Engine

Kubernetes Helm CLI

First — let’s install the kubernetes command line tools (assuming brew is already installed)

Google Cloud SDK

I’ll be using the Google (GCP) Kubernetes Engine. You’ll need to install the google-cloud-sdk . Google has great quick-start instructions . In short, download, extract and add to your PATH

Confluent Kubernetes Operator Helm bundle

Next, we actually need to download the Confluent Helm bundle. This Confluent bundle comprises Helm charts, templates, and scripts used to deploy Confluent Operator and Confluent Platform components for your Kubernetes cluster. Remember this is an Enterprise feature (ie., paid for if used in production).

Download and extract the bundle from Confluent Platform Operations from https://docs.confluent.io/current/installation/operator/co-deployment.html

GCP Kubernetes Engine Provision

Adding a Kubernetes clusters

Navigating to GCP Kubernetes Engine ; click “create cluster”

Add a node pool

For the new cluster; add a lot of capacity. I found 16 nodes of 2 vCPU’s was enough to get going

Adding resource quota

Initially I hit a problem where I could not provision enough resource due to a quota on CPUs.

If this happens to you navigate to IAM Admin Quotas (https://console.cloud.google.com/iam-admin/quotas) and increase to something like 32 for the Compute Engine API CPUs maximum

Setup GCP Client

Now you’ll need to initialises and authenticate your setup

Next, setup a Service Account

Install Confluent Operator

Good — now my laptop can communicate with a remote GCP Kubernetes Engine. Now to install the Confluent Operator. These steps are pretty much verbatim from https://docs.confluent.io/current/installation/operator/co-deployment.html

Install ZooKeeper

First — let’s install 3 zookeeper nodes

Setup DNS (Optional)

Not strictly necessary — but if you want to have direct (external) access to your Kafka cluster, you’ll need to provide an accessible DNS that can be pushed into your LISTENER config. The settings are in helm/providers/gcp.yaml

Install Kafka brokers

A very similar approach to setting up 3 kafka brokers

Install Schema Registry

Now to install a 2 node schema registry

Install Kafka Connect

Now for Kafka connect

Install Confluent Control Center

Install Confluent KSQL

Running Workload

After the deployments have completed, the running workloads should look like this

Does it work — let’s test it …

Test Kafka from within the cluster

Launch a bash session on a kafka broker

kubectl -n operator exec -it kafka-0 bash

Create a local kafka.properties file

And then run a test

Test local KSQL client against remote KSQL server

Establish a port mapping tunnel

And start a ksql session (locally) connecting to a remote server

Test Confluent Control Center

Now let’s test access to Confluent Control Center. Establish a port mapping tunnel for port 9021

And navigate to http://localhost:9021

The initial login is admin and the default password is Developer1.

Teardown and Cleanup

Be sure to remove the deployment once you’ve finished testing. For GCP Kubernetes Engine it’s pretty much down to deleting the cluster

Final Thoughts

A managed service anything is preferable to rolling things yourself. However, the Kafka Kubernetes Operator made a complex deployment pretty seem-less. This is pretty battle tested too as Confluent run their own public cloud offering (Confluent Cloud) on Kubernetes.

I had a fully clustered, secured and highly available streaming platform of my own with a few yaml files and some cli magic.

This is an amazing outcome — built in minutes (not days or weeks). This was so much easier than the last time I did this!

Day job: data steaming & system architecture. Night gig: IoT and random project hacking