Deploying Confluent Platform Using Helm Charts on Oracle Kubernetes Engine

Oracle Cloud

Cloud Providers / Oracle Cloud 5 years ago 1042 Views 0

Hello, my name is Pinkesh Valdria, and I'm a Solutions Architect working on Big Data for Oracle Cloud Infrastructure.& &

This post is a follow-up to our post about deploying Confluent on Oracle Cloud Infrastructure Compute instances. Now you can use Terraform automation to deploy Confluent Platform using Helm charts on Oracle Cloud Infrastructure Container Engine for Kubernetes. &

Oracle Cloud Infrastructure Container Engine for Kubernetes

Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) is a fully-managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. Use Container Engine for Kubernetes when your development team wants to reliably build, deploy, and manage cloud-native applications. You specify the compute resources that your applications require, and Container Engine for Kubernetes provisions them on Oracle Cloud Infrastructure in an existing tenancy.

Figure 1: OKE High-Level Architecture

Confluent Open Source Provides a More Complete Distribution of Apache Kafka

Confluent Open Source brings together the best distributed streaming technology from Apache Kafka and takes it to the next level by addressing the requirements of modern enterprise streaming applications. It includes clients for C, C++, Python, and Go programming languages; Connectors for JDBC, Elasticsearch, and HDFS; Confluent Schema Registry for managing metadata for Kafka topics; and Confluent REST Proxy for integrating with web applications.

Figure2: Confluent Platform Components

Helm

Helm& is an open source packaging tool that helps you install applications and services on Kubernetes. Helm uses a packaging format called charts, which are a collection of YAML templates that describe a related set of Kubernetes resources.

Deploying Confluent Platform on Container Engine for Kubernetes

The Terraform automation template performs the following steps:

  1. Deploys a Kubernetes cluster on Oracle Cloud Infrastructure in a new virtual cloud network (VCN), including subnets, a load balancer, a security list, and node pools across three availability domains.
  2. Prepares your local machine to access the Kubernetes cluster, as follows:
    1. Generates a kube configuration file to access the cluster.
    2. Installs kubectl.
    3. Installs Helm.
    4. Adds Confluent Helm charts to the Helm repo.
  3. Installs the Confluent Open Source platform (named my-confluent-oss, but you can change the name).

This Terraform template is available on our cloud partner repository.

Figure 3: Cluster with Node Pools

By default, the Confluent Helm chart deploys three pods for Zookeeper, three pods for Kafka, and one each for Schema Registry, Kafka Rest, Kafka Connect, and KSQL. Grafana and Prometheus monitoring are optional.

Figure 4:& Pods Deployed on Clusters

We hope that you are as excited as we are about Confluent Platform deployment on Container Engine for Kubernetes. Let us know what you think!& & &

Pinkesh Valdria

Principal Solutions Architect, Big Data

https://www.linkedin.com/in/pinkesh-valdria/

Read more

Comments