High Scalable Stream Processing with Apache Kafka and Azure Kubernetes Service - Part 2

Building a high scalable stream processing pipeline with Kubernetes

Published Mar 3, 2019 in microsoft, azure
Scroll

Last article I shared a high level architecture of how we can build a high scalable stream processing pipeline with Kubernetes. This articles talks about how we can deploy that onto Azure. We’ll be setting up our resources and environment so that we can deploy our demo later on.

Architecture Recap


Here is an overview of the architecture.

High Scalable Kafka and AKS Architecture

High Scalable Kafka and AKS Architecture


Deploy Azure Kubernetes Servie


Before we begin, let’s deploy Azure Kubernetes Service so that we can deploy our containerized applications later on. Azure Kubernetes Service is a managed Kubernetes Service by Azure so that we do not need to manage our own instance of Kubernetes and at the same time being able to deploy our containerized applications easily.

Head over to Azure Portal, let’s start by creating a Kubernetes Service resource.

Create Kubernetes Service Resource

Create Kubernetes Service Resource

Next, this is pretty much a step by step to create a Kubernetes cluster.

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

Create Kubernetes Cluster Step

That’s it, Your Kubernetese Cluster deployment is underway!


Create a Virtual Network

Before we provision ourselves a Kafka service on HDInsight, there is one intermediary step we need to do. We need to setup a virtual network. You might wonder, why do we need a virtual network? This is for us to do VNET Peering later on with our provisioned Azure Kubernetes Service and HDInsight so that the services could communicate to each other.

Create Virtual Networking Step

Create Virtual Networking Step

Create Virtual Networking Step

Create Virtual Networking Step

This is where we setup our VNET peering. We got to ensure that we set this in AKS as well as with the VNET we created for HDInsight. Do note that AKS VNET is represented by the following format “MC___".

Setup Virtual Networking Peering

Setup Virtual Networking Peering

Setup Virtual Networking Peering

Setup Virtual Networking Peering

Virtual Networking Peering Completed

Virtual Networking Peering Completed

Now, let’s configure the Virtual Network Peering for HDInsight network we created earlier.

Setup Virtual Networking Peering

Setup Virtual Networking Peering

Setup Virtual Networking Peering

Setup Virtual Networking Peering

Virtual Networking Peering Completed

Virtual Networking Peering Completed


Deploy Kafka on HDInsight

Microsoft & Azure has got an amazing support for Open Source technologies including Apache’s open source technology, Apache Kafka. HDInsight has an array of technologies supported. The brilliant thing about HDInsight + Kafka is, there isn’t a need to manage Kafka yourself. Azure HDInsight takes care of the service for you.


Let’s create our HDInsight. Head over to “Analytics” -> “HDInsight”.

Create HDInsight Resoure

Create HDInsight Resoure

Create HDInsight Resoure

Create HDInsight Resoure

Create HDInsight Resoure

Create HDInsight Resoure

I’ve chosen the default size and settings here for convenience. You can allocate the right resource on your project needs.

Create HDInsight Resoure

Create HDInsight Resoure


That’s it! We’re all set and ready to go!

Summary


I’ve showed you how you can create a Kubernetes cluster on Azure, setup a virtual networking for HDInsight, setup VNET peering with AKS together with HDInsight’s Virtual Network. Finally, creating Kafka on HDInsight. Now that all our resources are created, we can build a simple demo in the next article to stream Kafka events into our Kubernetes containers.


Cheers!

High Scalable Stream Processing with Apache Kafka and Azure Kubernetes Service - Part I

Building a high scalable stream processing pipeline with Kubernetes

Published Feb 3, 2019 in microsoft, azure