Last article I shared a high level architecture of how we can build a high scalable stream processing pipeline with Kubernetes. This articles talks about how we can deploy that onto Azure. We’ll be setting up our resources and environment so that we can deploy our demo later on.
Here is an overview of the architecture.
Deploy Azure Kubernetes Servie
Before we begin, let’s deploy Azure Kubernetes Service so that we can deploy our containerized applications later on. Azure Kubernetes Service is a managed Kubernetes Service by Azure so that we do not need to manage our own instance of Kubernetes and at the same time being able to deploy our containerized applications easily.
Head over to Azure Portal, let’s start by creating a Kubernetes Service resource.
Next, this is pretty much a step by step to create a Kubernetes cluster.
That’s it, Your Kubernetese Cluster deployment is underway!
Create a Virtual Network
Before we provision ourselves a Kafka service on HDInsight, there is one intermediary step we need to do. We need to setup a virtual network. You might wonder, why do we need a virtual network? This is for us to do VNET Peering later on with our provisioned Azure Kubernetes Service and HDInsight so that the services could communicate to each other.
This is where we setup our VNET peering. We got to ensure that we set this in AKS as well as with the VNET we created for HDInsight. Do note that AKS VNET is represented by the following format “MC_
Now, let’s configure the Virtual Network Peering for HDInsight network we created earlier.
Deploy Kafka on HDInsight
Microsoft & Azure has got an amazing support for Open Source technologies including Apache’s open source technology, Apache Kafka. HDInsight has an array of technologies supported. The brilliant thing about HDInsight + Kafka is, there isn’t a need to manage Kafka yourself. Azure HDInsight takes care of the service for you.
Let’s create our HDInsight. Head over to “Analytics” -> “HDInsight”.
I’ve chosen the default size and settings here for convenience. You can allocate the right resource on your project needs.
That’s it! We’re all set and ready to go!
I’ve showed you how you can create a Kubernetes cluster on Azure, setup a virtual networking for HDInsight, setup VNET peering with AKS together with HDInsight’s Virtual Network. Finally, creating Kafka on HDInsight. Now that all our resources are created, we can build a simple demo in the next article to stream Kafka events into our Kubernetes containers.