Ubuntu 18 package management is no different to Ubuntu lts previous version, again it lacked including or even recommending of Apache Kafka in its software centre, which is  distributed streaming platform. Many find the installation of Kafka difficult. Actually, it’s pretty straightforward and easy process, and this tutorial will show you step-by-step instructions on how to install Apache Kafka on Ubuntu 18.04 

Related article: Whatis Apache Kafka

• .Essentials for Kafka

        In Kafka the communication between the clients and the servers is done with a simple, high-performance, language agnostic. This protocol is versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in many languages. Apache Kafka runs on JAVA, if its already present on your system proceed to next step or simply run this command on Ubuntu.  Make sure if Ubuntu is updated, the below command will install the newer version of Java.

 To see that the java has been installed properly check the version using the command below:

java -version

openjdk version “10.0.2” 2018–07–17

OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4)

OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode)

1. Download Kafka

Start with downloading the newest version of Apache Kafka from its official website available here. The newer version is always the best and works like a charm Ubuntu.

2. Starting Zookeeper

Zookeeper is a key value store used to maintain server state of Kafka. Kafka uses Zookeeper, so we need to first start a ZooKeeper server. For now, you can use the convenience script packaged with Kafka to get a quick-and-dirty ZooKeeper instance. The & makes the command run in the background. Simply, type the following command given below:

sudo mkdir -p /data/zookeeper. sudo chown -R hadoop:hadoop /data.

You will recieve a confirmation that the Kafka server is started.

3. Extract Kafka and Change Directory

Similar to how you started Zookeeper, there are two files that represent the file to start (bin/kafka-server-start.sh) and configure (config/server.properties) the broker server. To extract use the command below.

tar -xvzf kafka_2.12–2.2.0.tgz && cd kafka_2.12–2.2.0

tar -xvzf kafka_2.12–2.2.0.tgz && cd kafka_2.12–2.2.0

4. Start the Kafka server

To start the Kafka server, run this command.

bin/kafka-server-start.sh config/server.properties &

...

2019-11-14 01:44:31,217] INFO Kafka version: 2.2.0 (org.apache.kafka.common.utils.AppInfoParser)

[2019-11-14 01:44:31,218] INFO Kafka commitId: 05fcfde8f69b0349 (org.apache.kafka.common.utils.AppInfoParser)

[2019-11-14 01:44:31,219] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

If you check the running jobs now, you should be able to see both of these:

1]-  Running                 bin/zookeeper-server-start.sh config/zookeeper.properties &

[2]+ Running                 bin/kafka-server-start.sh 

comfit/server.properties &

 

5. Validate Kafka Installation in Ubuntu 18.04

Before you can start putting data into your cluster, you need to create a topic to which the data will belong. The partitions options lets you decide how many brokers you want your data to be split between. You can set this option to 3(initially).

And, the replication factor shows how many copies of you data you want (in case one of the brokers goes down, you still have your data on the others).

To do this, run the command:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mytechlite

Once you create the topic, you should see a confirmation message.

And you can delete it with this command:

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytechlite

6. Create Producer in Apache Kafka

 The “producer” is the process that puts data into our Kafka cluster. The command line tools in the bin directory provide us with a console producer, that inputs data into the cluster every time your enter text into the console. The Producer allows an application to publish a stream of records to one or more Kafka topics.

To start the console producer, run the command:

kafka_2.11–2.0.1/bin/kafka-console-producer.sh — broker-list localhost:9092 — topic QUEUE

You should now see a command prompt, in which you can enter a bunch of text which gets inserted into the Kafka cluster you just created every time you hit enter.

7. Create Consumer in Apache Kafka on Ubuntu 18.04

The only thing left to do is read data from the cluste, to install kafka on Ubuntu. This  allows an application to subscribe to one or more topics and process the stream of records produced to them.

Simply, run the command:

kafka_2.11–2.0.1/bin/kafka-console-consumer.sh — bootstrap-server localhost:1111 — topic Mytechlite — from-beginning

When you run the above command, you should immediately see all the messages that you input using the producer, logged onto your console.

Additionally, if you input anymore messages with the producer while the consumer is running, you should see it output into the console in real time.

That’s it. You have successfully install Apache Kafka on Ubuntu 18.04