Installing Kafka and Kafka Visiualisation Tool

Introduction#

In today's data-driven world, real-time data streaming is crucial for building responsive and insightful applications. Apache Kafka has emerged as the leading platform for handling these high-volume data streams. This blog post will guide you through the process of installing Kafka and setting up a visualization tool to effectively monitor and understand your Kafka deployments.

Installing Kafka#

To install Kafka on your local machine you can visit website link given below, this will guide us to install Kafka

Link :- https://kafka.apache.org/quickstart

After visiting this website you will see there are total 8 steps which will guide you to setup kafka and use it. All the steps are given for MAC system, so in this blog we will cover all these steps for our windows system.

Step-1 Get Kafka#

Visit :- https://www.apache.org/dyn/closer.cgi?path=/kafka/3.9.0/kafka_2.13-3.9.0.tgz

You will get this above link to download .tgz file for kafka In the first step after visiting this https://kafka.apache.org/quickstart

After visiting open this link in browser and click on https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz

This will start downloading the .tgz file, after download is completed unzip it into a suitable location on your PC. I have extracted the folder directly in This PC C-drive location

After unzipping the folder rename the folder to Kafka and go to this path → C:\kafka\bin

Here you will see windows folder go inside windows folder → C:\kafka\bin\windows. if you are using mac then you don’t need to go inside windows folder simply stay at C:\kafka\bin

You will get all bat files inside windows folder which are required to run kafka, and all sh files are present outside windows folder which are required to run kafka on MAC.

Step-2 Start Kafka Environment with KRaft#

In-order to start Kafka environment make sure your local environment have java 8+ installed

Apache Kafka server can be run using KRaft mode or ZooKeeper, we will run Kafka server using KRaft since zooKeeper support is deprecated.

Before running KRaft we will first create cluster because all different brokers will belong to one cluster id. So to crate the cluster id follow the below given instructions.

Open cmd with this path C:\kafka\bin\windows and run the following commands, This will generate a random cluster id and set it into a variable.

C:\\kafka\\bin\\windows>kafka-storage.bat random-uuid
8hpjkfgFSemA19plkZH5Vg

C:\\kafka\\bin\\windows>set KAFKA_CLUSTER_ID=8hpjkfgFSemA19plkZH5Vg

Format Log directories by running the following command, log directories are the location where all data of kafka is stored.

kafka-storage.bat format -t %KAFKA_CLUSTER_ID% -c ../../config/kraft/server.properties

Here server.properties file have all the details related to starting the Kafka server with default configurations, you will also see that your kafka server has configured to start on port 9092, so make sure that no other service is running on this port, you can also change the port in server.properties file, we will keep everything default.

Now run this command to start the kafka server

kafka-server-start.bat ../../config/kraft/server.properties

After running the above command you will see the Kafka server started logs at the end

[2025-01-07 17:32:45,371] INFO [SocketServer listenerType=BROKER, nodeId=1] Enabling request processing. (kafka.network.SocketServer)
[2025-01-07 17:32:45,373] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.DataPlaneAcceptor)
[2025-01-07 17:32:45,374] INFO [BrokerServer id=1] Waiting for all of the authorizer futures to be completed (kafka.server.BrokerServer)
[2025-01-07 17:32:45,374] INFO [BrokerServer id=1] Finished waiting for all of the authorizer futures to be completed (kafka.server.BrokerServer)
[2025-01-07 17:32:45,375] INFO [BrokerServer id=1] Waiting for all of the SocketServer Acceptors to be started (kafka.server.BrokerServer)
[2025-01-07 17:32:45,375] INFO [BrokerServer id=1] Finished waiting for all of the SocketServer Acceptors to be started (kafka.server.BrokerServer)
[2025-01-07 17:32:45,375] INFO [BrokerServer id=1] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2025-01-07 17:32:45,376] INFO Kafka version: 3.8.0 (org.apache.kafka.common.utils.AppInfoParser)
[2025-01-07 17:32:45,377] INFO Kafka commitId: 771b9576b00ecf5b (org.apache.kafka.common.utils.AppInfoParser)
[2025-01-07 17:32:45,377] INFO Kafka startTimeMs: 1736251365376 (org.apache.kafka.common.utils.AppInfoParser)
[2025-01-07 17:32:45,379] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)

Once the Kafka server has started you can now create events, this events will be produced by producer and consumed by consumer.

Step-3 Create Topic for events#

In this step we will create topics for our events, to create topic run the below given command in new cmd from the same dir → C:\kafka\bin\windows.

C:\\kafka\\bin\\windows>kafka-topics.bat --create --topic first-kraft-topic --bootstrap-server localhost:9092
Created topic first-kraft-topic.

Our topic with name first-kraft-topic is now created.

Step-4 Write some events to topic that we have created as producer#

Now to start producing messages/events to first-kraft-topic run the following command.

kafka-console-producer.bat --topic first-kraft-topic --bootstrap-server localhost:9092

After running the command you can start writing the messages. I have wrote the messages “Hi from coding shuttle”.

C:\\kafka\\bin\\windows>kafka-console-producer.bat --topic first-kraft-topic --bootstrap-server localhost:9092
>Hi from coding shuttle

Step-5 Consume the messages as consumer wrote by producer to topic#

Now to consume the message “Hi from coding shuttle” produced by producer in topic “first-kraft-topic” we have to run consumer.bat file in new cmd from dir → C:\kafka\bin\windows.

Run the following command to run consumer, this will start consuming messages on topic “first-kraft-topic”.

kafka-console-consumer.bat --topic first-kraft-topic --from-beginning --bootstrap-server localhost:9092

After running immediately it will consume the message and print.

C:\\kafka\\bin\\windows>kafka-console-consumer.bat --topic first-kraft-topic --from-beginning --bootstrap-server localhost:9092
Hi from coding shuttle

Now just write second message from producer as “Hello this is second message”, consumer will consume this message immediately.

Step-6 Installing Kafbat for visualization#

We will install Kafbat UI and run it, Kafbat UI is a versatile, fast, and lightweight web UI for managing Apache Kafka clusters.

To install Kafbat visit :- https://ui.docs.kafbat.io/development/building/without-docker and download the kafbat-ui-v1.0.0.jar currently this is the latest version.

After downloading move the jar file from downloads folder to Kafka folder.

Note:- To run this jar file make java 17+ version is installed on local machine

Before running the kafbat jar file create application-local.yml file in the Kafka folder, with following configuration. In this configuration we have given port on which Kafbat UI will run i.e 8082 and also mentioned the port on which kafka server is running i.e 9092.

server:
  port: 8082
logging:
  level:
    root: INFO
    io.kafbat.ui: DEBUG
spring:
  jmx:
    enabled: true

kafka:
  clusters:
    - name: local
      bootstrapServers: localhost:9092

Now run the follwing command from Kafka dir → C:\kafka in powershell

java "-Dspring.config.additional-location=C:\\kafka\\application-local.yml" --add-opens java.rmi/javax.rmi.ssl=ALL-UNNAMED -jar kafbat-ui-v1.0.0.jar

You will get the following output:-

PS C:\\kafka> java "-Dspring.config.additional-location=C:\\kafka\\application-local.yml" --add-opens java.rmi/javax.rmi.ssl=ALL-UNNAMED -jar kafbat-ui-v1.0.0.jar
Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts
virtual THREAD FACTORY
 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \\| '_ / _` / _| ' \\/ -_) | ' </ _` |  _| / / _`|
 \\___/|___| |_| \\___|_|   /_/ \\_| .__\\__,_\\__|_||_\\___| |_|\\_\\__,_|_| |_\\_\\__,|
                                 |_|

2025-01-07 18:48:34,605 INFO  [main] i.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 22.0.1 with PID 15492 (C:\\kafka\\kafbat-ui-v1.0.0.jar started by Prem Mane in C:\\kafka)
2025-01-07 18:48:34,610 DEBUG [main] i.k.u.KafkaUiApplication: Running with Spring Boot v3.1.9, Spring v6.0.17
2025-01-07 18:48:34,612 INFO  [main] i.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"
2025-01-07 18:48:40,847 DEBUG [main] i.k.u.s.SerdesInitializer: Configuring serdes for cluster local
2025-01-07 18:48:42,186 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration:

Using generated security password: 1aab9178-16f3-4af2-ad8d-ff16fdca8463

2025-01-07 18:48:42,837 WARN  [main] i.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
2025-01-07 18:48:43,810 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 3 endpoint(s) beneath base path '/actuator'
2025-01-07 18:48:46,008 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8082
2025-01-07 18:48:46,048 INFO  [main] i.k.u.KafkaUiApplication: Started KafkaUiApplication in 13.166 seconds (process running for 15.492)
2025-01-07 18:48:46,632 DEBUG [parallel-2] i.k.u.s.ClustersStatisticsScheduler: Start getting metrics for kafkaCluster: local
2025-01-07 18:48:46,688 INFO  [parallel-2] o.a.k.c.a.AdminClientConfig: AdminClientConfig values:

Now you can access the Kafbat UI from :- http://localhost:8082/

Go to topics and select “first-kraft-topic” and then select messages, you will see all the messages that we produced from producer console on topic. You can explore more details related to running kafka cluster on Kafbat UI.

Conclusion#

By following these steps, you've successfully installed Kafka with KRaft mode, created a topic, produced and consumed messages, and lastly, set up Kafbat UI for visualization. Kafbat provides a user-friendly interface to monitor your Kafka cluster, view topics, and analyze messages.