Kafka Producer and Consumer Example

Kafka Producer and Consumer Example - Spring Boot

In the world of microservices, asynchronous communication is crucial for building scalable and resilient applications. Apache Kafka is a popular distributed streaming platform that allows you to publish and subscribe to streams of records in real-time. In this blog post, we will explore how to create a simple Spring Boot application that acts as both a Kafka producer and consumer.

Kafka Producer and Consumer Example - Spring Boot

Prerequisites

Before we dive into the code, ensure you have the following:

Java Development Kit (JDK) 11 or higher
Apache Kafka installed and running
Maven or Gradle for dependency management
Basic knowledge of Spring Boot and Kafka

Project Setup

Step 1: Create a Spring Boot Project

You can create a Spring Boot project using Spring Initializr (https://start.spring.io/) with the following dependencies:

Spring Web
Spring for Apache Kafka

Alternatively, you can create a Maven project with the following pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>kafka-demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>kafka-demo</name>
    <description>Demo project for Spring Boot and Kafka</description>
    <properties>
        <java.version>11</java.version>
        <spring-boot.version>2.5.4</spring-boot.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

Step 2: Configure Kafka Properties

In your application.properties file, add the following Kafka configuration:

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=my-group
spring.kafka.consumer.auto-offset-reset=earliest

Step 3: Create Kafka Producer

Create a new class KafkaProducerService to send messages to a Kafka topic.

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaProducerService {

    private static final String TOPIC = "my_topic";

    @Autowired
    private Kafka Template<String, String> kafkaTemplate;

    public void sendMessage(String message) {
        kafkaTemplate.send(TOPIC, message);
        System.out.println("Message sent: " + message);
    }
}

Step 4: Create Kafka Consumer

Next, create a class KafkaConsumerService to listen for messages from the Kafka topic.

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class KafkaConsumerService {

    @KafkaListener(topics = "my_topic", groupId = "my-group")
    public void listen(String message) {
        System.out.println("Message received: " + message);
    }
}

Step 5: Create a REST Controller

Now, create a REST controller to expose an endpoint for sending messages.

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class MessageController {

    @Autowired
    private KafkaProducerService kafkaProducerService;

    @PostMapping("/send")
    public void sendMessage(@RequestBody String message) {
        kafkaProducerService.sendMessage(message);
    }
}

Step 6: Run the Application

Run your Spring Boot application. Make sure your Kafka server is running. You can use the following command to start your application:

mvn spring-boot:run

Step 7: Testing the Application

You can test the application using tools like Postman or cURL. To send a message, make a POST request to the /send endpoint with a JSON body containing the message.

Example cURL command:

curl -X POST http://localhost:8080/send -H "Content-Type: application/json" -d ""Hello Kafka!""

Text-Based Diagrams

Architecture Overview

+-------------------+       +-------------------+
|   Kafka Producer  | ----> |   Kafka Topic     |
| (Message Sender)  |       |   (my_topic)      |
+-------------------+       +-------------------+
                                 |
                                 |
                                 v
+-------------------+       +-------------------+
|   Kafka Consumer  | <---- |   REST Controller |
| (Message Receiver)|       | (Message Endpoint)|
+-------------------+       +-------------------+

Message Flow

User  sends a message
       |
       v
+-------------------+
|   REST Controller |
+-------------------+
       |
       v
+-------------------+
|   Kafka Producer  |
+-------------------+
       |
       v
+-------------------+
|   Kafka Topic     |
+-------------------+
       |
       v
+-------------------+
|   Kafka Consumer  |
+-------------------+
       |
       v
Message processed

Conclusion

In this blog post, we have created a simple Spring Boot application that demonstrates how to use Kafka for messaging. We set up a producer to send messages to a Kafka topic and a consumer to listen for those messages. This pattern is essential for building scalable microservices that can communicate asynchronously. With this foundation, you can explore more advanced features of Kafka and Spring Boot to enhance your applications.

Kafka Producer and Consumer Example - Spring Boot