Kafka Without Zookeeper: Step By Step

Unlock Kafka's True Potential: Master the Zookeeper-Free Revolution, Step By Step

Discover how to deploy and manage Apache Kafka without Zookeeper, leveraging the powerful KRaft metadata mode. This guide provides a step-by-step walkthrough to simplify your Kafka operations and enhance performance.

Read More

Introduction: The Dawn of Zookeeper-Less Kafka

For years, Apache Kafka, the distributed streaming platform, relied heavily on Apache Zookeeper for its metadata management, leader election, and cluster coordination. While Zookeeper served its purpose, it introduced an additional dependency, increasing operational complexity and potential points of failure. The advent of Kafka Raft (KRaft) metadata mode marks a significant evolution, allowing Kafka to manage its own metadata internally, effectively eliminating the need for Zookeeper. This post will guide you through setting up and running Kafka in KRaft mode, providing a comprehensive, step-by-step approach to embrace this simplified and more robust architecture.

Why Kafka Without Zookeeper (KRaft)?

Migrating to KRaft mode offers several compelling advantages:

• Simplified Architecture: Removing Zookeeper means one less distributed system to manage, monitor, and troubleshoot. This significantly reduces operational overhead.
• Improved Scalability: KRaft allows for a much larger number of partitions and faster metadata operations compared to Zookeeper-based Kafka.
• Faster Controller Failovers: The KRaft controller election process is significantly faster than Zookeeper's, leading to quicker recovery times during outages.
• Unified System: Kafka becomes a self-contained system, managing its own metadata directly using the Raft consensus algorithm.
• Enhanced Security: Reduced attack surface by consolidating components.

Key Components of KRaft Architecture

Understanding the core roles within a KRaft cluster is essential:

• Controller Role: In KRaft mode, Kafka brokers can assume the controller role, responsible for managing the cluster's metadata, including topics, partitions, and leader elections. This role directly replaces Zookeeper's previous function.
• Broker Role: These are the workhorse nodes that handle message production and consumption, storing partition data.
• Combined Role: For simpler deployments (e.g., development, single-node setups), a single Kafka instance can act as both a broker and a controller. This is often referred to as a "combined" node.
• Metadata Log: Instead of Zookeeper storing metadata, KRaft brokers maintain a Raft-based metadata log. This log contains all cluster state changes and is replicated among the controller quorum for fault tolerance.

Step-by-Step Guide: Setting Up Kafka in KRaft Mode

Let's dive into setting up a single-node Kafka cluster using KRaft. This configuration will feature a combined broker and controller role.

Prerequisites

• Java 11 or higher installed.
• Apache Kafka downloaded from the official website (ensure it's a version that supports KRaft, e.g., 2.8.0 or newer).

Step 1: Download Apache Kafka

If you haven't already, download the latest stable release of Apache Kafka. For this guide, we assume you've extracted it to a directory, let's call it `KAFKA_HOME`.

wget https://downloads.apache.org/kafka/[YOUR_KAFKA_VERSION]/kafka_2.13-[YOUR_KAFKA_VERSION].tgz
tar -xzf kafka_2.13-[YOUR_KAFKA_VERSION].tgz
cd kafka_2.13-[YOUR_KAFKA_VERSION]

Step 2: Generate a Cluster ID

Every KRaft cluster requires a unique identifier. This ID is used by all nodes in the cluster to identify themselves as part of the same Kafka deployment. You only need to generate this once per cluster.

bin/kafka-storage.sh random-uuid

This command will output a UUID, e.g., 8d0e7a4f-5b1c-4e8d-8a0f-1a2b3c4d5e6f. Copy this ID, as you'll need it in the next step.

Step 3: Configure Kafka for KRaft

Navigate to the config/kraft directory. You'll find example configuration files. We'll modify server.properties (or server-kraft.properties in some distributions) to enable KRaft mode and configure our combined node. Open your chosen server.properties file and make the following changes:

# Basic node configuration
node.id=1 # Unique ID for this node within the cluster
process.roles=broker,controller # This node acts as both broker and controller
cluster.id=YOUR_GENERATED_CLUSTER_ID # Replace with the ID from Step 2

# Listener configuration
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

# Storage paths
log.dirs=/tmp/kraft-kafka-logs # Directory for message data
metadata.log.dir=/tmp/kraft-kafka-metadata # Directory for KRaft metadata log

# Other common settings (optional, adjust as needed)
num.partitions=1
default.replication.factor=1
min.insync.replicas=1

# Remove or comment out Zookeeper-related configurations if they exist
# zookeeper.connect=localhost:2181
# zookeeper.connection.timeout.ms=18000

Explanation of Key Properties:

• node.id: A unique integer identifier for this Kafka node.
• process.roles: Specifies the roles this Kafka instance will play. For a single-node setup, broker,controller is used. For a multi-node setup, you might have dedicated controller nodes and broker nodes.
• cluster.id: The UUID generated in Step 2. All nodes in the same cluster must share this ID.
• listeners & advertised.listeners: Configure how clients and other brokers connect to this node.
• log.dirs: The directory where Kafka stores message data.
• metadata.log.dir: The dedicated directory for the KRaft metadata log. This should ideally be separate from log.dirs for better performance and manageability.

Step 4: Format the Storage Directories

Before starting Kafka, you need to format the storage directories defined in your server.properties with the cluster ID. This initializes the metadata log.

bin/kafka-storage.sh format -t YOUR_GENERATED_CLUSTER_ID -c config/kraft/server.properties

Replace YOUR_GENERATED_CLUSTER_ID with the actual UUID. This command will prepare the directories for Kafka's data and metadata.

Step 5: Start the Kafka Broker

Now you can start your Zookeeper-less Kafka broker:

bin/kafka-server-start.sh config/kraft/server.properties

You should see output indicating that Kafka is starting up and serving as both a broker and a controller. Look for messages about "KraftController" and "KafkaServer".

Step 6: Verify Operation

Once Kafka is running, let's verify its functionality by creating a topic, producing messages, and consuming them.

Create a Topic

Open a new terminal and run:

bin/kafka-topics.sh --create --topic my-kraft-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

You should see confirmation that the topic was created.

Produce Messages

Open another terminal and use the console producer:

bin/kafka-console-producer.sh --topic my-kraft-topic --bootstrap-server localhost:9092

Type some messages and press Enter. For example:

Hello from KRaft Kafka!
This is a test message.
Kafka without Zookeeper is awesome!

Consume Messages

In a third terminal, start the console consumer to read the messages:

bin/kafka-console-consumer.sh --topic my-kraft-topic --from-beginning --bootstrap-server localhost:9092

You should see the messages you produced appear in this terminal.

Deployment Considerations

While this guide focuses on a single-node combined setup for simplicity, production deployments typically involve a multi-node cluster with a dedicated controller quorum. For example, three controller nodes (configured with process.roles=controller) would manage the metadata, while separate broker nodes (configured with process.roles=broker) handle data.

The controller.quorum.voters property would be used in the server.properties of all controller and broker nodes to list the node.id:host:port of each controller in the quorum. For instance:

controller.quorum.voters=1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093

This ensures high availability for the metadata management layer.

Conclusion

By following this guide, you’ve successfully set up and verified a Zookeeper-free Apache Kafka cluster using the KRaft metadata mode. This fundamental shift simplifies your Kafka architecture, reduces operational overhead, and paves the way for a more scalable and resilient streaming platform. Happy coding!

Show your love, follow us javaoneworld

Java 21

Java 21 Unleashed: Master the Future of Concurrent, Expressive, and Efficient Code Now!

Discover Java 21, the latest LTS release, bringing monumental enhancements like Virtual Threads for unparalleled concurrency and Pattern Matching for cleaner, more robust code. Dive deep into its core features that will revolutionize your development workflow and future-proof your applications!

Java 21, released in September 2023, marks a significant milestone as the latest Long-Term Support (LTS) version. This release isn't just an incremental update; it delivers a suite of powerful features designed to fundamentally improve how developers write, run, and maintain high-performance, scalable applications. From groundbreaking concurrency models to more expressive language constructs, Java 21 is poised to redefine modern Java development.

What Makes Java 21 an LTS Release?

LTS releases are critical for enterprises and large projects due to their extended support and stability. Java 21 will receive updates for several years, providing a reliable and secure foundation for long-term deployments. This commitment to stability, combined with a wealth of new features, makes Java 21 an attractive upgrade target for many organizations.

Key Features and Enhancements in Java 21

1. Virtual Threads (JEP 444) - Final

Virtual Threads (Project Loom) are undoubtedly one of the most anticipated and impactful features in Java 21. They aim to dramatically simplify the development of high-throughput concurrent applications by reducing the overhead associated with traditional platform threads.

• Lightweight: Virtual threads consume significantly fewer resources than platform threads, allowing millions to run concurrently.
• Blocking-Friendly: They enable developers to write blocking code naturally, without the complexities of asynchronous programming (callbacks, futures).
• Scalability: By decoupling the number of application threads from the number of OS threads, applications can scale much more efficiently.

Example: Creating and using Virtual Threads

import java.time.Duration;
import java.util.concurrent.Executors;

public class VirtualThreadsExample {
    public static void main(String[] args) throws InterruptedException {
        long startTime = System.currentTimeMillis();

        try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
            for (int i = 0; i < 100_000; i++) {
                final int taskNum = i;
                executor.submit(() -> {
                    try {
                        Thread.sleep(Duration.ofMillis(10)); // Simulate blocking I/O
                        // System.out.println("Task " + taskNum + " executed by thread: " + Thread.currentThread());
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                    }
                });
            }
        } // Executor automatically shuts down

        long endTime = System.currentTimeMillis();
        System.out.println("Finished 100,000 tasks in " + (endTime - startTime) + " ms");
    }
}
        

2. Record Patterns (JEP 440) - Final

Record Patterns allow you to deconstruct record instances directly in pattern matching constructs like `instanceof` and `switch` expressions. This significantly cleans up code that deals with data processing, especially when working with nested records.

Example: Deconstructing Records

record Point(int x, int y) {}
record ColoredPoint(Point p, String color) {}

public class RecordPatternsExample {
    public static void main(String[] args) {
        Object obj = new ColoredPoint(new Point(10, 20), "Red");

        if (obj instanceof ColoredPoint(Point p, String color)) {
            System.out.println("Colored point at (" + p.x() + ", " + p.y() + ") with color " + color);
        }

        // Nested record patterns
        if (obj instanceof ColoredPoint(Point(int x, int y), String color)) {
            System.out.println("Nested deconstruction: x=" + x + ", y=" + y + ", color=" + color);
        }
    }
}
        

3. Pattern Matching for switch (JEP 441) - Final

This feature, building on previous previews, allows `switch` expressions and statements to use type patterns, guarded patterns, and when clauses, making them much more powerful and expressive than traditional `switch` statements.

• Type Patterns: Match on the type of an expression.
• Guarded Patterns: Add a `when` clause to a type pattern for additional conditions.
• Null-Safety: `switch` expressions now handle `null` more gracefully, reducing boilerplate null checks.

Example: Enhanced switch statement

public class SwitchPatternMatchingExample {
    public static String describeObject(Object obj) {
        return switch (obj) {
            case Integer i -> String.format("An Integer: %d", i);
            case String s -> String.format("A String: %s (length %d)", s, s.length());
            case Double d when d > 0 -> String.format("A positive Double: %.2f", d);
            case null -> "It's null!";
            default -> "An unknown object";
        };
    }

    public static void main(String[] args) {
        System.out.println(describeObject(100));
        System.out.println(describeObject("Hello Java 21"));
        System.out.println(describeObject(3.14));
        System.out.println(describeObject(-5.0));
        System.out.println(describeObject(null));
        System.out.println(describeObject(new Object()));
    }
}
        

4. Sequenced Collections (JEP 431) - Final

Java 21 introduces new interfaces to represent collections with a defined encounter order: `SequencedCollection`, `SequencedSet`, and `SequencedMap`. These interfaces provide unified methods for accessing the first and last elements, and for reversing the collection.

• Unified API: Provides `getFirst()`, `getLast()`, `addFirst()`, `addLast()`, `removeFirst()`, `removeLast()`.
• Reversible Views: All sequenced collections now support a `reversed()` method that returns a reversed view of the collection.

Example: Using SequencedCollection

import java.util.ArrayList;
import java.util.List;
import java.util.SequencedCollection;

public class SequencedCollectionsExample {
    public static void main(String[] args) {
        List<String> myList = new ArrayList<>();
        myList.add("Apple");
        myList.add("Banana");
        myList.add("Cherry");

        // Now we can cast to SequencedCollection and use its methods
        SequencedCollection<String> sequencedList = (SequencedCollection<String>) myList;

        System.out.println("First element: " + sequencedList.getFirst()); // Apple
        System.out.println("Last element: " + sequencedList.getLast());   // Cherry

        sequencedList.addFirst("Dates");
        sequencedList.addLast("Elderberry");
        System.out.println("After additions: " + sequencedList); // [Dates, Apple, Banana, Cherry, Elderberry]

        System.out.println("Reversed view: " + sequencedList.reversed()); // [Elderberry, Cherry, Banana, Apple, Dates]
    }
}
        

5. Unnamed Patterns and Variables (JEP 443) - Preview

Introduces the `_` (underscore) for unnamed patterns and variables. This allows you to indicate that a variable or a component of a record pattern is intentionally unused or irrelevant, improving code readability and clarity.

Example: Unnamed Variables

import java.util.Map;

public class UnnamedVariablesExample {
    public static void main(String[] args) {
        // Unnamed variable in a for-each loop (Java 21 Preview)
        Map<String, String> settings = Map.of("theme", "dark", "language", "en");
        for (var entry : settings.entrySet()) {
            String _ = entry.getKey(); // 'key' is explicitly unused
            String value = entry.getValue();
            System.out.println("Setting value (with unnamed key): " + value);
        }

        // Unnamed pattern in record deconstruction (Java 21 Preview)
        record Event(String name, long timestamp) {}
        Event event = new Event("Login", System.currentTimeMillis());
        if (event instanceof Event(_, long timestamp)) { // Only care about timestamp
            System.out.println("Event occurred at: " + timestamp);
        }
    }
}
        

6. Unnamed Classes and Instance Main Methods (JEP 445) - Preview

Aims to simplify the learning curve for Java beginners by allowing simple programs to be written without explicit class declarations or static main methods. This reduces boilerplate for "Hello World" type programs.

Example: Simplified main method (Java 21 Preview)

// No class declaration needed for a simple script
void main() {
    System.out.println("Hello, Java 21 World!");
}
        

This is extremely useful for small scripts and initial learning phases.

Other Notable JEPs in Java 21:

• JEP 439: Generational ZGC (Final): Enhances the Z Garbage Collector with generational capabilities, leading to improved performance and lower latency for applications using ZGC.
• JEP 436: Foreign Function & Memory API (Third Preview): Continues the evolution of a safer, more efficient way for Java programs to interact with code and data outside the JVM.
• JEP 430: String Templates (Preview): Offers a new way to write string literals that include embedded expressions, similar to template literals in other languages, making string formatting more readable and safe.

Why Upgrade to Java 21?

• Enhanced Performance: Virtual Threads and Generational ZGC provide significant performance and scalability improvements.
• Cleaner Code: Record Patterns, Pattern Matching for switch, and Unnamed Patterns/Variables lead to more concise, readable, and maintainable code.
• Future-Proofing: As an LTS release, Java 21 offers long-term stability and a robust platform for developing next-generation applications.
• Simplified Development: Features like Virtual Threads reduce the complexity of concurrent programming, while Unnamed Classes simplify entry for new developers.

How to Get Java 21

You can download Java 21 from various sources, including:

1. Oracle JDK: Official builds from Oracle's website.
2. Open Adoptium (formerly AdoptOpenJDK): Community-driven, high-quality OpenJDK builds.
3. SDKMAN!: A popular tool for managing multiple SDK versions on Unix-like systems.

Ensure your build tools (Maven, Gradle) and IDEs (IntelliJ IDEA, Eclipse, VS Code) are updated to support Java 21 for the best development experience.

Conclusion

By following this guide, you’ve successfully gained a comprehensive understanding of Java 21's most impactful features and why it's a pivotal release for modern Java development. Happy coding!

Show your love, follow us javaoneworld

Kafka vs RabbitMQ vs Redis Streams — Which One Should Java Developers Choose in 2025?

Unlock Your Data Streaming Future: Kafka vs RabbitMQ vs Redis Streams - 2025 Guide!

Dive into the world of message brokers! Discover whether Kafka's scalability, RabbitMQ's flexibility, or Redis Streams' simplicity best suits your Java development needs in 2025.

Read More

Introduction

In the ever-evolving landscape of software development, choosing the right message broker is crucial for building scalable, reliable, and efficient applications. As we approach 2025, Java developers face a plethora of options, each with its unique strengths and weaknesses. This article delves into three popular choices: Kafka, RabbitMQ, and Redis Streams, providing a comprehensive comparison to help you make an informed decision.

Kafka: The Distributed Streaming Platform

Apache Kafka is a distributed, fault-tolerant streaming platform designed for building real-time data pipelines and streaming applications. It excels in handling high-volume data streams and is often used for use cases like event sourcing, log aggregation, and real-time analytics.

Key Features of Kafka:

• High Throughput: Kafka can handle millions of messages per second.
• Scalability: It's designed to scale horizontally by adding more brokers to the cluster.
• Fault Tolerance: Kafka replicates data across multiple brokers to ensure data durability and availability.
• Persistence: Messages are persisted on disk, allowing for replay and reprocessing.
• Real-time Data Pipelines: Optimized for building real-time streaming applications.

When to Choose Kafka:

• You need to handle high-volume data streams.
• You require fault tolerance and data durability.
• You're building real-time data pipelines or streaming applications.
• You need to replay and reprocess messages.

Java Code Example (Kafka Producer):

 import org.apache.kafka.clients.producer.*;
 import java.util.Properties;

 public class KafkaProducerExample {
  public static void main(String[] args) {
  Properties props = new Properties();
  props.put("bootstrap.servers", "localhost:9092");
  props.put("acks", "all");
  props.put("retries", 0);
  props.put("batch.size", 16384);
  props.put("linger.ms", 1);
  props.put("buffer.memory", 33554432);
  props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
  props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

  Producer<String, String> producer = new KafkaProducer<>(props);
  for(int i = 0; i < 100; i++)
  producer.send(new ProducerRecord<>("my-topic", Integer.toString(i), "message " + Integer.toString(i)));

  producer.close();
  }
 }
  

RabbitMQ: The Versatile Message Broker

RabbitMQ is a versatile message broker that supports multiple messaging protocols. It's known for its flexibility and ease of use, making it a popular choice for a wide range of applications, including task queues, message integration, and microservices communication.

Key Features of RabbitMQ:

• Multiple Messaging Protocols: Supports AMQP, MQTT, STOMP, and more.
• Flexible Routing: Supports various exchange types (direct, fanout, topic, headers) for message routing.
• Message Queues: Provides reliable message queuing and delivery.
• Clustering: Supports clustering for high availability and scalability.
• User-Friendly Management UI: Offers a web-based UI for monitoring and managing the broker.

When to Choose RabbitMQ:

• You need a versatile message broker that supports multiple protocols.
• You require flexible message routing.
• You need reliable message queuing and delivery.
• You need a user-friendly management interface.

Java Code Example (RabbitMQ Producer):

 import com.rabbitmq.client.ConnectionFactory;
 import com.rabbitmq.client.Connection;
 import com.rabbitmq.client.Channel;

 public class RabbitMQProducerExample {

  private final static String QUEUE_NAME = "hello";

  public static void main(String[] argv) throws Exception {
  ConnectionFactory factory = new ConnectionFactory();
  factory.setHost("localhost");
  try (Connection connection = factory.newConnection();
  Channel channel = connection.createChannel()) {
  channel.queueDeclare(QUEUE_NAME, false, false, false, null);
  String message = "Hello World!";
  channel.basicPublish("", QUEUE_NAME, null, message.getBytes("UTF-8"));
  System.out.println(" [x] Sent '" + message + "'");
  }
  }
 }
  

Redis Streams: The Real-Time Data Stream

Redis Streams is a data structure introduced in Redis 5.0 that allows you to build real-time data streams. It combines the simplicity of Redis with the functionality of a message queue, making it suitable for use cases like activity feeds, real-time analytics, and chat applications.

Key Features of Redis Streams:

• Simple and Fast: Built on top of Redis, known for its speed and simplicity.
• Persistence: Messages are persisted in memory and optionally on disk.
• Consumer Groups: Supports consumer groups for parallel processing of messages.
• Blocking Operations: Allows consumers to block and wait for new messages.
• Replay and Reprocessing: Supports replaying and reprocessing messages.

When to Choose Redis Streams:

• You need a simple and fast data stream solution.
• You already use Redis in your application.
• You need consumer groups for parallel processing.
• You require blocking operations for real-time updates.

Java Code Example (Redis Streams Producer):

 import redis.clients.jedis.Jedis;

 import java.util.Map;
 import java.util.HashMap;

 public class RedisStreamsProducerExample {

  public static void main(String[] args) {
  Jedis jedis = new Jedis("localhost");
  String streamKey = "my-stream";

  Map<String, String> message = new HashMap<>();
  message.put("user", "John");
  message.put("message", "Hello, Redis Streams!");

  String entryId = jedis.xadd(streamKey, "*", message);
  System.out.println("Message added with ID: " + entryId);

  jedis.close();
  }
 }
  

Comparison Table

Here's a summary table to help you compare Kafka, RabbitMQ, and Redis Streams:

Feature	Kafka	RabbitMQ	Redis Streams
Throughput	High	Medium	Medium
Scalability	Excellent	Good	Good
Persistence	Yes	Yes	Yes (Optional)
Protocols	Kafka Protocol	AMQP, MQTT, STOMP	Redis Protocol
Complexity	High	Medium	Low
Use Cases	Real-time data pipelines, log aggregation	Task queues, message integration	Activity feeds, real-time analytics

Conclusion

By following this guide, you’ve successfully evaluated Kafka, RabbitMQ and Redis Streams and you are now equiped with making an informed decision for your Java project. Happy coding!

Show your love, follow us javaoneworld