Kafka to Chronicle Queue: Save Money on Cloud Bills

Save Money on Your Cloud Bill: Switch from Kafka to Chronicle Queue

Cloud computing offers unmatched flexibility and scalability. However, the operational costs for cloud services can quickly become overwhelming. This article demonstrates how migrating from Kafka to Chronicle Queue —an open-source, low-latency, and resource-efficient queue implementation—can significantly reduce cloud expenses for Event-Driven Architecture (EDA) Java applications.

What is Event-Driven Architecture (EDA)?

Event-Driven Architecture (EDA) refers to a distributed application design where messages (or Data Transfer Objects—DTOs) are produced, detected, consumed, and acted upon in real-time. In the context of this article, we’ll focus on an EDA application where messages are persisted in queues. This architecture is ideal for latency-sensitive applications where speed and responsiveness are critical.

Setting the Scene: High-Performance Messaging Requirements

Let’s consider an EDA application with five interconnected services. The system needs to handle 1,000 messages per second, with 99.9% of messages needing to have a latency of less than 100 ms. If the system exceeds this latency threshold even once in every 1,000 messages, it would fail to meet the performance requirements.

For benchmarking, the messages contain two key data points: a timestamp (in nanoseconds) and an integer value that increments with each hop between services. The system compares the initial timestamp with the final one to calculate the total latency across all services, which is essential for measuring performance.

The Challenge: How Many Instances Can We Run?

The key question in this article is: How many instances of these EDA chains can we run on the same hardware while maintaining low latency, and how can we reduce operational costs? This challenge can be addressed by choosing the right messaging queue technology. While Kafka is widely used in EDA applications, switching to Chronicle Queue can offer a more resource-efficient alternative.

Default Setup: Kafka vs. Chronicle Queue

For our experiment, we compare two popular queue implementations: Apache Kafka and Chronicle Queue. Both are open-source and support high-performance event streaming. However, Chronicle Queue offers significant advantages when it comes to latency and resource consumption, making it particularly suitable for use cases like financial applications or other latency-sensitive systems.

Kafka Setup

Kafka is often chosen for large-scale distributed systems, capable of handling high-throughput data streams. In this test, we use Kafka’s default setup with six distinct topics, optimized for low latency. For performance, we run the Kafka broker on the same machine as the services, utilizing the local loopback interface.

Chronicle Queue Setup

Chronicle Queue, on the other hand, uses off-heap memory and memory-mapping techniques to mitigate garbage collection overhead, making it ideal for applications requiring deterministic low-latency messaging. In this test, we similarly create six distinct Chronicle Queue instances.

The Code: Simple Yet Effective

Below are the Java code snippets used for both Kafka and Chronicle Queue implementations:

Kafka Code Example:

while (!shutDown.get()) {
    ConsumerRecords<Integer, Long> records = inQ.poll(Duration.ofNanos(INTER_MESSAGE_TIME_NS / 8));
    for (ConsumerRecord<Integer, Long> record : records) {
        long beginTimeNs = record.value();
        int value = record.key();
        outQ.send(new ProducerRecord<>(topic, value + 1, beginTimeNs));
    }
}

Chronicle Queue Code Example:

while (!shutDown.get()) {
    try (final DocumentContext rdc = tailer.readingDocument()) {
        if (rdc.isPresent()) {
            ValueIn valueIn = rdc.wire().getValueIn();
            long beginTime = valueIn.readLong();
            int value = valueIn.readInt();
            try (final DocumentContext wdc = appender.writingDocument()) {
                final ValueOut valueOut = wdc.wire().getValueOut();
                valueOut.writeLong(beginTime);
                valueOut.writeInt(value + 1);
            }
        } else {
            LockSupport.parkNanos(INTER_MESSAGE_TIME_NS / 8);
        }
    }
}

Benchmark Results: Kafka to Chronicle Queue

In our benchmark tests, we compared the performance and CPU utilization of both Kafka and Chronicle Queue.

Kafka Benchmarks:

Instances	Median Latency	99% Latency	99.9% Latency	CPU Utilization
1	0.9 ms	19 ms	30 ms	670%
2	16 ms	72 ms	106 ms	700% (saturated)

As seen in the Kafka benchmark, the latency grows significantly as the number of instances increases. After two instances, the 99.9% latency exceeded 100 ms, failing to meet performance requirements.

Chronicle Queue Benchmarks:

Instances	Median Latency	99% Latency	99.9% Latency	CPU Utilization
1	0.5 ms	0.8 ms	0.9 ms	5.2%
10	0.5 ms	0.9 ms	0.9 ms	79%
100	1.0 ms	5 ms	20 ms	700% (saturated)

Chronicle Queue performs exponentially better, handling up to 500 instances with low latency and minimal CPU utilization.

Conclusion for Switching from Kafka to Chronicle Queue

Switching from Kafka to Chronicle Queue allows you to run up to 400 times more instances on the same hardware, dramatically reducing cloud infrastructure costs. For latency-sensitive EDA applications, this could equate to nearly 99.8% savings in operational expenses.

By opting for Chronicle Queue, organizations can maintain low-latency performance while optimizing resource usage and minimizing cloud costs.

If your organization is looking to implement or optimize its EDA solutions, ZippyOPS offers expert consulting, implementation, and managed services in areas like DevOps, Cloud, Security, and more. Whether you’re deploying microservices, automating operations, or scaling your infrastructure, we can help streamline your processes with cutting-edge technologies. Learn more about our services here, or explore our solutions.

For more information, reach out to us at sales@zippyops.com.