Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Filter by
Sorted by
Tagged with
146votes
3answers
50kviews

Kafka: Consumer API vs Streams API

I recently started learning Kafka and end up with these questions. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in ...
user avatar
  • 2,801
48votes
4answers
18kviews

Akka Stream Kafka vs Kafka Streams

I am currently working with Akka Stream Kafka to interact with kafka and I was wonderings what were the differences with Kafka Streams. I know that the Akka based approach implements the reactive ...
user avatar
  • 1,574
46votes
5answers
31kviews

Handling bad messages using Kafka's Streams API

I have a basic stream processing flow which looks like master topic -> my processing in a mapper/filter -> output topics and I am wondering about the best way to handle "bad messages". This ...
user avatar
  • 2,185
40votes
3answers
22kviews

How to send final kafka-streams aggregation result of a time windowed KTable?

What I'd like to do is this: Consume records from a numbers topic (Long's) Aggregate (count) the values for each 5 sec window Send the FINAL aggregation result to another topic My code looks like ...
user avatar
  • 599
38votes
1answer
19kviews

Kafka Streams API: KStream to KTable

I have a Kafka topic where I send location events (key=user_id, value=user_location). I am able to read and process it as a KStream: KStreamBuilder builder = new KStreamBuilder(); KStream<String, ...
user avatar
  • 44.2k
33votes
8answers
19kviews

UnsatisfiedLinkError: /tmp/snappy-1.1.4-libsnappyjava.so Error loading shared library ld-linux-x86-64.so.2: No such file or directory

I am trying to run a Kafka Streams application in kubernetes. When I launch the pod I get the following exception: Exception in thread "streams-pipe-e19c2d9a-d403-4944-8d26-0ef27ed5c057-StreamThread-...
user avatar
  • 2,490
31votes
1answer
22kviews

Why Apache Kafka Streams uses RocksDB and if how is it possible to change it?

During investigation within new features in Apache Kafka 0.9 and 0.10, we had used KStreams and KTables. There is an interesting fact that Kafka uses RocksDB internally. See Introducing Kafka Streams:...
user avatar
28votes
2answers
20kviews

Kafka Streaming Concurrency?

I have some basic Kafka Streaming code that reads records from one topic, does some processing, and outputs records to another topic. How does Kafka streaming handle concurrency? Is everything run in ...
user avatar
  • 15.2k
26votes
1answer
10kviews

Difference between Apache Samza and Apache Kafka Streams (focus on parallelism and communication)

In Samza and Kafka Streams, data stream processing is performed in a sequence/graph (called "dataflow graph" in Samza and "topology" in Kafka Streams) of processing steps (called "job" in Samza" and "...
user avatar
24votes
1answer
14kviews

Is it possible to access message headers with Kafka Streams?

With the addition of Headers to the records (ProducerRecord & ConsumerRecord) in Kafka 0.11, is it possible to get these headers when processing a topic with Kafka Streams? When calling methods ...
user avatar
23votes
4answers
22kviews

Handling exceptions in Kafka streams

Had gone through multiple posts but most of them are related handling Bad messages not about exception handling while processing them. I want to know to how to handle the messages that is been ...
user avatar
  • 2,233
21votes
1answer
14kviews

How to commit manually with Kafka Stream?

Is there a way to commit manually with Kafka Stream? Usually with using the KafkaConsumer, I do something like below: while (true) { ConsumerRecords<String, String> records = consumer.poll(...
user avatar
  • 18.6k
20votes
1answer
10kviews

Difference between KTable and local store

What the difference between these entities? As i think, KTable - simple kafka topic with compaction deletion policy. Also, if logging is enabled for KTable, then there is also changelog and then, ...
user avatar
18votes
3answers
36kviews

Kafka Streams with Spring Boot

I want to work with Kafka Streams real time processing in my spring boot project. So I need Kafka Streams configuration or I want to use KStreams or KTable, but I could not find example on the ...
user avatar
17votes
2answers
21kviews

Does Kafka python API support stream processing?

I have used Kafka Streams in Java. I could not find similar API in python. Do Apache Kafka support stream processing in python?
user avatar

15 30 50 per page
1
2 3 4 5
231