kafka-python

AFAIK, The concept of partitions and (consumer) groups in kafka was introduced to implement parallelism. I am working with kafka through python. I have a certain topic, which has (say) 2 partitions. This means, if I start a consumer group with 2 consumers in it, they will be mapped(subscribed) to different partitions. But, using kafka library in python, I came across a weird issue. I started 2 consumers with essentially the same group-ids, and started the threads for them to consume messages. But, every message in the kafka-stream is being consumed by both of them !! This seems ridiculous to

Using kafka-python-1.0.2. If I have a topic with 10 partitions, how do I go about committing a particular partition, while looping through the various partitions and messages. I just cant seem find an example of this anywhere, in the docs or otherwise From the docs, I want to use: consumer.commit(offset=offsets) Specifically, how do I create the partition and OffsetAndMetadata dictionary required for offsets (dict, optional) – {TopicPartition: OffsetAndMetadata}. I was hoping the function call would just be something like: consumer.commit(partition, offset) but this does not seem to be the

I have been using the python-kaka module to consume from a kafka broker. I want to consume from the same topic with 'x' number of partitions in parallel. The documentation has this : # Use multiple consumers in parallel w/ 0.9 kafka brokers # typically you would run each on a different server / process / CPU consumer1 = KafkaConsumer('my-topic', group_id='my-group', bootstrap_servers='my.server.com') consumer2 = KafkaConsumer('my-topic', group_id='my-group', bootstrap_servers='my.server.com') Does this mean I can create a separate consumer for each process that I spawn? Also, will there be an

I am trying to pass data from kafka to spark streaming. This is what I've done till now: Installed both kafka and spark Started zookeeper with default properties config Started kafka server with default properties config Started kafka producer Started kafka consumer Sent message from producer to consumer. Works fine. Wrote kafka-spark.py to receive messages from kafka to spark. I try running ./bin/spark-submit examples/src/main/python/kafka-spark.py I get an error. kafka-spark.py - from __future__ import print_function import sys from pyspark.streaming import StreamingContext from pyspark

I'm trying to write a simple pyspark job, which would receive data from a kafka broker topic, did some transformation on that data, and put the transformed data on a different kafka broker topic. I have the following code, which reads data from a kafka topic, but has no effect running sendkafka function: from pyspark import SparkConf, SparkContext from operator import add import sys from pyspark.streaming import StreamingContext from pyspark.streaming.kafka import KafkaUtils import json from kafka import SimpleProducer, KafkaClient def sendkafka(messages): kafka = KafkaClient("localhost:9092")

I am writing a Kafka producer using Python 3.6,the Python-kafka client version is 1.4.4。The Kafka version is: 2.1.0 & 1.1.1(two version are tried),but when I write a message to producer,throw this error: KafkaTimeoutError('Failed to update metadata after 60.0 secs.') This is my client code: producer = KafkaProducer( bootstrap_servers=['mq-server:9092'], api_version = (0,10,2,0) # solve no broker error ) producer.send("dolphin-test".encode('utf-8'),b"test") This is server config I am modified: listeners=PLAINTEXT://10.142.0.2:9092 advertised.listeners=PLAINTEXT://10.142.0.2:9092 When using