kafka-python

问题 Am sending the CSV data to Kafka topic using Kafka-Python . Data is sent and received by Consumer successfully. Now am trying to stream a csv file continuously, any new entry added to the file should be automatically sent to Kafka topic. Any suggestion would be helpful on continuous streaming of CSV file Below is my existing code, from kafka import KafkaProducer import logging from json import dumps, loads import csv logging.basicConfig(level=logging.INFO) producer = KafkaProducer(bootstrap

问题 I'm trying to make a simple chat app using tornado based on this But also I want to use kafka to store the messages. How can I do that? Now, I used this to make a consumer and somehow it's working but it's only printing on the console and I need the messages to show on the webpage, like the tornade app, only it's saved in kafka. Here's my app.py code as of now #!/usr/bin/env python # # Copyright 2009 Facebook # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not

问题 I'm trying to make a simple chat app using tornado based on this But also I want to use kafka to store the messages. How can I do that? Now, I used this to make a consumer and somehow it's working but it's only printing on the console and I need the messages to show on the webpage, like the tornade app, only it's saved in kafka. Here's my app.py code as of now #!/usr/bin/env python # # Copyright 2009 Facebook # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not

问题 Is there any way we can programmatically find lag in the Kafka Consumer. I don't want external Kafka Manager tools to install and check on dashboard. We can list all the consumer group and check for lag for each group. Currently we do have command to check the lag and it requires the relative path where the Kafka resides. Spring-Kafka, kafka-python, Kafka Admin client or using JMX - is there any way we can code and find out the lag. We were careless and didn't monitor the process, the

来源： https://stackoverflow.com/questions/51959495/how-to-create-kafka-python-producer-with-ssl-configuration

来源： https://stackoverflow.com/questions/46402672/python-kafkaconsumer-start-consuming-messages-from-a-timestamp

问题 I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset. Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs otherwise $ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic {topic_name} --from-beginning If I run it through python, it hangs, which I suspect to be caused by incorrect consumer configs consumer = KafkaConsumer(topic_name, bootstrap

问题 I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset. Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs otherwise $ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic {topic_name} --from-beginning If I run it through python, it hangs, which I suspect to be caused by incorrect consumer configs consumer = KafkaConsumer(topic_name, bootstrap

问题 my sink properties : { "name": "jdbc-oracle", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "orders", "connection.url": "jdbc:oracle:thin:@10.1.2.3:1071/orac", "connection.user": "ersin", "connection.password": "ersin!", "auto.create": "true", "delete.enabled": "true", "pk.mode": "record_key", "pk.fields": "id", "insert.mode": "upsert", "plugin.path": "/home/ersin/confluent-5.4.1/share/java/", "name": "jdbc-oracle" }, "tasks": [ {

问题 my sink properties : { "name": "jdbc-oracle", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "orders", "connection.url": "jdbc:oracle:thin:@10.1.2.3:1071/orac", "connection.user": "ersin", "connection.password": "ersin!", "auto.create": "true", "delete.enabled": "true", "pk.mode": "record_key", "pk.fields": "id", "insert.mode": "upsert", "plugin.path": "/home/ersin/confluent-5.4.1/share/java/", "name": "jdbc-oracle" }, "tasks": [ {