问题
Reading:Kafka Connect FileStreamSource ignores appended lines
An Answer from 2018 states :
Kafka Connect does not "watch" or "tail" a file. I don't believe it is documented anywhere that it does do that.
It seems Kafka does now support this as reading:
https://docs.confluent.io/5.5.0/connect/managing/configuring.html#standalone-example
does state the file is watched:
FileSource Connector The FileSource Connector reads data from a file and sends it to Apache Kafka®. Beyond the configurations common to all connectors it takes only an input file and output topic as properties. Here is an example configuration:
name=local-file-source connector.class=FileStreamSource tasks.max=1 file=/tmp/test.txt topic=connect-test This connector will read only one file and send the data within that file to Kafka. It will then watch the file for appended updates only. Any modification of file lines already sent to Kafka will not be reprocessed.
My configuration is same as question posted Kafka Connect FileStreamSource ignores appended lines
connect-file-source.properties contains:
name=my-file-connector
connector.class=FileStreamSource
tasks.max=1
file=/data/users/zamara/suivi_prod/app/data/logs.txt
topic=connect-test
Starting connect standalone with
connect-standalone connect-standalone.properties connect-file-source.properties
Adds all the contents of the file logs.txt to the topic connect-test , adding new lines to logs.txt does not add the lines to the topic. Is there configuration required to enable Kafka to watch the file so that new data added to logs.txt is added to the topic connect-test ?
回答1:
Unless you're just experimenting with FileStreamSource for educational purposes, you're heading down a blind alley here. The connector only exists as a sample connector.
To ingest files into Kafka use Kafka Connect Spooldir, Kafka Connect FilePulse, or look at things like Filebeat from Elastic.
来源：https://stackoverflow.com/questions/65359885/kafka-not-publishing-file-changes-to-topic