where does confluent s3 sink put the key?

问题

I setup a confluent s3 sink connect, it stores .avro files in s3.
I dump those files, and find out that they are just the message itself, I don't know where can I find the message key, any idea?

The config is like:

{
    "name": "s3-sink-test",
    "config": {
        "connector.class": "io.confluent.connect.s3.S3SinkConnector",
        "tasks.max": "1",
        "topics": "book",
        "s3.region": "eu-central-1",
        "s3.bucket.name": "kafka",
        "s3.part.size": "5242880",
        "storage.class": "io.confluent.connect.s3.storage.S3Storage",
        "format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
        "schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
        "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
        "path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
        "locale": "US",
        "timezone": "UTC",
        "partition.duration.ms": "3600000",
        "timestamp.extractor": "RecordField",
        "timestamp.field": "local_timestamp",
        "flush.size": "2",
        "schema.compatibility": "NONE"
    }
}

回答1:

The Kafka message key is not persevered by any of the storage Kafka Connectors by Confluent out of the box

Try compiling and setting up the Archive Transform, which can be setup using these properties in the Connect Configuration

"transforms" : "Archive",
"transforms.Archive.type" : "com.github.jcustenborder.kafka.connect.archive.Archive"

For more about SMTs in Kafka Connect, see this blog post

来源：https://stackoverflow.com/questions/48598928/where-does-confluent-s3-sink-put-the-key

标签

apache-kafka

apache-kafka-connect

confluent