问题
I am doing poc of confluent kafka connect version 5.2.3. We are trying to copy message of topic a file as backup and from this file back to topic when we need it.
Topic has Key =string Value=protbuf
I am using
key.convertor=org.apache.kafka.connect.storgare.StringConvertor
value.convertor=com.blueapron.connect.protobuf.ProtobufConvertor
value.convertor.protoClassName=<proto class name>
Sink config
name=test
connector.class=FileStreamSink
tasks.max=1
file=test.txt
topics=testtopic
Source config
name=test
connector.class=FileStreamSource
tasks.max=1
file=test.txt
topics=testtopic_connect
I am able successfully sink it to a file with file content as below
Struct{<message in name value pair>}
Struct{<message in name value pair>}
....
The same file i use to source it back to a different topic. When i run the source it throw error
String cannot be casted to org.apache.kafka.connect.data.Struct.
Questions are
- Why i do not see any key in the file when my kafka topic has key value pair.
- Why source is not able to copy the content from file to topic and throwing casting related error.
- I get the similar error when i use ByteArrayConvertor provided by kafka. String cannot be casted to bytes. Ideally ByteArrayConvertor should work for any kind of data.
- Does blueapron only works with protobuf3 version?
回答1:
Why i do not see any key in the file when my kafka topic has key value pair
The FileSink doesn't write the key, only the value, by doing a .toString of the Struct or non-Struct of that data. There is an SMT to get the key moved over into the value - https://github.com/jcustenborder/kafka-connect-transform-archive
But the rows of the file would still look like Struct{key= ..., value=...}
Why source is not able to copy the content from file to topic and throwing casting related error.
I get the similar error when i use ByteArrayConvertor provided by kafka. String cannot be casted to bytes. Ideally ByteArrayConvertor should work for any kind of data.
The FileSource only reads line-delimited fields from the file as strings.
Does blueapron only works with protobuf3 version?
It seems like it - https://github.com/blueapron/kafka-connect-protobuf-converter/blob/v2.2.1/pom.xml#L21
Essentially, if you want to fully backup a topic (including timestamps, headers, and possibly even offsets), you'll need something that dumps and reads actual binary data (which is not what ByteArrayConverter does, that only (de)serializes the data from/to Kafka, not the source/sink, and the FileSource connector will try to read the file back always as a string)
回答2:
Does blueapron only works with protobuf3 version?
Yes, since we only use proto3 internally (we needed Python, Java and Ruby code generation - Ruby is only available for proto3), unfortunately we didn't build in support for proto2 definitions when first building this tool. I've updated the project readme to reflect that (https://github.com/blueapron/kafka-connect-protobuf-converter#compatibility).
来源：https://stackoverflow.com/questions/58339015/kafka-connect-string-cannot-be-casted-to-struct