问题
We are working on a POC to keep our db data in sync with our in house DB2 to external MS-Sql. We are using QREP to replicate/ CDC. For the POC we are only using simple tables for now. Each table are sending the messages to their respective topics in Kafka(we are receiving it). Apart from these messages, we also need to capture Transaction details(DML Records, insert/update/delete).
documentation - https://www.ibm.com/support/knowledgecenter/SSTRGZ_11.4.0/com.ibm.cdcdoc.cdckafka.doc/tasks/kafkatccdev.html
demo - https://www.confluent.io/kafka-summit-sf18/a-solution-for-leveraging-kafka-to-provide-end-to-end-acid-transactions/
As per the documentation and demo mentioned above, we are getting source commit stream information with Bookmark Details when we run AvroConsole command.
Shawn mentioned that we can specify the audit settings like audit.jcfs=ENTTYP,CCID,TIMSTAMP in the properties file, but still have some queries
- can below information be captured for all transactional tables as well? if yes, where/how can we specify the settings?
- Transaction Id
- Before / After images/rows/values
- DML operation (insert/update/delete)
should the transaction details be streamed to separate topics or can they be combined to be sent to one commit stream topic.( as shown in the video starting from 23:30 onwards)
In case we were able to stream these information, should we use string deserializer or kafka deserializer to consume it from java code
Sorry for all these questions, but we are still at very basic stage without much knowledge. Hope some experts can give some guidance.
来源：https://stackoverflow.com/questions/63395751/iidr-cdc-with-transaction-details-to-kafka