From Postgres table to KSQL table with updates tracking

99封情书 提交于 2020-05-17 06:55:34

问题


My task is transfer data from Postgres table to KSQL table (for future joins with streams). Let's imagine table has three records:

id | name   | description
-------------------------
1  | name1  | description1
2  | name2  | description2
3  | name3  | description3

It is easy to do by means of Kafka JdbcSourceConnector. But there is one little problem - data in table may be changed. Changes must be in KTable too.

According to documentation there is no way to track changes except bulk mode. But bulk mode takes absolutely all rows and inserts them into topic.

I thought to set up bulk mode for connector. Create a KSream for that topic. Create a KTable for that stream...

And here I do not know what to do. How to make sure changes in Postgres table were in KTable too?


回答1:


Bulk mode would work, you just define the key of the stream, then new bulk writes will update the KTable of the same key. In other words, you need to ensure the primary keys don't change in your database


Alternatively, Debezium is the CDC version of Kafka Connect.

JDBC source doesn't capture UPDATE queries, as you've stated.

Debezium will produce records that contain previous and new versions of the modified rows



来源:https://stackoverflow.com/questions/60478580/from-postgres-table-to-ksql-table-with-updates-tracking

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!