How to Process a kafka KStream and write to database directly instead of sending it another topic

三世轮回 提交于 2019-12-24 01:55:09

问题


I don't want to write processed KStream to another topic, I directly want to write enriched KStream to database. How should I proceed?


回答1:


You can implement a custom Processor that opens a DB connection and apply it via KStream#process(). Cf. https://docs.confluent.io/current/streams/developer-guide.html#applying-processors-and-transformers-processor-api-integration

Note, you will need to do sync writes into your DB to guard against data loss.

Thus, not writing back to a topic has multiple disadvantages:

  • reduced throughput because of sync writes
  • you cannot use exactly-once semantics
  • coupling your application with the database (if DB goes down, your app goes down, too, as it can't write its results anymore)

Therefore, it's recommended to write the results back into a topic and use Connect API to get the data into your database.



来源:https://stackoverflow.com/questions/46524930/how-to-process-a-kafka-kstream-and-write-to-database-directly-instead-of-sending

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!