ksql

KSQL: append multiple child records to parent record

由 无人久伴 提交于 2019-12-12 01:16:48
问题 I'm trying to use KSQL (as part of confluent-5.0.0) to create a single record out of a set of parent records and child records, where every parent record has multiple child records (spefically, payment details and the parties involved in the payment). These parent/child records are linked by the parent's id. To illustrate, I'm dealing with records of roughly this structure in the source system: payment: | id | currency | amount | payment_date | |------------------------------------------| |

Does KSQL support Kafka stream Processing Guarantees?

由 别说谁变了你拦得住时间么 提交于 2019-12-11 11:01:48
问题 I wonder if KSQL support https://docs.confluent.io/current/streams/concepts.html#processing-guarantees Exactly_once semantic ? 回答1: KSQL is implemented on Kafka Streams API which means it provides exactly once delivery guarantee , linear scalability, fault tolerance and can run as a library without requiring a separate cluster. It is stated in Confluent's KSQL: Streaming SQL Engine for Apache Kafka. See last sentence of the abstract. 来源： https://stackoverflow.com/questions/57878221/does-ksql

KSQL - drop topic

由 只愿长相守 提交于 2019-12-11 06:59:39
问题 Is there a way to drop the topic from KSQL? According to github it is possible, and I tried DROP TOPIC my-topic DROP TOPIC "my-topic" DROP TOPIC 'my-topic' DROP TOPIC `my-topic` But neither of this commands works. I get message Message ------------------------------------------------------------------------------- io.confluent.ksql.util.KsqlException: No topic with name true was registered. 回答1: KSQL topic is different concept than Kafka topic. KSQL topic is an internal concept for KSQL that

In KSQL, How to select max window record after windowing and group by?

由 倾然丶 夕夏残阳落幕 提交于 2019-12-11 04:47:39
问题 ksql> CREATE TABLE HOPPING_TABLE AS SELECT ID, WINDOWSTART() AS WINSTART, COUNT(*) AS CNT FROM MY_STREAM HOPPING WINDOW (SIZE 30 DAYS, ADVANCED BY 1 DAYS) GROUP BY ID; ksql> SELECT ID, WINSTART, CNT FROM HOPPING_TABLE; id winstart cnt ------------------------------------------- 874163197805291909 1547164800000 23 874163197805291909 1547424000000 11 874163197805291909 1547510400000 26 874163197805291909 1547683200000 12 660071199310134801 1545868800000 6 660071199310134801 1546560000000 7

Kafka及周边深度了解

由 我怕爱的太早我们不能终老 提交于 2019-12-09 11:36:21
本文属于原创，转载注明出处，欢迎关注微信小程序 小白AI博客 微信公众号 小白AI 或者网站 https://xiaobaiai.net 或者我的CSDN http://blog.csdn.net/freeape [TOC] 0 前言 文章有点长，但是写的都挺直白的，慢慢看下来还是比较容易看懂，从Kafka的大体简介到Kafka的周边产品比较，再到Kafka与Zookeeper的关系，进一步理解Kafka的特性，包括Kafka的分区和副本以及消费组的特点及应用场景简介。 1 简介 Apache Kafka 是一个 分布式流处理平台 ，注意是平台： 发布 & 订阅，类似消息系统，并发能力强，通过集群可以实现数据总线作用，轻轻松松实现流式记录数据分布式读写 以高容错的方式存储海量流式数据 可以在流式记录数据产生时就进行处理 从上面的一个Kafka小型应用架构图可以了解Kafka周边及它的实际能扮演的角色，图中Kafka集群连接了六个数据输入输出部分，分别是 Kafka Producer 、 Kafka Connect Source 、 Kafka Streams/KSQL 、 Kafka Consumer 、 Kafka Connect Sink 。而这些数据的输入输出都可以通过Kafka提供的四个核心API组去解决(除 Kafka AdminClient API 外)： Kafka

Kafka及周边深度了解

由 寵の児 提交于 2019-12-08 17:11:36
本文属于原创，转载注明出处，欢迎关注微信小程序 小白AI博客 微信公众号 小白AI 或者网站 https://xiaobaiai.net 或者我的CSDN http://blog.csdn.net/freeape [TOC] 0 前言 文章有点长，但是写的都挺直白的，慢慢看下来还是比较容易看懂，从Kafka的大体简介到Kafka的周边产品比较，再到Kafka与Zookeeper的关系，进一步理解Kafka的特性，包括Kafka的分区和副本以及消费组的特点及应用场景简介。 1 简介 Apache Kafka 是一个 分布式流处理平台 ，注意是平台： 发布 & 订阅，类似消息系统，并发能力强，通过集群可以实现数据总线作用，轻轻松松实现流式记录数据分布式读写 以高容错的方式存储海量流式数据 可以在流式记录数据产生时就进行处理 从上面的一个Kafka小型应用架构图可以了解Kafka周边及它的实际能扮演的角色，图中Kafka集群连接了六个数据输入输出部分，分别是 Kafka Producer 、 Kafka Connect Source 、 Kafka Streams/KSQL 、 Kafka Consumer 、 Kafka Connect Sink 。而这些数据的输入输出都可以通过Kafka提供的四个核心API组去解决(除 Kafka AdminClient API 外)： Kafka

ksql, select on table not show anything

由 醉酒当歌 提交于 2019-12-08 09:37:31
问题 I created a source topic subscriber have input message like this: { "ip_router": "", "ip_lan": "", "isdn": "2046573688", "end_datetime": "", "shop_code": "1000405100", "reg_type_id": "5131615", "contract_id": "", "update_datetime": "20170801171355", "project": "", "telecom_service_id": "2", "local_speed": "", "password": "", "price_plan": "", "vip": "", "local_price_plan": "", "sub_id": "1083168000", "sta_datetime": "20090511152847", "update_number_1": "1", "act_status": "000", "network_class

How to fetch Kafka source connector schema based on connector name

由 ＊爱你＆永不变心＊ 提交于 2019-12-08 06:51:11
问题 I am using Confluent JDBC Kafka connector to publish messages into topic. The source connector will send data to topic along with schema on each poll. I want to retrieve this schema. Is it possible? How? Can anyone suggest me My intention is to create a KSQL stream or table based on schema build by Kafka connector on poll. 回答1: The best way to do this is to use Avro, in which the schema is stored separately and automatically used by Kafka Connect and KSQL. You can use Avro by configuring

KSQL Hopping Window : accessing only oldest subwindow

由 不打扰是莪最后的温柔 提交于 2019-12-08 05:55:21
问题 I am tracking the rolling sum of a particular field by using a query which looks something like this : SELECT id, SUM(quantity) AS quantity from stream \ WINDOW HOPPING (SIZE 1 MINUTE, ADVANCE BY 10 SECONDS) \ GROUP BY id; Now, for every input tick, it seems to return me 6 different aggregated values I guess which are for the following time periods : [start, start+60] seconds [start+10, start+60] seconds [start+20, start+60] seconds [start+30, start+60] seconds [start+40, start+60] seconds

KSQL Hopping Window : accessing only oldest subwindow

由 [亡魂溺海] 提交于 2019-12-07 18:05:27
I am tracking the rolling sum of a particular field by using a query which looks something like this : SELECT id, SUM(quantity) AS quantity from stream \ WINDOW HOPPING (SIZE 1 MINUTE, ADVANCE BY 10 SECONDS) \ GROUP BY id; Now, for every input tick, it seems to return me 6 different aggregated values I guess which are for the following time periods : [start, start+60] seconds [start+10, start+60] seconds [start+20, start+60] seconds [start+30, start+60] seconds [start+40, start+60] seconds [start+50, start+60] seconds What if I am interested is only getting the [start, start+60] seconds result
订阅 ksql