ksqldb

来源： https://stackoverflow.com/questions/62749698/is-there-a-guarantee-that-all-exploded-rows-in-a-stream-will-get-updated-into-ta

问题 My JSON looks like: { "Obj1": { "a": "abc", "b": "def", "c": "ghi" }, "ArrayObj": [ { "key1": "1", "Key2": "2", "Key3": "3", }, { "key1": "4", "Key2": "5", "Key3": "6", }, { "key1": "7", "Key2": "8", "Key3": "9", } ] } I have written KSQL streams to convert it to AVRO and save to a topic, So that I can push it to JDBC Sink connector CREATE STREAM Example1(ArrayObj ARRAY<STRUCT<key1 VARCHAR, Key2 VARCHAR>>,Obj1 STRUCT<a VARCHAR>)WITH(kafka_topic='sample_topic', value_format='JSON'); CREATE

问题 Is there a way to create a stream from a topic specifying that the entire record should be considered a VARCHAR so that I can create streams from it using extractjsonfield()? Sample records might looks something like: { "Header": { "RecType": "RecA", ... more header records in a fairly consistent format ... }, "RAFld1": { "someFld": "some data", "someOtherField": 1.001, }, "RAFld2": { "aFld": "data", "anotherFld": 98.6, ... }, ... } But the next record might look like: { "Header": { "RecType"

问题 I've been using KSQL and so far it's been working great. But now I'd like to sink the output to BigQuery via Kafka Connect, and need to attach a JSON schema. I'm having trouble figuring out how to do this. Here's my query: CREATE STREAM tweets_original ( CreatedAt BIGINT, Id BIGINT, Text VARCHAR, Source VARCHAR, GeoLocation VARCHAR, User STRUCT<Id BIGINT, Name VARCHAR, Description VARCHAR, ScreenName VARCHAR, URL VARCHAR, FollowersCount BIGINT, FriendsCount BIGINT> ) WITH (kafka_topic='tweets

问题 I am trying to group events by one of its properties and over time using the KSQL Windowed Aggregation, specifically the Session Window. I have a STREAM made from a kafka topic with the TIMESTAMP property well specified. When I try to create a STREAM with a Session Windowing with a query like: CREATE STREAM SESSION_STREAM AS SELECT ... FROM EVENT_STREAM WINDOW SESSION (5 MINUTES) GROUP BY ...; I always get the error: Your SELECT query produces a TABLE. Please use CREATE TABLE AS SELECT

问题 I am using following program to produce records in kafka: import java.io.IOException; import java.security.SecureRandom; public class SensorStatusProducer { private final static String TOPIC = "SENSOR_STATUS_DETAILS"; private final static String PRODUCER_URI = "http://xxx.xxx.xxx.xxx:8082/topics/" + TOPIC; private final static SecureRandom randomNumber = new SecureRandom(); private final static SensorDetails sensorDetails = new SensorDetails(); public static void main(String[] args) { int[]

问题 I have produced following data on topic named SENSOR_STATUS_DETAILS in json : 1001 { "sensorid": 1001, "status": "CONNECTED", "lastconnectedtime": "2020-05-31 22:31:54" } 1002 { "sensorid": 1002, "status": "CONNECTED", "lastconnectedtime": "2020-05-31 22:33:37" } I am trying to make a table from it as: CREATE TABLE STATUS_IB_TABLE (ROWKEY INT KEY, sensorid INTEGER, status VARCHAR, lastconnectedtime STRING) WITH (TIMESTAMP='lastconnectedtime', TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss', KAFKA

问题 I have produced following data on topic named SENSOR_STATUS_DETAILS in json : 1001 { "sensorid": 1001, "status": "CONNECTED", "lastconnectedtime": "2020-05-31 22:31:54" } 1002 { "sensorid": 1002, "status": "CONNECTED", "lastconnectedtime": "2020-05-31 22:33:37" } I am trying to make a table from it as: CREATE TABLE STATUS_IB_TABLE (ROWKEY INT KEY, sensorid INTEGER, status VARCHAR, lastconnectedtime STRING) WITH (TIMESTAMP='lastconnectedtime', TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss', KAFKA

问题 My task is transfer data from Postgres table to KSQL table (for future joins with streams). Let's imagine table has three records: id | name | description ------------------------- 1 | name1 | description1 2 | name2 | description2 3 | name3 | description3 It is easy to do by means of Kafka JdbcSourceConnector. But there is one little problem - data in table may be changed. Changes must be in KTable too. According to documentation there is no way to track changes except bulk mode. But bulk

问题 I am working on POC, I have to read pipe separated value file and insert these records into ms sql server. I am using confluent 5.4.1 to use value_delimiter create stream property. But its giving exception: Delimeter only supported with DELIMITED format 1. Start Confluent (version: 5.4.1):: [Dev root @ myip ~] # confluent local start The local commands are intended for a single-node development environment only, NOT for production usage. https://docs.confluent.io/current/cli/index.html Using