flink-streaming

问题 I am trying to use event time in my Flink job, and using BoundedOutOfOrdernessTimestampExtractor to extract timestamp and generate watermark. But I have some input Kafka having sparse stream, it can have no data for a long time, which makes the getResult in AggregateFunction not called at all. I can see data going into add function. I have set getEnv().getConfig().setAutoWatermarkInterval(1000L); I tried eventsWithKey .keyBy(entry -> (String) entry.get(key)) .window(TumblingEventTimeWindows

问题 Apologies if this sounds dumb! We are working with flink to make async IO calls. A lot of the times, the IO calls are repeated (same set of parameters) and about 80% of the APIs that we call return the same response for the same parameters. So, we would like to avoid making the calls again. We thought we could use state to store previous responses and use them again. The issue is that though our responses can be used again, the number of such responses is huge and therefore requires a lot of

问题 Let say Kafka message contain flink window size configuration. I want read the message from kafka and create global window in flink. Problem Statement: Can we handle above scenario by using BroadcastStream ? Or Any other approach which will support above case ? 回答1: Flink's window API does not support dynamically changing window sizes. What you'll need to do is to implement your own windowing using a process function. In this case a KeyedBroadcastProcessFunction, where the window

问题 Let say Kafka message contain flink window size configuration. I want read the message from kafka and create global window in flink. Problem Statement: Can we handle above scenario by using BroadcastStream ? Or Any other approach which will support above case ? 回答1: Flink's window API does not support dynamically changing window sizes. What you'll need to do is to implement your own windowing using a process function. In this case a KeyedBroadcastProcessFunction, where the window

问题 I'm trying to debug my flink from intellij using the flink UI. the problem it somethims doesn't launched throwing java.net.BindException: Could not start rest endpoint on any port in port range 8081 my piece of code that should let the flink ui run (from windows) is: String osName = System.getProperty("os.name"); if (osName.toLowerCase().contains("win")) { Configuration conf = new Configuration(); conf.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true); env = StreamExecutionEnvironment

问题 As described in this post I have been unable to successfully run any code from the book "Stream Processing with Apache Flink, including the precompiled jar. It is not my practice to use an IDE but I thought I would try to use IntelliJ as Chapter 3 "Run and Debug Flink Applications in an IDE" describes how to do that specifically for the code for this book. The book describes a project import process that I have not found a way to use. It describes setting options on import, for example select

问题 I have a simple Flink application, which sums up the events with the same id and timestamp within the last minute: DataStream<String> input = env .addSource(consumerProps) .uid("app"); DataStream<Event> events = input.map(record -> mapper.readValue(record, Event.class)); pixels .assignTimestampsAndWatermarks(new TimestampsAndWatermarks()) .keyBy("id") .timeWindow(Time.minutes(1)) .sum("constant") .addSink(simpleNotificationServiceSink); env.execute(jobName); private static class

问题 Flink job submission $ ./bin/flink run -m 10.0.2.4:6123 /streaming/mvn-flinkstreaming-scala/mvn-flinkstreaming-scala-1.0.jar Stream processing!!!!!!!!!!!!!!!!! org.apache.flink.streaming.api.datastream.DataStreamSink@40ef3420 ------------------------------------------------------------ The program finished with the following exception: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed at java

问题 Flink job submission $ ./bin/flink run -m 10.0.2.4:6123 /streaming/mvn-flinkstreaming-scala/mvn-flinkstreaming-scala-1.0.jar Stream processing!!!!!!!!!!!!!!!!! org.apache.flink.streaming.api.datastream.DataStreamSink@40ef3420 ------------------------------------------------------------ The program finished with the following exception: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: No pooled slot available and request to ResourceManager for new slot failed at java

问题 Flink job submission # ./bin/flink run -m 10.0.2.4:6123 /storage/flink-1.10.0/examples/streaming/WordCount.jar --input /storage/flink-1.10.0/test.txt --output /storage/flink-1.10.0/test01.txt ------------------------------------------------------------ The program finished with the following exception: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: