flink-batch

问题 We are using Elasticsearch 6.8.4 and Flink 1.0.18. We have an index with 1 shard and 1 replica in elasticsearch and I want to create the custom input format to read and write data in elasticsearch using apache Flink dataset API with more than 1 input splits in order to achieve better performance. so is there any way I can achieve this requirement? Note: Per document size is larger(almost 8mb) and I can read only 10 documents at a time because of size constraint and per reading request, we

来源： https://stackoverflow.com/questions/62680606/how-to-execute-sql-queries-at-run-time-from-datastreams-map-or-flatmap-in-flink

来源： https://stackoverflow.com/questions/62680606/how-to-execute-sql-queries-at-run-time-from-datastreams-map-or-flatmap-in-flink

问题 I am reading data from Kafka and trying to write it to the HDFS file system in ORC format. I have used the below link reference from their official website. But I can see that Flink write exact same content for all data and make so many files and all files are ok 103KB https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/streamfile_sink.html#orc-format Please find my code below. object BeaconBatchIngest extends StreamingBase { val env: StreamExecutionEnvironment =

问题 Does Flink supports Side Outputs feature in Dataset(Batch Api) ? If not, how to handle valid and invalid records when loading from file ? 回答1: You can always do something like this: DataSet<EventOrInvalidRecord> goodAndBadTogether = input.map(new CreateObjectIfPossible()) goodAndBadTogether.filter(new KeepOnlyGood())... goodAndBadTogether.filter(new KeepOnlyBad())... Another reasonable option in some cases is to go ahead and use the DataStream API, even if you don't have streaming sources. 来源