Apache Beam using pCollections to window - batch - pack events to consumer

大憨熊 提交于 2021-01-29 09:26:25

问题


I have some sample code that batches events by a fixed window. I would like to combine these events that are in the fixed window - into a single string of messages. Then publish the combined packed message. Here is the code:

final PCollection<String> filteredEvents = ind.apply(FilterEvents.STEP_NAME,
        ParDo.of(new FilterEvents()));

final PCollection<String> windowEvents = filteredEvents.apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(1))));

final PCollection<String> combinedEvents = windowEvents.apply(Combine.globally(new CombineStrings()).withoutDefaults());

combinedEvents.apply("Write PageEvents (String)",
        PubsubIO.writeStrings().to(options.getOutputTopic()));

the client still gets only 1 event per message. Any Ideas?

来源:https://stackoverflow.com/questions/62379467/apache-beam-using-pcollections-to-window-batch-pack-events-to-consumer

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!