apache-beam

问题 I'm trying to run my python dataflow job with flex template. job works fine locally when I run with direct runner (without flex template) however when I try to run it with flex template, job stuck in "Queued" status for a while and then fail with timeout. Here is some of logs I found in GCE console: INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/local/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/dataflow/template

问题 I'm trying to run my python dataflow job with flex template. job works fine locally when I run with direct runner (without flex template) however when I try to run it with flex template, job stuck in "Queued" status for a while and then fail with timeout. Here is some of logs I found in GCE console: INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/local/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/dataflow/template

问题 Has anybody tried reading/writing Parquet file using Apache Beam. Support is added recently in version 2.5.0, hence not much documentation. I am trying to read json input file and would like to write to parquet format. Thanks in advance. 回答1: You will need to use ParquetIO.Sink. It implements FileIO. 回答2: Add the following dependency as ParquetIO in different module. <dependency> <groupId>org.apache.beam</groupId>; <artifactId>beam-sdks-java-io-parquet</artifactId>; <version>2.6.0</version>;

问题 Has anybody tried reading/writing Parquet file using Apache Beam. Support is added recently in version 2.5.0, hence not much documentation. I am trying to read json input file and would like to write to parquet format. Thanks in advance. 回答1: You will need to use ParquetIO.Sink. It implements FileIO. 回答2: Add the following dependency as ParquetIO in different module. <dependency> <groupId>org.apache.beam</groupId>; <artifactId>beam-sdks-java-io-parquet</artifactId>; <version>2.6.0</version>;

问题 We are trying to cover the following scenario in a streaming setting: calculate an aggregate (let’s say a count) of user events since the start of the job The number of user events is unbounded (hence only using local state is not an option) I'll discuss three options we are considering, where the two first options are prone to dataloss and the final one is unclear. We'd like to get more insight into this final one. Alternative approaches are of course welcome too. Thanks! Approach 1: Session

问题 I'm making dataflow pipeline with python. I want to share global variables across pipeline transform and across worker nodes like global variables (across multiple workers). Is there any way to support this? thanx in advance 回答1: Stateful processing may be of use for sharing state between workers of a specific node (would not be able to share between transforms though): https://beam.apache.org/blog/2017/02/13/stateful-processing.html 来源： https://stackoverflow.com/questions/44432556/is-there

问题 I noticed in the java sdk, there is a function that allows you to write the headers of a csv file. https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/io/TextIO.Write.html#withHeader-java.lang.String- Is this features mirrored on the python skd? 回答1: You can now write to a text and specify a header using the text sink. From the documentation: class apache_beam.io.textio.WriteToText(file_path_prefix, file_name_suffix='', append_trailing_newlines=True, num_shards=0,

问题 purpose: I want to load stream data, then add a key and then count them by key. problem: Apache Beam Dataflow pipline gets a memory error when i try to load and group-by-key a big-size data using streaming approach (unbounded data) . Because it seems that data is accumulated in group-by and it does not fire data earlier with triggering of each window. If I decrease the elements size (elements count will not change) it works! because actually group-by step waits for all the data to be grouped

问题 We migrated from Google Dataflow 1.9 to Apache Beam 0.6. We are noticing a change in the behavior to the timestamps after applying the globalwindow. In Google Dataflow 1.9, we would get the correct timestamps in the DoFn after windowing/combine function. Now we get some huge value for the timestamp e.g. 9223371950454775, Did the default behavior for the globalwindow change in Apache Beam version? input.apply(name(id, "Assign To Shard"), ParDo.of(new AssignToTest())) .apply(name(id, "Window"),

问题 We migrated from Google Dataflow 1.9 to Apache Beam 0.6. We are noticing a change in the behavior to the timestamps after applying the globalwindow. In Google Dataflow 1.9, we would get the correct timestamps in the DoFn after windowing/combine function. Now we get some huge value for the timestamp e.g. 9223371950454775, Did the default behavior for the globalwindow change in Apache Beam version? input.apply(name(id, "Assign To Shard"), ParDo.of(new AssignToTest())) .apply(name(id, "Window"),