apache-beam

问题 I have some sample code that batches events by a fixed window. I would like to combine these events that are in the fixed window - into a single string of messages. Then publish the combined packed message. Here is the code: final PCollection<String> filteredEvents = ind.apply(FilterEvents.STEP_NAME, ParDo.of(new FilterEvents())); final PCollection<String> windowEvents = filteredEvents.apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(1)))); final PCollection<String>

问题 Ok I have to be missing something here. What do i need to stage a pipeline as a template? When I try to stage my template with via these instructions, it runs the module but doesn't stage anything., it appears to function as expected without errors, but I don't see any files actually get added to the bucket location listen in my --template_location. Should my python code be showing up there? I assume so right? I have made sure i have all the beam and google cloud SDKs installed, but maybe I'm

问题 I've created a dataflow template with some parameters. When I write the data to BigQuery, I would like to make use of these parameters to determine which table it is supposed to write to. I've tried calling WriteToBigQuery in a ParDo as suggested in the following link. How can I write to Big Query using a runtime value provider in Apache Beam? The pipeline ran successfully but it is not creating or loading data to BigQuery. Any idea what might be the issue? def run(): pipeline_options =

问题 UI have a python streaming pipeline on GCP Dataflow that reads thousands of messages from a PubSub, like this: with beam.Pipeline(options=pipeline_options) as p: lines = p | "read" >> ReadFromPubSub(topic=str(job_options.inputTopic)) lines = lines | "decode" >> beam.Map(decode_message) lines = lines | "Parse" >> beam.Map(parse_json) lines = lines | beam.WindowInto(beam.window.FixedWindows(1*60)) lines = lines | "Add device id key" >> beam.Map(lambda elem: (elem.get('id'), elem)) lines = lines

问题 Unable to set triggering_frequency for Dataflow Streaming job. transformed | 'Write' >> beam.io.WriteToBigQuery( known_args.target_table, schema=schema, create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED, write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, method=bigquery.WriteToBigQuery.Method.FILE_LOADS, triggering_frequency=5 ) Error: triggering_frequency can only be used with FILE_LOADS method of writing to BigQuery 回答1: This is a bug. The WriteToBigQuery transform

问题 I am developing an ETL pipeline for Google Cloud Dataflow where I have several branching ParDo transforms which each require a local audio file. The branched results are then combined and exported as text. This was initially a Python script that ran on a single machine that I am attempting to adapt for VM worker parallelisation using GC Dataflow. The extraction process downloads the files from a single GCS bucket location then deletes them after the transform is completed to keep storage

问题 I am giving apache beam (with python sdk) a try here so I created a simple pipeline and I tried to deploy it on a Spark cluster. from apache_beam.options.pipeline_options import PipelineOptions import apache_beam as beam op = PipelineOptions([ "--runner=DirectRunner" ] ) with beam.Pipeline(options=op) as p: p | beam.Create([1, 2, 3]) | beam.Map(lambda x: x+1) | beam.Map(print) This pipeline is working well with DirectRunner. So to deploy the same code on Spark (as the portability is a key

问题 Background We have a pipeline that starts by receiving messages from PubSub, each with the name of a file. These files are exploded to line level, parsed to JSON object nodes and then sent to an external decoding service (which decodes some encoded data). Object nodes are eventually converted to Table Rows and written to Big Query. It appeared that Dataflow was not acknowledging the PubSub messages until they arrived at the decoding service. The decoding service is slow, resulting in a

问题 I have a working streaming pipeline in apache beam [python] that ingests data from pub/sub, performs enrichment in dataflow and passes it to big-query. Withing the streaming window, I would like to ensure that messages are not getting duplicated, (as pub/sub guarantees only at least once delivery). So, I figured I'd just use the distinct method from beam, but as soon as I use it my pipeline breaks (can't proceed any further, any local prints are also not visible). Here is my pipeline code:

问题 I have a streaming pipeline developed in Apache Beam (using Spark Runner) which reads from kinesis stream. I am looking out for options in Apache Beam to manage kinesis checkpointing (i.e. stores periodically the current position of kinesis stream) so as it allows the system to recover from failures and continue processing where the stream left off. Is there a provision available for Apache Beam to support kinesis checkpointing as similar to Spark Streaming (Reference link - https://spark