dataflow

问题 Is there a way in Nifi to run a python script which has modules imported from a different folder, requirements specified in a pipfile and has arguments to pass? In short, how to execute a python script which usually runs in my virtual environment using Nifi? The end goal for me is to pick up a file using Get File and post it to API. I tried execute process, execute streamcommand processors. 回答1: To perform follow-on processing on the flowfile using Python, you can use the ExecuteStreamCommand

问题 Is there a way in Nifi to run a python script which has modules imported from a different folder, requirements specified in a pipfile and has arguments to pass? In short, how to execute a python script which usually runs in my virtual environment using Nifi? The end goal for me is to pick up a file using Get File and post it to API. I tried execute process, execute streamcommand processors. 回答1: To perform follow-on processing on the flowfile using Python, you can use the ExecuteStreamCommand

问题 I'm doing a simple pipeline using Apache Beam in python (on GCP Dataflow) to read from PubSub and write on Big Query but can't handle exceptions on pipeline to create alternatives flows. On a simple WriteToBigQuery example: output = json_output | 'Write to BigQuery' >> beam.io.WriteToBigQuery('some-project:dataset.table_name') I tried to put this inside a try/except code, but it doesnt work because when it fails, exceptions seems to be throwed on a Java layer outside my python execution: INFO

问题 I need some suggestions for one of the use cases I am working on. Use Case: We have data in Cloud SQL around 5-10 tables, some are treated as lookup and others transactional. We need to get this to BigQuery in a way to make 3-4 tables(Flattened, Nested or Denormalized) out of these which will be used for reporting in Data Studio, Looker, etc. Data should be processed incrementally and changes in Cloud SQL could happen every 5 min, which means that data should be available to BigQuery

问题 Context: We have a Dataflow job that transforms PubSub messages into Avro GenericRecords and writes them into GCS as ".avro". The transformation between PubSub messages and GenericRecords requires a schema. This schema changes weekly with field additions only. We want to be able to update the fields without updating the Dataflow job. What we did: We took the advice from this post and created a Guava Cache that refreshes the content every minute. The refresh function will pull schema from GCS.

问题 Say in Dataflow/Apache Beam program, I am trying to read table which has data that is exponentially growing. I want to improve the performance of the read. BigQueryIO.Read.from("projectid:dataset.tablename") or BigQueryIO.Read.fromQuery("SELECT A, B FROM [projectid:dataset.tablename]") Will the performance of my read improve, if i am only selecting the required columns in the table, rather than the entire table in above? I am aware that selecting few columns results in the reduced cost. But

问题 I'm setting up a slow-changing lookup Map in my Apache-Beam pipeline. It continuously updates the lookup map. For each key in lookup map, I retrieve the latest value in the global window with accumulating mode. But it always meets Exception : org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.IllegalArgumentException: Duplicate values for mykey Is anything wrong with this snippet code? If I use .discardingFiredPanes() instead, I will lose information in the last emit. pipeline

问题 How can I do dataflow (pipes and filters, stream processing, flow based) in C? And not with UNIX pipes. I recently came across stream.py. Streams are iterables with a pipelining mechanism to enable data-flow programming and easy parallelization. The idea is to take the output of a function that turns an iterable into another iterable and plug that as the input of another such function. While you can already do this using function composition, this package provides an elegant notation for it

问题 How can I do dataflow (pipes and filters, stream processing, flow based) in C? And not with UNIX pipes. I recently came across stream.py. Streams are iterables with a pipelining mechanism to enable data-flow programming and easy parallelization. The idea is to take the output of a function that turns an iterable into another iterable and plug that as the input of another such function. While you can already do this using function composition, this package provides an elegant notation for it

问题 I posed the question generically, because maybe it is a generic answer. But a specific example is comparing 2 BigQuery tables with the same schema, but potentially different data. I want a diff, i.e. what was added, deleted, modified, with respect to a composite key, e.g. the first 2 columns. Table A C1 C2 C3 ----------- a a 1 a b 1 a c 1 Table B C1 C2 C3 # Notes if comparing B to A ------------------------------------- a a 1 # No Change to the key a + a a b 2 # Key a + b Changed from 1 to 2