dataflow

问题 Is there any way to extract first n elements in a beam pcollection? The documentation doesn't seem to indicate any such function. I think such an operation would require first a global element number assignment and then a filter - would be nice to have this functionality. I use Google DataFlow Java SDK 2.2.0 . 回答1: PCollection's are unordered per se, so the notion of "first N elements" does not exist - however: In case you need the top N elements by some criterion, you can use the Top

问题 I'd like to create a different WindowFn in a such way to assign Windows to any of my input elements based on another field instead of based on my input entry's timestamp. I know the pre-defined WindowFn 's from Google DataFlow SDK use the timestamp as a criteria to assign window. More specifically I'd like to create a kind of SlidingWindows but instead of considering timestamp as the Window assignment criteria I'd like to consider another field as that criteria. How could I create my

问题 When you need to read all the data from one or more tables in bigquery in a dataflow job there are two approaches to it I would say. The first one is to use BigQueryIO with from , which reads the table in question, and the second approach is to use fromQuery where you specify a query that reads all the data from the same table. So my question is: Is it any cost or performance benefit for using one over the other? I haven't find anything in the docs about this, but I would really like to know.

问题 This question already has answers here : Angular 2 change detection - How are circular dependecies between components resolved? (3 answers) Closed 3 years ago . Angular 2 supports unidirectional data flow, would appreciate if someone could explain or give some references of a resource that explains unidirectional data flow in detail with diagrams. 回答1: parent to child Angular2 has only uni-directional data-binding from parent to child using this binding syntax // child @Input() childProp; <!-

问题 I am trying to read from a csv from in GCP Storage, converting that into dictionaries and then write to a Bigquery table as follows: p | ReadFromText("gs://bucket/file.csv") | (beam.ParDo(BuildAdsRecordFn())) | WriteToBigQuery('ads_table',dataset='dds',project='doubleclick-2',schema=ads_schema) where: 'doubleclick-2' and 'dds' are existing project and dataset, ads_schema is defined as follows: ads_schema='Advertiser_ID:INTEGER,Campaign_ID:INTEGER,Ad_ID:INTEGER,Ad_Name:STRING,Click_through_URL

问题 As you can see in the image below, I have a table in SQL Server that I am filling via a flat file source. There are two columns in the destination table that I want to update based on the logic listed below: SessionID - all rows from the first CSV import will have a value of 1; the second import will have a value of 2, and so on. TimeCreated - datetime value of when the CSV imports happened. I don't need help with how to write the TSQL code to get this done. Instead, I would like someone to

问题 I try to set up controller service account for Dataflow. In my dataflow options I have: options.setGcpCredential(GoogleCredentials.fromStream(new FileInputStream("key.json")).createScoped(someArrays)); options.setServiceAccount("xxx@yyy.iam.gserviceaccount.com"); But I'm getting: WARNING: Request failed with code 403, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying):

问题 cross-posted to http://social.msdn.microsoft.com/Forums/en-US/tpldataflow/thread/89b3f71d-3777-4fad-9c11-50d8dc81a4a9 I know... I'm not really using TplDataflow to its maximum potential. ATM I'm simply using BufferBlock as a safe queue for message passing, where producer and consumer are running at different rates. I'm seeing some strange behaviour that leaves me stumped as to how to proceed. private BufferBlock<object> messageQueue = new BufferBlock<object>(); public void Send(object message

问题 I have a table column with length more than 255 characters and I need to insert this value into excel. When ever the column value exceeds 255 characters, I am getting " Excel Destination [15]: Truncation may occur due to inserting data from data flow column "Copy of Column 0" with a length of 500 to database column "column1" with a length of 255. " error at excel destination component. Please help me to find a way to insert a column with more than 255 characters into an excel. 回答1: I'd

问题 I understand the basics of dataflow programming and have encountered it a bit in Clojure APIs, talks from Jonas Boner, GPars in Groovy, etc. I know it's prevalent in languages like Io (although I have not studied Io). What I am missing is a compelling reason to care about dataflow as a paradigm when building a concurrent program. Why would I use a dataflow model instead of a mutable state+threads+locks model (common in Java, C++, etc) or an actor model (common in Erlang or Scala) or something