amazon-data-pipeline

问题 My goal is to take daily snapshots of an RDS table and put it in a DynamoDB table. The table should only contain data from a single day. For this have a Data Pipeline set up to query a RDS table and publish the results into S3 in CSV format. Then a HiveActivity imports this CSV into a DynamoDB table by creating external tables for the file and an existing DynamoDB table. This works great, but older entries from the previous day still exist in the DynamoDB table. I want to do this within Data

问题 Documentation on AWS Data Pipeline describes the Data Pipeline related concepts of Components, Instances and Attempts here. http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-how-tasks-scheduled.html I am trying to identify the status of a data pipeline ( whether its running or finished) using DescribeObjects API method described here. https://docs.aws.amazon.com/datapipeline/latest/APIReference/API_DescribeObjects.html Using this API method I can get the status of a particular

问题 I'm using the Load S3 data into RDS MySql table template in AWS Data Pipeline to import csv's from a S3 bucket into our RDS MySql. However I (as IAM user with full-admin rights) run into a warning I can't solve: Object:Ec2Instance - WARNING: Could not validate S3 Access for role. Please ensure role ('DataPipelineDefaultRole') has s3:Get*, s3:List*, s3:Put* and sts:AssumeRole permissions for DataPipeline. Google told me not to use the default policies for the DataPipelineDefaultRole and

问题 I'm trying to insert several csv located in the S3 directory with the AWS DATA Pipeline But, I'm taking this error. at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) Caused by: com.google.gson.stream.MalformedJsonException: Expected ':' at line 1 column 10 at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1505) at com.google

问题 I'm trying to insert several csv located in the S3 directory with the AWS DATA Pipeline But, I'm taking this error. at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) Caused by: com.google.gson.stream.MalformedJsonException: Expected ':' at line 1 column 10 at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1505) at com.google

问题 I'm trying to insert several csv located in the S3 directory with the AWS DATA Pipeline But, I'm taking this error. at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169) Caused by: com.google.gson.stream.MalformedJsonException: Expected ':' at line 1 column 10 at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1505) at com.google

问题 I would like to ask about a processing task I am trying to complete using a data pipeline in AWS, but I have not been able to get it to work. Basically, I have 2 data nodes representing 2 MySQL databases, where the data is supposed to be extracted from periodically and placed in an S3 bucket. This copy activity is working fine selecting daily every row that has been added, let's say today - 1 day. However, that bucket containing the collected data as CSVs should become the input for an EMR

问题 I would like to ask about a processing task I am trying to complete using a data pipeline in AWS, but I have not been able to get it to work. Basically, I have 2 data nodes representing 2 MySQL databases, where the data is supposed to be extracted from periodically and placed in an S3 bucket. This copy activity is working fine selecting daily every row that has been added, let's say today - 1 day. However, that bucket containing the collected data as CSVs should become the input for an EMR

问题 I need to backup my dynamoDB table data to S3 using amazon Data pipeline. My question is- Can i use a single data pipeline to backup multiple dynamoDB tables to S3, or do I have to make a separate pipeline for each of them?? Also, since my tables have a year_month prefix( ex- 2014_3_tableName) , I was thinking of using datapipeline SDK to change the table name in pipeline definition once the month changes.Will this work? Is there an alternate/better way?? Thanks!! 回答1: If you are setting up

问题 I am considering Google DataFlow as an option for running a pipeline that involves steps like: Downloading images from the web; Processing images. I like that DataFlow manages the lifetime of VMs required to complete the job, so I don't need to start or stop them myself, but all examples I came across use it for data mining kind of tasks. I wonder if it is a viable option for other batch tasks like image processing and crawling. 回答1: This use case is a possible application for Dataflow/Beam.