aws-glue

问题 Using AWS::Glue::Table, you can set up an Athena table like here. Athena supports partitioning data based on folder structure in S3. I would like to partition my Athena table from my Glue template. From AWS Glue Table TableInput, it appears that I can use PartitionKeys to partition my data, but when I try to use the below template, Athena fails and can't get any data. Resources: ... MyGlueTable: Type: AWS::Glue::Table Properties: DatabaseName: !Ref MyGlueDatabase CatalogId: !Ref AWS:

问题 In the documentation, I cannot find any way of checking the run status of a crawler. The only way I am doing it currently is constantly checking AWS to check if the file/table has been created. Is there a better way to block until crawler finishes its run? 回答1: You can use boto3 (or similar) to do it. There is the get_crawler method. You will find needed information in "LastCrawl" section https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get

问题 In the documentation, I cannot find any way of checking the run status of a crawler. The only way I am doing it currently is constantly checking AWS to check if the file/table has been created. Is there a better way to block until crawler finishes its run? 回答1: You can use boto3 (or similar) to do it. There is the get_crawler method. You will find needed information in "LastCrawl" section https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get

问题 We have some pipe delimited .txt reports coming into a folder in S3, on which we run Glue crawler to determine the schema and query in Athena. The format of the report changed recently so there are two new columns in the middle. Old files: Columns A B C D E F Data a1 b1 c1 d1 e1 f1 New files with extra "G" and "H" columns: Columns A B G H C D E F Data a2 b2 g2 h2 c2 d2 e2 f2 What we get in the table created by the crawler as seen in Athena: Columns A B C D E F G H <- Puts new columns at the

问题 I am reading a dataset dataset1 and dataset2 from S3 locations. I then transform them and write back to the same location where dataset2 was read from. However, I get below error message: An error occurred while calling o118.save. No such file or directory 's3://<myPrefix>/part-00001-a123a120-7d11-581a-b9df-bc53076d57894-c000.snappy.parquet If I try to write to a new S3 location e.g. s3://dataset_new_path.../ then the code works fine. my_df \ .write.mode('overwrite') \ .format('parquet') \

问题 I have a large nested json document for each year (say 2018, 2017), which has aggregated data by each month (Jan-Dec) and each day (1-31). { "2018" : { "Jan": { "1": { "u": 1, "n": 2 } "2": { "u": 4, "n": 7 } }, "Feb": { "1": { "u": 3, "n": 2 }, "4": { "u": 4, "n": 5 } } } } I have used AWS Glue Relationalize.apply function to convert above hierarchal data into flat structure: dfc = Relationalize.apply(frame = datasource0, staging_path = my_temp_bucket, name = my_ref_relationalize_table,

问题 I got a job that needs to perform the following task Relationalize the data Rename the field names that contains the '.'s so that it can be imported into PostgreSQL as normal looking field name. Here is the code import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job ## @params: [JOB_NAME] args = getResolvedOptions(sys.argv, ['JOB_NAME']) sc =

问题 I got a job that needs to perform the following task Relationalize the data Rename the field names that contains the '.'s so that it can be imported into PostgreSQL as normal looking field name. Here is the code import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job ## @params: [JOB_NAME] args = getResolvedOptions(sys.argv, ['JOB_NAME']) sc =

问题 First stack overflow question here. Hope I do this correctly: I need to use an external python library in AWS glue. "Openpyxl" is the name of the library. I follow these directions: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html However, after I have my zip file saved in the correct s3 location and point my glue job to that location, I'm not sure what to actually write in the script. I tried your typical Import openpyxl , but that just returns the

问题 First stack overflow question here. Hope I do this correctly: I need to use an external python library in AWS glue. "Openpyxl" is the name of the library. I follow these directions: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html However, after I have my zip file saved in the correct s3 location and point my glue job to that location, I'm not sure what to actually write in the script. I tried your typical Import openpyxl , but that just returns the