emr

问题 First and foremost I'm a Nutch/Hadoop newbie. I have installed Cassandra. I have installed Nutch on the Master node of my EMR cluster. When I attempt to execute a crawl using the following command: sudo bin/crawl crawl urls -dir crawl -depth 3 -topN 5 I get Error: JAVA_HOME is not set. If I run the command without 'sudo' I get: Injector: starting at 2014-07-16 02:12:24 Injector: crawlDb: urls/crawldb Injector: urlDir: crawl Injector: Converting injected urls to crawl db entries. Injector: org

问题 First and foremost I'm a Nutch/Hadoop newbie. I have installed Cassandra. I have installed Nutch on the Master node of my EMR cluster. When I attempt to execute a crawl using the following command: sudo bin/crawl crawl urls -dir crawl -depth 3 -topN 5 I get Error: JAVA_HOME is not set. If I run the command without 'sudo' I get: Injector: starting at 2014-07-16 02:12:24 Injector: crawlDb: urls/crawldb Injector: urlDir: crawl Injector: Converting injected urls to crawl db entries. Injector: org

问题 I need to be able to use Java 8 in EMR I have found this post https://crazydoc1.wordpress.com/2015/08/23/java-8-on-amazon-emr-ami-4-0-0/ Which provides a bootstrap shell script https://gist.github.com/pstorch/c217d8324c4133a003c4 Which installs java 8. When looking at documentation on how to use bootstrap scripts its not apparent at all how to use a shell script with bootstrap actions since in documentation it asks for a Jar location (https://docs.aws.amazon.com/ElasticMapReduce/latest

问题 I am trying to deploy a production cluster for my Flink program. I am using a standard hadoop-core EMR cluster with Flink 1.3.2 installed, using YARN to run it. I am trying to configure my RocksDB to write my checkpoints to an S3 bucket. I am trying to go through these docs: https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#set-s3-filesystem. The problem seems to be getting the dependencies working correctly. I receive this error when trying run the program: java.lang

问题 I'm using Amazon EMR. I have some log data in s3, all in the same bucket, but under different subdirectories like: "s3://bucketname/2014/08/01/abc/file1.bz" "s3://bucketname/2014/08/01/abc/file2.bz" "s3://bucketname/2014/08/01/xyz/file1.bz" "s3://bucketname/2014/08/01/xyz/file3.bz" I'm using : Set hive.mapred.supports.subdirectories=true; Set mapred.input.dir.recursive=true; When trying to load all data from "s3://bucketname/2014/08/": CREATE EXTERNAL TABLE table1(id string, at string, custom

问题 I'm using Amazon EMR. I have some log data in s3, all in the same bucket, but under different subdirectories like: "s3://bucketname/2014/08/01/abc/file1.bz" "s3://bucketname/2014/08/01/abc/file2.bz" "s3://bucketname/2014/08/01/xyz/file1.bz" "s3://bucketname/2014/08/01/xyz/file3.bz" I'm using : Set hive.mapred.supports.subdirectories=true; Set mapred.input.dir.recursive=true; When trying to load all data from "s3://bucketname/2014/08/": CREATE EXTERNAL TABLE table1(id string, at string, custom

问题 I have some pyspark program running on AWS cluster. I am monitoring the job through Spark UI (see attached). However, I noticed that unlike the scala or Java spark program, which shows each Stage is corresponding to which line of code, I can't find which Stage is corresponding to which line of code in the pyspark code. Is there a way I can figure out which Stage is corresponding to which line of the pyspark code? Thanks! 来源： https://stackoverflow.com/questions/38315344/sparkui-for-pyspark

问题 I want to set up a series of spark steps on an EMR spark cluster, and terminate the current step if it's taking too long. However, when I ssh into the master node and run hadoop jobs -list, the master node seems to believe that there is no jobs running. I don't want to terminate the cluster, because doing so would force me to buy a whole new hour of whatever cluster I'm running. Can anyone please help me terminate a spark-step in EMR without terminating the entire cluster? 回答1: That's easy:

问题 I've created a Hive Table through an Elastic MapReduce interactive session and populated it from a CSV file like this: CREATE TABLE csvimport(id BIGINT, time STRING, log STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'; LOAD DATA LOCAL INPATH '/home/hadoop/file.csv' OVERWRITE INTO TABLE csvimport; I now want to store the Hive table in a S3 bucket so the table is preserved once I terminate the MapReduce instance. Does anyone know how to do this? 回答1: Yes you have to export and import

问题 I'm running a spark job on EMR but need to create a checkpoint. I tried using s3 but got this error message 17/02/24 14:34:35 ERROR ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: Wrong FS: s3://spark- jobs/checkpoint/31d57e4f-dbd8-4a50-ba60-0ab1d5b7b14d/connected- components-e3210fd6/2, expected: hdfs://ip-172-18-13-18.ec2.internal:8020 java.lang.IllegalArgumentException: Wrong FS: s3://spark- jobs/checkpoint/31d57e4f-dbd8-4a50-ba60-0ab1d5b7b14d/connected-