问题
I'm getting this error when trying to run a Pysparkling script on an AWS EMR cluster. I can get everything to work when downloading Sparkling water 2.1.8 and running it from a pysparkling shell. However, spark-submit does not seem to work.
Error:
NameError: name 'H2OContext' is not defined
My spark-submit:
spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-water-examples_2.11:2.1.7 --conf spark.dynamicAllocation.enabled=false spark.py
Python file
from pysparkling import *
hc = H2OContext.getOrCreate(sc)
Also, I've tried actually making a spark context but that just results in the same error, but takes longer.
Bootstrap file:
#!/usr/bin/env bash
# install conda (conda 4.2 defaults to pythone35)
wget --quiet http://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh -O ~/anaconda.sh \
&& /bin/bash ~/anaconda.sh -b -p $HOME/conda
echo -e '\nexport PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc
# install packages
conda install -y ipython jupyter
# needed for PySparkling
conda install requests
conda install six
conda install future
conda install tabulate
# install pysparkling
pip install h2o
# pip install pysparkling
pip install h2o_pysparkling_2.1
More detailed output:
[hadoop@ip-172-31-32-30 test]$ spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-water-examples_2.11:2.1.7 --conf spark.dynamicAllocation.enabled=false spark.py
Ivy Default Cache set to: /home/hadoop/.ivy2/cache
The jars for the packages stored in: /home/hadoop/.ivy2/jars
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
ai.h2o#sparkling-water-core_2.11 added as a dependency
ai.h2o#sparkling-water-examples_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found ai.h2o#sparkling-water-core_2.11;2.1.7 in central
found ai.h2o#h2o-genmodel;3.10.4.7 in central
found net.sf.opencsv#opencsv;2.3 in central
found ai.h2o#deepwater-backend-api;1.0.2 in central
found com.google.guava#guava;19.0 in central
found ai.h2o#h2o-core;3.10.4.7 in central
found joda-time#joda-time;2.3 in central
found gov.nist.math#jama;1.0.3 in central
found org.javassist#javassist;3.18.2-GA in central
found org.apache.commons#commons-math3;3.3 in central
found commons-io#commons-io;2.4 in central
found ai.h2o#google-analytics-java;1.1.2-H2O-CUSTOM in central
found org.apache.httpcomponents#httpclient;4.1 in central
found org.apache.httpcomponents#httpcore;4.1 in central
found commons-logging#commons-logging;1.1.1 in central
found commons-codec#commons-codec;1.4 in central
found org.eclipse.jetty.aggregate#jetty-servlet;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-plus;8.1.17.v20150415 in central
found org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 in central
found org.eclipse.jetty#jetty-webapp;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-xml;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-util;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-servlet;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-security;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-server;8.1.17.v20150415 in central
found org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 in central
found org.eclipse.jetty#jetty-continuation;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-http;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-io;8.1.17.v20150415 in central
found org.eclipse.jetty#jetty-jndi;8.1.17.v20150415 in central
found org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020 in central
found org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 in central
found com.github.rwl#jtransforms;2.4.0 in central
found ai.h2o#h2o-jaas-pam;3.10.4.7 in central
found org.kohsuke#libpam4j;1.8 in central
found net.java.dev.jna#jna;4.0.0 in central
found log4j#log4j;1.2.15 in central
found com.google.code.gson#gson;2.3.1 in central
found commons-lang#commons-lang;2.6 in central
found ai.h2o#reflections;0.9.11-h2o-custom in central
found com.google.code.findbugs#jsr305;3.0.0 in central
found ai.h2o#h2o-algos;3.10.4.7 in central
found ai.h2o#h2o-web;3.10.4.7 in central
found ai.h2o#h2o-avro-parser;3.10.4.7 in central
found ai.h2o#h2o-parquet-parser;3.10.4.7 in central
found ai.h2o#h2o-orc-parser;3.10.4.7 in central
found ai.h2o#h2o-scala_2.11;3.10.4.7 in central
found ai.h2o#h2o-persist-hdfs;3.10.4.7 in central
found net.java.dev.jets3t#jets3t;0.6.1 in central
found commons-httpclient#commons-httpclient;3.1 in central
found ai.h2o#h2o-persist-s3;3.10.4.7 in central
found com.amazonaws#aws-java-sdk-s3;1.10.47 in central
found com.amazonaws#aws-java-sdk-kms;1.10.47 in central
found com.amazonaws#aws-java-sdk-core;1.10.47 in central
found commons-logging#commons-logging;1.1.3 in central
found org.apache.httpcomponents#httpclient;4.3.6 in central
found org.apache.httpcomponents#httpcore;4.3.3 in central
found commons-codec#commons-codec;1.6 in central
found joda-time#joda-time;2.8.1 in central
found ai.h2o#sparkling-water-repl_2.11;2.1.7 in central
found org.joda#joda-convert;1.7 in central
found ai.h2o#sparkling-water-examples_2.11;2.1.7 in central
found ai.h2o#sparkling-water-ml_2.11;2.1.7 in central
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-core_2.11/2.1.7/sparkling-water-core_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-core_2.11;2.1.7!sparkling-water-core_2.11.jar (56ms)
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-examples_2.11/2.1.7/sparkling-water-examples_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-examples_2.11;2.1.7!sparkling-water-examples_2.11.jar (15ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-genmodel/3.10.4.7/h2o-genmodel-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-genmodel;3.10.4.7!h2o-genmodel.jar (7ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-core/3.10.4.7/h2o-core-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-core;3.10.4.7!h2o-core.jar (129ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-algos/3.10.4.7/h2o-algos-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-algos;3.10.4.7!h2o-algos.jar (35ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-web/3.10.4.7/h2o-web-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-web;3.10.4.7!h2o-web.jar (512ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-scala_2.11/3.10.4.7/h2o-scala_2.11-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-scala_2.11;3.10.4.7!h2o-scala_2.11.jar (4ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-persist-hdfs/3.10.4.7/h2o-persist-hdfs-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-persist-hdfs;3.10.4.7!h2o-persist-hdfs.jar (2ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-persist-s3/3.10.4.7/h2o-persist-s3-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-persist-s3;3.10.4.7!h2o-persist-s3.jar (2ms)
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-repl_2.11/2.1.7/sparkling-water-repl_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-repl_2.11;2.1.7!sparkling-water-repl_2.11.jar (4ms)
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-jaas-pam/3.10.4.7/h2o-jaas-pam-3.10.4.7.jar ...
[SUCCESSFUL ] ai.h2o#h2o-jaas-pam;3.10.4.7!h2o-jaas-pam.jar (2ms)
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-ml_2.11/2.1.7/sparkling-water-ml_2.11-2.1.7.jar ...
[SUCCESSFUL ] ai.h2o#sparkling-water-ml_2.11;2.1.7!sparkling-water-ml_2.11.jar (10ms)
:: resolution report :: resolve 6024ms :: artifacts dl 802ms
:: modules in use:
ai.h2o#deepwater-backend-api;1.0.2 from central in [default]
ai.h2o#google-analytics-java;1.1.2-H2O-CUSTOM from central in [default]
ai.h2o#h2o-algos;3.10.4.7 from central in [default]
ai.h2o#h2o-avro-parser;3.10.4.7 from central in [default]
ai.h2o#h2o-core;3.10.4.7 from central in [default]
ai.h2o#h2o-genmodel;3.10.4.7 from central in [default]
ai.h2o#h2o-jaas-pam;3.10.4.7 from central in [default]
ai.h2o#h2o-orc-parser;3.10.4.7 from central in [default]
ai.h2o#h2o-parquet-parser;3.10.4.7 from central in [default]
ai.h2o#h2o-persist-hdfs;3.10.4.7 from central in [default]
ai.h2o#h2o-persist-s3;3.10.4.7 from central in [default]
ai.h2o#h2o-scala_2.11;3.10.4.7 from central in [default]
ai.h2o#h2o-web;3.10.4.7 from central in [default]
ai.h2o#reflections;0.9.11-h2o-custom from central in [default]
ai.h2o#sparkling-water-core_2.11;2.1.7 from central in [default]
ai.h2o#sparkling-water-examples_2.11;2.1.7 from central in [default]
ai.h2o#sparkling-water-ml_2.11;2.1.7 from central in [default]
ai.h2o#sparkling-water-repl_2.11;2.1.7 from central in [default]
com.amazonaws#aws-java-sdk-core;1.10.47 from central in [default]
com.amazonaws#aws-java-sdk-kms;1.10.47 from central in [default]
com.amazonaws#aws-java-sdk-s3;1.10.47 from central in [default]
com.github.rwl#jtransforms;2.4.0 from central in [default]
com.google.code.findbugs#jsr305;3.0.0 from central in [default]
com.google.code.gson#gson;2.3.1 from central in [default]
com.google.guava#guava;19.0 from central in [default]
commons-codec#commons-codec;1.6 from central in [default]
commons-httpclient#commons-httpclient;3.1 from central in [default]
commons-io#commons-io;2.4 from central in [default]
commons-lang#commons-lang;2.6 from central in [default]
commons-logging#commons-logging;1.1.3 from central in [default]
gov.nist.math#jama;1.0.3 from central in [default]
joda-time#joda-time;2.8.1 from central in [default]
log4j#log4j;1.2.15 from central in [default]
net.java.dev.jets3t#jets3t;0.6.1 from central in [default]
net.java.dev.jna#jna;4.0.0 from central in [default]
net.sf.opencsv#opencsv;2.3 from central in [default]
org.apache.commons#commons-math3;3.3 from central in [default]
org.apache.httpcomponents#httpclient;4.3.6 from central in [default]
org.apache.httpcomponents#httpcore;4.3.3 from central in [default]
org.eclipse.jetty#jetty-continuation;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-http;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-io;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-jndi;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-plus;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-security;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-server;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-servlet;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-util;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-webapp;8.1.17.v20150415 from central in [default]
org.eclipse.jetty#jetty-xml;8.1.17.v20150415 from central in [default]
org.eclipse.jetty.aggregate#jetty-servlet;8.1.17.v20150415 from central in [default]
org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 from central in [default]
org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020 from central in [default]
org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 from central in [default]
org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 from central in [default]
org.javassist#javassist;3.18.2-GA from central in [default]
org.joda#joda-convert;1.7 from central in [default]
org.kohsuke#libpam4j;1.8 from central in [default]
:: evicted modules:
joda-time#joda-time;2.3 by [joda-time#joda-time;2.8.1] in [default]
org.apache.httpcomponents#httpclient;4.1 by [org.apache.httpcomponents#httpclient;4.3.6] in [default]
org.apache.httpcomponents#httpcore;4.1 by [org.apache.httpcomponents#httpcore;4.3.3] in [default]
commons-logging#commons-logging;1.1.1 by [commons-logging#commons-logging;1.1.3] in [default]
commons-codec#commons-codec;1.4 by [commons-codec#commons-codec;1.6] in [default]
com.google.guava#guava;16.0.1 by [com.google.guava#guava;19.0] in [default]
com.google.guava#guava;18.0 by [com.google.guava#guava;19.0] in [default]
commons-codec#commons-codec;1.3 by [commons-codec#commons-codec;1.4] in [default]
commons-logging#commons-logging;1.0.4 by [commons-logging#commons-logging;1.1.1] in [default]
commons-codec#commons-codec;1.2 by [commons-codec#commons-codec;1.4] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 68 | 15 | 15 | 10 || 55 | 12 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
12 artifacts copied, 43 already retrieved (23416kB/63ms)
Traceback (most recent call last):
File "/home/hadoop/scripts/test/spark.py", line 3, in <module>
hc = H2OContext.getOrCreate(sc)
NameError: name 'H2OContext' is not defined
回答1:
You do not need to attach Sparkling Water packages (option --packages), but you need to provide pysparkling Python package (it contains all necessary binary dependencies internally).
The best way is to download binary distribution of Sparkling Water from http://h2o.ai/download and use bin/pysparkling script or use spark directly:
$SPARK_HOME/bin/pyspark --py-files h2o_pysparkling_2.1-2.1.8.zip
来源：https://stackoverflow.com/questions/44349062/error-with-h20context-running-pysparkling-with-spark-2-1