How to add any new library like spark-csv in Apache Spark prebuilt version

后端 未结 6 660
死守一世寂寞
死守一世寂寞 2020-12-12 19:35

I have build the Spark-csv and able to use the same from pyspark shell using the following command

bin/spark-shell --packages com.databricks:spark-csv_2.10:1         


        
6条回答
  •  有刺的猬
    2020-12-12 20:26

    Below command helped me -: With Scala 2.10 version

    /opt/mapr/spark/spark-1.5.2/bin/spark-shell --master local[*] --packages com.databricks:spark-csv_2.10:1.4.0
    

    Has below dependencies -:

    com.databricks#spark-csv_2.10;1.4.0!spark-csv_2.10.jar (2043ms)
    org.apache.commons#commons-csv;1.1!commons-csv.jar (419ms)
    com.univocity#univocity-parsers;1.5.1!univocity-parsers.jar (1481ms)
    

提交回复
热议问题