Specifiying custom profilers for pyspark running Spark 2.0

我怕爱的太早我们不能终老 提交于 2019-12-12 00:48:42

问题


I would like to know how to specify a custom profiler class in PySpark for Spark version 2+. Under 1.6, I know I can do so like this:

sc = SparkContext('local', 'test', profiler_cls='MyProfiler')

but when I create the SparkSession in 2.0 I don't explicitly have access to the SparkContext. Can someone please advise how to do this for Spark 2.0+ ?


回答1:


SparkSession can be initialized with an existing SparkContext, for example:

from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.profiler import BasicProfiler

spark = SparkSession(SparkContext('local', 'test', profiler_cls=BasicProfiler))


来源:https://stackoverflow.com/questions/42676078/specifiying-custom-profilers-for-pyspark-running-spark-2-0

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!