问题
Here's my spark code. It works fine and returns 2517. All I want to do is to print "2517 degrees"...but I'm not sure how to extract that 2517 into a variable. I can only display the dataframe but not extract values from it. Sounds super easy but unfortunately I'm stuck! Any help will be appreciated. Thanks!
df = sqlContext.read.format("csv").option("header", "true").option("inferSchema", "true").option("delimiter", "\t").load("dbfs:/databricks-datasets/power-plant/data")
df.createOrReplaceTempView("MyTable")
df = spark.sql("SELECT COUNT (DISTINCT AP) FROM MyTable")
display(df)
回答1:
I think you're looking for collect
. Something like this should get you the value:
df.collect()[0]['count(DISTINCT AP)']
assuming the column name is 'count(DISTINCT AP)'
来源:https://stackoverflow.com/questions/55406069/how-to-extract-a-single-column-row-value-from-a-dataframe-using-pyspark