How to delete columns in pyspark dataframe

Question

>>> a
DataFrame[id: bigint, julian_date: string, user_id: bigint]
>>> b
DataFrame[id: bigint, quan_created_money: decimal(10,0), quan_created_cnt: bigi         


        

Answer 1

Reading the Spark documentation I found an easier solution.

Since version 1.4 of spark there is a function drop(col) which can be used in pyspark on a dataframe.

You can use it in two ways

1. df.drop('age').collect()
2. df.drop(df.age).collect()

Pyspark Documentation - Drop