How do I truncate a PySpark dataframe of timestamp type to the day?

Question

I have a PySpark dataframe that includes timestamps in a column (call the column \'dt\'), like this:

2018-04-07 16:46:00
2018-03-06 22:18:00

<

Answer 1

You use wrong function. trunc supports only a few formats:

Returns date truncated to the unit specified by the format.

:param format: 'year', 'yyyy', 'yy' or 'month', 'mon', 'mm'

Use date_trunc instead:

Returns timestamp truncated to the unit specified by the format.

:param format: 'year', 'yyyy', 'yy', 'month', 'mon', 'mm', 'day', 'dd', 'hour', 'minute', 'second', 'week', 'quarter'

Example:

from pyspark.sql.functions import col, date_trunc

df = spark.createDataFrame(["2018-04-07 23:33:21"], "string").toDF("dt").select(col("dt").cast("timestamp"))

df.select(date_trunc("day", "dt")).show()
# +-------------------+                                                           
# |date_trunc(day, dt)|
# +-------------------+
# |2018-04-07 00:00:00|
# +-------------------+

Answer 2

One simple way to do it with string manipulation:

from pyspark.sql.functions import lit, concat

df = df.withColumn('date', concat(df.date.substr(0, 10), lit(' 00:00:00'))) 

Answer 3

For spark <= 2.2.0

Please use this:

from pyspark.sql.functions import weekofyear, year, to_date, concat, lit, col
from pyspark.sql.session import SparkSession
from pyspark.sql.types import TimestampType

spark = SparkSession.builder.getOrCreate()

spark.createDataFrame([['2020-10-03 05:00:00']], schema=['timestamp']) \
    .withColumn('timestamp', col('timestamp').astype(TimestampType())) \
    .withColumn('date', to_date('timestamp').astype(TimestampType())) \
    .show(truncate=False)

+-------------------+-------------------+
|timestamp          |date               |
+-------------------+-------------------+
|2020-10-03 05:00:00|2020-10-03 00:00:00|
+-------------------+-------------------+

For spark > 2.2.0 datetime patterns in spark 3.0.0

from pyspark.sql.functions import date_trunc, col
from pyspark.sql.session import SparkSession
from pyspark.sql.types import TimestampType

spark = SparkSession.builder.getOrCreate()

spark.createDataFrame([['2020-10-03 05:00:00']], schema=['timestamp']) \
    .withColumn('timestamp', col('timestamp').astype(TimestampType())) \
    .withColumn('date', date_trunc(timestamp='timestamp', format='day')) \
    .show(truncate=False)

+-------------------+-------------------+
|timestamp          |date               |
+-------------------+-------------------+
|2020-10-03 05:00:00|2020-10-03 00:00:00|
+-------------------+-------------------+