Transfer and write Parquet with python and pandas got timestamp error

前端 未结 5 2272
春和景丽
春和景丽 2021-02-19 00:44

I tried to concat() two parquet file with pandas in python .
It can work , but when I try to write and save the Data frame to a parquet file ,it display the error :

<
5条回答
  •  庸人自扰
    2021-02-19 01:29

    I experienced a similar problem while using pd.to_parquet, my final workaround was to use the argument engine='fastparquet', but I realize this doesn't help if you need to use PyArrow specifically.

    Things I tried which did not work:

    • @DrDeadKnee's workaround of manually casting columns .astype("datetime64[ms]") did not work for me (pandas v. 0.24.2)
    • Passing coerce_timestamps='ms' as a kwarg to the underlying parquet operation did not change behaviour.

提交回复
热议问题