Pandas cannot read parquet files created in PySpark

Question

I am writing a parquet file from a Spark DataFrame the following way:

df.write.parquet(\"path/myfile.parquet\", mode = \"overwrite\", compression=\"gzip\")
<         


        

Answer 1

If the parquet file has been created with spark, (so it's a directory) to import it to pandas use

from pyarrow.parquet import ParquetDataset

dataset = ParquetDataset("file.parquet")
table = dataset.read()
df = table.to_pandas()