How to find pyspark dataframe memory usage?

Question

For python dataframe, info() function provides memory usage. Is there any equivalent in pyspark ? Thanks

Answer 1

I have something in mind, its just a rough estimation. as far as i know spark doesn't have a straight forward way to get dataframe memory usage, But Pandas dataframe does. so what you can do is.

1. select 1% of data sample = df.sample(fraction = 0.01)
2. pdf = df.toPandas()
3. get pandas dataframe memory usage by pdf.info()
4. Multiply that values by 100, this should give a rough estimate of your whole spark dataframe memory usage.
5. Correct me if i am wrong :|