How to find pyspark dataframe memory usage?

Question

For python dataframe, info() function provides memory usage. Is there any equivalent in pyspark ? Thanks

Answer 1

You can persist dataframe in memory and take action as df.count(). You would be able to check the size under storage tab on spark web ui.. let me know if it works for you.