Applying a function in each row of a big PySpark dataframe?

Question

I have a big dataframe (~30M rows). I have a function f. The business of f is to run through each row, check some logics and feed the outputs into a di

Answer 1

By using collect you pull all the data out of the Spark Executors into your Driver. You really should avoid this, as it makes using Spark pointless (you could just use plain python in that case).

What could you do:

• reimplement your logic using functions already available: pyspark.sql.functions doc

• if you cannot do the first, because there is functionality missing, you can define a User Defined Function