In PySpark, how can I log to log4j from inside a transformation

Question

I want to log to the standard logger inside an executor during transformation with log levels and formatting respected. Unfortunately I can\'t get access to the log4j logger obj

Answer 1

Have a look at this question

Similar situation

You can get your map function to return you an object that can contain a stack trace string or a real object, and a bool flag stating if there was an error. This can be useful to debug a task that has side effects, or if you have specific data conditions that cause failures.