Pyspark socket timeout exception after application running for a while

Question

I am using pyspark to estimate parameters for a logistic regression model. I use spark to calculate the likelihood and gradients and then use scipy\'s minimize function for

Answer 1

Check the executors logs for details. I have seen similar errors when executors die or are killed by the cluster manager (usually for using more memory than the container is configured for).