spark-submit fails to detect the installed modulus in pip

Question

I have a python code which have the following 3rd party dependencies:

import boto3
from warcio.archiveiterator import ArchiveIterator
from warcio.recordloade         


        

Answer 1

All checks mentioned above worked ok but setting PYSPARK_PYTHON solved the issue for me.