Pyspark import .py file not working

Question

My goal is to import a custom .py file into my spark application and call some of the functions included inside that file

Here is what I tried:

I have a test

Answer 1

Alright, actually my question is rather stupid. After doing:

sc.addFile("/[AbsolutePathTo]/Test.py")

I still have to import the Test.py file like I would import a regular python file with:

import Test

then I can call

Test.func()

and it works. I thought that the "import Test" is not necessary since I add the file to the spark context, but apparently that does not have the same effect. Thanks mark91 for pointing me into the right direction.

UPDATE 28.10.2017:

as asked in the comments, here more details on the app.py

from pyspark import SparkContext
from pyspark.conf import SparkConf

conf = SparkConf()
conf.setMaster("local[4]")
conf.setAppName("Spark Stream")
sc = SparkContext(conf=conf)
sc.addFile("Test.py")

import Test

Test.func()