How to get around the pickling error of python multiprocessing without being in the top-level?

Question

I\'ve researched this question multiple times, but haven\'t found a workaround that either works in my case, or one that I understand, so please bear with me.

Basically

Answer 1

The problem you encountered is actually a feature. The pickle source is actually designed to prevent this sort of behavior in order to prevent malicious code from being executed. Please consider that when addressing any applicable security implementation.

First off we have some imports.

import marshal
import pickle
import types

Here we have a function which takes in a function as an argument, pickles the parts of the object, then returns a tuple containing all the parts:

def pack(fn):
    code = marshal.dumps(fn.__code__)
    name = pickle.dumps(fn.__name__)
    defs = pickle.dumps(fn.__defaults__)
    clos = pickle.dumps(fn.__closure__)
    return (code, name, defs, clos)

Next we have a function which takes the four parts of our converted function. It translates those four parts, and creates then returns a function out of those parts. You should take note that globals are re-introduced into here because our process does not handle those:

def unpack(code, name, defs, clos):
    code = marshal.loads(code)
    glob = globals()
    name = pickle.loads(name)
    defs = pickle.loads(defs)
    clos = pickle.loads(clos)
    return types.FunctionType(code, glob, name, defs, clos)

Here we have a test function. Notice I put an import within the scope of the function. Globals are not handled through our pickling process:

def test_function(a, b):
    from random import randint
    return randint(a, b)

Finally we pack our test object and print the result to make sure everything is working:

packed = pack(test_function)
print((packed))

Lastly, we unpack our function, assign it to a variable, call it, and print its output:

unpacked = unpack(*packed)
print((unpacked(2, 20)))

Comment if you have any questions.