Can't pickle : attribute lookup class_name on abc failed

Question

I\'m getting the above error as I try to create dependencies (subtasks) based on dependency relationship defined in a dictionary (\"cmdList). For instance, \"BDX010\" is a d

Answer 1

When creating classes dynamically with a meta-class of ABC, the module becomes abc, and when a worker tries to find the task it goes to the abstract base class module and tries to find it there, but of course it does not exist.

To solve this, make sure luigi know where to find the code that build the class by manually resetting the __module__ variable.

Change the line to:

klass = type(queryKey, (BDX_Task,),{'__module__':__name__})

As far as I know, this is only a problem on Windows.

Edit: Sorry, stupid of me. Just must also make sure all custom classes gets re-created and added if a new process just imports the module.

# Run this first outside any other logic so it gets run if someone imports the module:
for queryKey in cmdList.keys():
    globals()[queryKey] = type(queryKey,(BDX_Task,){'__module__':__name__})

#Then you requires function can look like:
class BDX_Query_0XX(SQLTask):

    # ...

    def requires(self):
        for queryKey, (queryCmd, dependQry) in cmdList.items():
            yield globals()[queryKey](
                acctDate = self.acctDate,
                ssisDate = self.ssisDate,
                queryKey = queryKey,
                queryCmd = queryCmd,
                runDesc = self.runDesc,  
                dependQry = dependQry
            )