How to avoid pickling errors when sharing objects between threads?

Question

I have a program in which I need to store a global variable into a file. I am doing this using the pickle module.

I have another thread(Daemon=False, from threading module) which sometimes changes the value of the global variable. The value is also modified in global scope(the main program).

I am dumping the value of the variable into a .pkl file every 5 seconds (using another thread from threading module).

But I found the following error when dump method was executed:

TypeError: can't pickle _thread.lock objects

Why is this happening? And what can I do to fix it?

Note: I have found some similar answers with multiprocessing module. But I need an answer for threading module.

Code:

def save_state():
    while True:
        global variable

        lastSession = open('lastSession.pkl', 'wb')

        # error occurs on this line
        pickle.dump(variable, lastSession)

        lastSession.close()          
        time.sleep(5)

state_thread = threading.Thread(target = save_state)
state_thread.setDaemon(False)
state_thread.start()

# variable is changed outside this function and also in another thread(not state_thread).

Answer 1

As others have mentioned, you cannot pickle "volatile" entities (threads, connections, synchonization primitives etc.) because they don't make sense as persistent data.

Looks like what you're trying to do is be saving session variables for later continuation of said session. For that task, there's nothing you can do to save objects that, well, cannot be saved due to their nature.

The simplest solution is just to ignore them. Replacing them with some "stubs" that would yield an error anytime you do anything with them makes little sense, as a missing variable yields an error anyway (it won't if it was masking a global variable but that is a questionable practice in itself).

Alternatively, you can set up these objects anew when restoring a session. But such a logic is by necessity task-specific.

Finally, e.g. IPython already has session saving/restoration logic, so you probably don't need to reinvent the wheel at all.