Unload shared library inside ctypes loaded shared library

Question

I\'m calling a so file from my python script. As far as I understand, I don\'t really need to free shared library that is opened in python using ctypes. However, inside my s

Answer 1

example to unload dependencies

Tested on Linux Fedora 32, Python 3.7.6 (anaconda), ctypes 1.1.0, g++ 10.2.1. Dependency is OpenCv (Version 4.2). More details here: How can I unload a DLL using ctypes in Python?

code.cpp

#include <opencv2/core/core.hpp>
#include <iostream> 


extern "C" int my_fct(int n)
{
    cv::Mat1b mat = cv::Mat1b(10,8,(unsigned char) 1 );  // change 1 to test unloading
    
    return mat(0,1) * n;
}

Compile with g++ code.cpp -shared -fPIC -Wall -std=c++17 -I/usr/include/opencv4 -lopencv_core -o so_opencv.so

Python code

from sys import platform
import ctypes


class CtypesLib:

    def __init__(self, fp_lib, dependencies=[]):
        self._dependencies = [CtypesLib(fp_dep) for fp_dep in dependencies]

        if platform == "linux" or platform == "linux2":  # Linux
            self._dlclose_func = ctypes.cdll.LoadLibrary('').dlclose
            self._dlclose_func.argtypes = [ctypes.c_void_p]
            self._ctypes_lib = ctypes.cdll.LoadLibrary(fp_lib)
        elif platform == "win32":  # Windows
            self._ctypes_lib = ctypes.WinDLL(fp_lib)

        self._handle = self._ctypes_lib._handle

    def __getattr__(self, attr):
        return self._ctypes_lib.__getattr__(attr)

    def __del__(self):
        for dep in self._dependencies:
            del dep

        del self._ctypes_lib

        if platform == "linux" or platform == "linux2":  # Linux
            self._dlclose_func(self._handle)
        elif platform == "win32":  # Windows
            ctypes.windll.kernel32.FreeLibrary(self._handle)


fp_lib = './so_opencv.so'

ctypes_lib = CtypesLib(fp_lib, ['/usr/lib64/libopencv_core.so'])

valIn = 1
ctypes_lib.my_fct.argtypes = [ctypes.c_int]
ctypes_lib.my_fct.restype = ctypes.c_int
valOut = ctypes_lib.my_fct(valIn)
print(valIn, valOut)

del ctypes_lib

Answer 2

The rule Clean up after yourself always applies (although modern technologies take care of the cleaning aspect for you).

[Python 3.5]: ctypes - A foreign function library for Python contains lots of useful info, and should be your friend.

ctypes uses dlopen whel loading a .dll. As I noticed, it doesn't call the corresponding dlclose meaning that the .dll (and all of its dependents that were loaded when loading it) will remain in memory until the process terminates (or until explicitly unloaded).

From [man7]: DLOPEN(3):

If the object specified by filename has dependencies on other shared objects, then these are also automatically loaded by the dynamic linker using the same rules. (This process may occur recursively, if those objects in turn have dependencies, and so on.)
...
If the same shared object is loaded again with dlopen(), the same object handle is returned. The dynamic linker maintains reference counts for object handles, so a dynamically loaded shared object is not deallocated until dlclose() has been called on it as many times as dlopen() has succeeded on it. Any initialization returns (see below) are called just once.

So, I don't think you'd have a problem (of course, everything depends on the context). As you noticed, loading a library multiple times doesn't actually load it every time, so the chance to run out of memory is pretty small (well unless you are loading a huge number of different .dlls, each with lots of different dependencies).

One case that I can think of is loading a .dll that uses a symbol from another .dll. If that symbol is also defined in another (3^rd) .dll, which was loaded before, then the code would behave differently than expected.

Anyway, you can manually unload (or better: decrease its refcount) a .dll (I'm not sure how this fits into the recommended ways or best practices ), like shown in the example below.

dll.c:

#include <stdio.h>


int test() {
    printf("[%s] (%d) - [%s]\n", __FILE__, __LINE__, __FUNCTION__);
    return 0;
}

code.py:

import sys
from ctypes import CDLL, \
    c_int, c_void_p


DLL = "./dll.so"

dlclose_func = CDLL(None).dlclose  # This WON'T work on Win
dlclose_func.argtypes = [c_void_p]


def _load_dll(dll_name):
    dll_dll = CDLL(dll_name)
    print("{:}".format(dll_dll))
    return dll_dll


def _load_test_func(dll):
    test_func = dll.test
    test_func.restype = c_int
    return test_func


def main():
    print("Loading a dll via `ctypes`, then delete the object. The dll is not unloaded. Call `dlclose` to unload. A 2nd call will fail.")
    dll_dll = _load_dll(DLL)
    dll_handle = dll_dll._handle
    del dll_dll
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll_handle)))  # Even if the ctypes dll object was destroyed, the dll wasn't unloaded
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll_handle)))  # A new dlclose call will fail

    print("\nUse `ctypes` to load the dll twice. The dll is not actually loaded only the 1st time (both have the same handle), but its ref count is increased. `dlclose` must be also called twice.")
    dll0_dll = _load_dll(DLL)
    dll1_dll = _load_dll(DLL)
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll0_dll._handle)))
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll1_dll._handle)))
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll1_dll._handle)))

    print("\nLoad a dll via `ctypes`, and load one of its funcs. Try calling it before and after unloading the dll.")
    dll_dll = _load_dll(DLL)
    test_func = _load_test_func(dll_dll)
    print("{:} returned {:d}".format(test_func.__name__, test_func()))
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll_dll._handle)))
    print("{:} returned {:d}".format(test_func.__name__, test_func()))  # Comment this line as it would segfault !!!



if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

Output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q052179325]> ls
code.py  dll.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q052179325]> gcc -fPIC -shared -o dll.so dll.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q052179325]> python3 ./code.py
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux

Loading a dll via `ctypes`, then delete the object. The dll is not unloaded. Call `dlclose` to unload. A 2nd call will fail.
<CDLL './dll.so', handle 1d7aa20 at 0x7fa38715f240>
dlclose returned 0
dlclose returned -1

Use `ctypes` to load the dll twice. The dll is not actually loaded only the 1st time (both have the same handle), but its ref count is increased. `dlclose` must be also called twice.
<CDLL './dll.so', handle 1de2c80 at 0x7fa38715f240>
<CDLL './dll.so', handle 1de2c80 at 0x7fa38715f278>
dlclose returned 0
dlclose returned 0
dlclose returned -1

Load a dll via `ctypes`, and load one of its funcs. Try calling it before and after unloading the dll.
<CDLL './dll.so', handle 1de39c0 at 0x7fa38715f8d0>
[dll.c] (5) - [test]
test returned 0
dlclose returned 0
Segmentation fault (core dumped)