Can't remove netlink kernel module while a userspace app is connected to it

Question

I have a kernel module that communicates with userspace using Netlink, it looks like that

Kernel Module

#define NETLINK_USER 31
struct sock *nl_sk = NULL