Efficient multiprocessing file parsing with deduplication between processes

Question

The job is to parse hundreds of GBs of files (millions of small files) and insert them into a database. To do that job efficiently, it is possible to split the files between