You are right that ConcurrentBag creates a ThreadLocal copy, in fact they are optimized for scenarios where the same thread is reading and writing the data to the bag: "... ConcurrentBag is a thread-safe bag implementation, optimized for scenarios where the same thread will be both producing and consuming data stored in the bag."
On the otherhand, I do not see a strange behaviour here; the thread lives and the concurrent bag lives. When thread finishes GC will do it's job.