Lets imagine that we have two assemblies:
Unless you use explicitly use SxS, your .NET 2 assembly will be loaded into the .NET 4 CLR. There shouldn't be any performance problems at all... the code which originally targeted .NET 2 will just run using the .NET 4 libraries and CLR, etc. And yes, you can use an assembly compiled with .NET 1.1 in a .NET 2 or higher app, too.
There is a problem if you try to load a mixed-mode assembly (some native code, some IL) in .NET 4, IIRC... but if it's "just" a .NET class library, you should be fine.
In the situation you described in the question, everything will Just Work™ without any problems.
A .NET 4.0 application will load a .NET 2.0 library directly into the .NET 4.0 runtime environment. It will not use side-by-side execution unless you explicitly ask it to. There's a lot of misinformation or unclear statements made about this around the web, so be careful in evaluating what you read.
I finally came across this article, which confirms what Jon Skeet says in his answer. In particular, the article explains (emphasis added):
In-Proc SxS does not solve the compatibility problems faced by library developers. Any libraries directly loaded by an application--either via a direct reference or an Assembly.Load--will continue to load directly into the runtime and AppDomain of the application loading it. This means that if an application is recompiled to run against the .NET Framework 4 runtime and still has dependent assemblies built against .NET 2.0, those dependents will load on the .NET 4 runtime as well. Therefore, we still recommend testing your libraries against all version[s] of the framework you wish to support. This is one of the reasons we have continued to maintain our high level of backward compatibility.
But if you're interested in even more of the nitty-gritty details, it's worth mentioning that one of the new features introduced with version 4.0 of the CLR is the ability to run multiple versions of the runtime at the same time from in a single process. This is called "In Process Side-by-Side Execution" (or "Inproc SxS" if you're very cool), and is a very powerful tool. You can read more about it here on the CLR team's blog.
Here are some pretty pictures for demonstration purposes:
Note the "layer-cake" model that was used to build the .NET 3.0 and 3.5 releases on top of .NET 2.0. That means that they'll all run side-by-side without any problems. But the isolation of the other versions will create a problem.
However, as mentioned above, .NET 4.0 solves this problem with Side-by-Side Execution:
Of course, this is mostly applicable to situations involving COM and interop with external applications like Outlook, but it's still a very cool feature.
But either way, you shouldn't see any problems whatsoever with this approach, either with compatibility, performance, or whatever else.