There are lots of good answers and pointers on this thread already, but let me add one thing.
DO NOT RELY ON TESTING TO FIND RACE CONDITIONS AND DEADLOCKS
Assuming you have all the good development processes in place: unit tests for each component, smoke tests for each nightly build, requiring each developer's changes to pass tests before checkin , etc.
All that is well and good, but it leads to an attitude of "well, it passed the test suite, so it can't be a bug in my code." which will not serve you well in concurrent programming. Real-time concurrency bugs are fiendishly hard to reproduce. You can run a piece of code with a race condition a billion times before it fails.
You will have to adjust your process to put greater emphasis on code reviews, conducted by your best minds. Having a seperate code review just for concurrency issues only is not a bad idea.
You will have to put more emphasis on making your application self-debugging. That is, when you get a failure in the test lab or at your customer's site, you need to make sure enough information is captured and logged to let you do a definitive postmortem, since the odds of your being able to reproduce the bug report at your convenience are negligible.
You will have to put more emphasis on paranoid sanity checks in your code so that a fault is detected as close to the problem as possible, and not 50,000 lines of code away.
Be paranoid. very paranoid.