I have an application that needs to allow users to write expressions similar to excel:
(H1 + (D1 / C3)) * I8
and more complex things like
If(H1 = \'True
I would recommend the book Constructing Little Languages. It takes you through many compiler basics needed for completing this task properly.
You've brought up the fact that regular expressions will not work unless you have some stringent limits on your language. Like others have said, a Recursive Descent Parser will do the trick.
The choice next would be whether to use a Parser Generator like ANTLR, or to write one from scratch.