You shouldn't use regular expression to do this. Instead you can iterate over the string character by character and keep track of the nesting level.
Initially the nesting is 0. When you see a (
increase the nesting by 1, and when you see )
decrease the nesting. The expression is correctly balanced if the final nesting is 0 and the nesting never goes below 0.
public static boolean checkParentheses(String s) {
int nesting = 0;
for (int i = 0; i < s.length(); ++i)
{
char c = s.charAt(i);
switch (c) {
case '(':
nesting++;
break;
case ')':
nesting--;
if (nesting < 0) {
return false;
}
break;
}
}
return nesting == 0;
}