From MSDN (emphasis mine):
$"{person.Name, 20} is {person.Age:D3} year {(p.Age == 1 ? "" : "s")} old."
You do not need to quote the quotation characters within the contained interpolation expressions because interpolated string expressions start with $, and the compiler scans the contained interpolation expressions as balanced text until it finds a comma, colon, or close curly brace. For the same reasons, the last example uses parentheses to allow the conditional expression (p.Age == 1 ? "" : "s") to be inside the interpolation expression without the colon starting a format specification. Outside of the contained interpolation expression (but still within the interpolated string expression) you escape quotation characters as you normally would.
Without the parentheses, the parser is treating the portion after the colon as a format specifier instead (compare the {person.Age:D3} portion of the example above).
