(backtick) and \' are two different characters for single quote. I have a mysql script that shows the former two quote characters, if I change them to \', it breaks the syntax.
You are referring to the character commonly called a "backtick" (`). It is not a single quote, although in some fonts it can look like one. It has a completely different meaning in MySQL than a single quote, as it is used to escape table and column names, whereas the single quote is used to enclose data values.
It is at the top-left of your keyboard in all likelihood, often just to the left of the 1 key (as Dav pointed out below, on US keyboard layouts, it's the non-shift version of the tilde (~) key).