Is there some directive I could use in my HTML to tell the browser not to ask for the favicon?
No, I don't think there is. From Wikipedia:
Most web browsers do not require any HTML to retrieve a favicon that conforms to the de facto file name and type (favicon.ico) located in the web site's root. If no favicon link is detected upon HTML page load completion and no previous site visits are recorded in the browser's history, a favicon.ico is requested automatically.[8]
The only thing you could do is explicitly point the browser to a different location (that for example could return a 204 no content)