Googlebot receiving missing template error for an existing template

Question

In the last couple of days, we have started to receive a missing template error when the google bot attempts to access our main home page (welcome/index). I have been staring at

Answer 1

These errors are coming from the way GoogleBot formats its HTTP_ACCEPT header. While valid (see W3 reference), it adds a q=0.6 (last figure may change) which is used as a separator. Since there is no other media type specified, this q=0.6 is not necessary and I assume this is why Rails doesn't treat the header correctly.

(It seems to depend on Rails version. On Rails 3.0.12, it raises a MissingTemplate exception.)

Adding the following code from a previous answer to the concerned controller is not sufficient: it responds with an error 406.

respond_to do |format|
  format.html
end

To make this work under Rails 3.0.12 and have something returned to the GoogleBot (better than a 406 error), you need to add this code which sets the request's format to html as soon a */*;q=0.6-like HTTP_ACCEPT is detected (Rails load the header value into request.format).

# If the request 'HTTP_ACCEPT' header indicates a '*/*;q=0.6' format,
# we set the format to :html.
# This is necessary for GoogleBot which perform its requests with '*/*;q=0.6'
# or similar HTTP_ACCEPT headers.
if request.format.to_s =~ %r%\*\/\*%
  request.format = :html
end

respond_to do |format|
  format.html
end

While working, this solution needs the code to be added to any controller action you want to be indexed by the GoogleBot, what is really not DRY!

To fix this issue once for all, I implemented a small Rack middleware which does even better: it checks the request's HTTP_ACCEPT header, and will replace any header matching */*;q=0.6 (the figures can vary) by the common */*. This is even better because since the q=0.6 has no meaning if it is not followed by another media type, this change of the header doesn't change its meaning. We don't force Rails into any given format, we just tell it any will do in a way it can understand.

You can find the middleware, the loading initializer and an integration test in this gist.

Gem version here: https://github.com/ouvrages/rails_fix_google_bot_accept