Why are my Google AppEngine instances crashing with 'error code 121'?

Question

I have been running a java high-replication web application on Google AppEngine for some time now. About two days ago - basically out of nowhere - a lot of requests began to fai

Answer 1

I don't have enough points to reply but I have a specific use case that seems interesting:

• python27
• hrd
• dynamic backed configured with 8 instances

Everything works as expected, except for one instance, instance=2, which basically cycles

• '/_ah/start'
• (Error code 121)
• '/_ah/stop'

Backends allows you to address a specific instance such as 2.backendname.appname.appspot.com and apparently something is wrong with that instance.

I suppose it's reassuring to know that there's one bad instance that's repeatedly failing due to a vague error code, instead of many instances failing randomly due to a vague error code. It'd be more reassuring if that instance were dropped, etc particularly if it's cycling this pattern.

Answer 2

Check your log retention limits, and that you haven't exceeded them. You wouldn't expect that bypassing your log retention limits would cause an exception that caused the instance to fail, but after I increased mine, I stopped seeing this error crop up and my backend cron jobs were able to complete.

Answer 3

Have not seen anything for sure about this from Google. They are often shy to explain such issues. One of the threads do mention datastore migration. See if that helps;

http://code.google.com/p/googleappengine/issues/detail?id=7022