TransactionFailedError on GAE when no transaction

问题

I got this error:

TransactionFailedError: too much contention on these datastore entities. please try again.

Even though I'm not doing any transactions. The line of my code that causes the error is

ndb.put_multi(entity_list) # entity_list is a list of 100 entities

This error doesn't happen often so it isn't a big deal, but I'm curious why I get this error. Any ideas?

Here is most of the traceback:

Traceback (most recent call last):
  ...
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 318, in post
    self.run_from_request()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 313, in run_from_request
    run(self.request.body)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 155, in run
    return func(*args, **kwds)
  File "/base/data/home/apps/s~opavote/2017-09-15.404125237783169549/tasks.py", line 70, in start_election
    models.Voter.create(e.eid, chunk)
  File "/base/data/home/apps/s~opavote/2017-09-15.404125237783169549/models.py", line 2426, in create
    ndb.put_multi(voters + vbs)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3958, in put_multi
    for future in put_multi_async(entities, **ctx_options)]
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
    self.check_success()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 824, in put
    key = yield self._put_batcher.add(entity, options)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 358, in _put_tasklet
    keys = yield self._conn.async_put(options, datastore_entities)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
    result = rpc.get_result()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 928, in get_result
    result = rpc.get_result()
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1893, in __put_hook
    self.check_rpc_success(rpc)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1385, in check_rpc_success
    raise _ToDatastoreError(err)
TransactionFailedError: too much contention on these datastore entities. please try again.

回答1:

Note that the error is actually received from the datastore itself, in the RPC response: self.check_rpc_success(rpc).

Which makes me suspect that on the datastore side, to ensure operation consistency/reliability across the redundant pieces of infra supporting it, every write operation is actually using the same/similar mechanisms as for transactional operations. The difference would be that those also have some transactional checks on the client side, before/after the RPC exchange and maybe explicit RPC transaction start/end triggers for the datastore.

From Life of a Datastore Write, a quote suggesting that some common mechanisms are being used regardless of the operations being transactional or not (emphasis mine):

If the commit phase has succeeded but the apply phase failed, the datastore will roll forward to apply the changes to indexes under two circumstances:

1. The next time you execute a read or write or start a transaction on this entity group, the datastore will first roll forward and fully apply this committed but unapplied write, based on the data in the log.

And one of the possible reasons for failures would be simply too many parallel accesses to the same entities, even if they're just read-only. See Contention problems in Google App Engine, though in that case they're for transactions on the client side.

Note that this is just a theory ;)

回答2:

It might be worth re-reviewing transactions and entity groups, noting the various definitions and limits.

Putting "Every attempt to create, update, or delete an entity takes place in the context of a transaction," and, "There is a write throughput limit of about one transaction per second within a single entity group," probably speaks to what you're seeing, particularly if entity_list contains entities that would fall into the same entity group.

来源：https://stackoverflow.com/questions/46367551/transactionfailederror-on-gae-when-no-transaction