Boto intermittent “unable to load credentials” with EC2 IAM roles

问题

I use an Elastic Beanstalk environment for deploying a web application, and I've set up an IAM role for the instances the application will run on.

Everything works flawlessy 99.99% of the time, however intermittently I will see errors in our logs with request failures showing botocore errors like the following:

File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 339, in property_loader
  self.load()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 505, in do_action
  response = action(self, *args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
  response = getattr(parent.meta.client, operation_name)(**params)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
  return self._make_api_call(operation_name, kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 648, in _make_api_call
  operation_model, request_dict, request_context)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/client.py", line 667, in _make_request
  return self._endpoint.make_request(operation_model, request_dict)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
  return self._send_request(request_dict, operation_model)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
  request = self.create_request(request_dict, operation_model)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
  operation_name=operation_model.name)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
  return self._emitter.emit(aliased_event_name, **kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
  return self._emit(event_name, kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
  response = handler(**kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
  return self.sign(operation_name, request)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/signers.py", line 157, in sign
  auth.add_auth(request)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
  raise NoCredentialsError
tocore.exceptions.NoCredentialsError: Unable to locate credentials

I don't know if this is just a race condition where IAM credentials are getting rotated (or something like that) and the instance's boto clients are trying to use them.

Any tips on remediating this or handling this failure gracefully?

P.S.

boto3==1.9.75
botocore==1.12.239
python==3.6.8

回答1:

We have the same issue. I had opened an issue in botocore repo Intermittent NoCredentialsError('Unable to locate credentials') error #1965

The workaround that we use now is to make retries via a retry on error decorator. So far, it works.

来源：https://stackoverflow.com/questions/58210314/boto-intermittent-unable-to-load-credentials-with-ec2-iam-roles

标签

amazon-ec2

boto3

amazon-iam

amazon-elastic-beanstalk

botocore