InvalidAccessKeyId - mocking no longer works

See original GitHub issue

Versions

  • boto==2.49.0
  • boto3==1.9.225
  • botocore==1.12.225
  • moto==1.3.13

This issue has been raised many times before it appears, however after reading through all the reports and changing versions up and down to no avail, I’ve decided to raise another issue. When I try to mock out AWS using moto, I am getting the following error:

(InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.

I’ve tried setting fake credentials using os.environ, I’ve tried deleting my credentials from the cli configuration, I’ve tried downgrading botocore/boto3 versions, I’ve tried re-arranging imports. None of this has helped unfortunately. Hopefully I’m missing something, but if not, there is something wrong.

The code

@mock_sqs
@mock_s3
@mock_sns
def test_bad_data(self):
    ...
    response = function.lambda_handler(
        None,
        None
    )
    ...

Calling the lambda_handler from the test is what lead us to the error:

def lambda_handler(event, context):
    ...
    try:
        ...
        s3 = boto3.resource('s3', region_name="eu-west-2")
        sqs = boto3.client('sqs', region_name="eu-west-2")
        lambda_client = boto3.client('lambda', region_name="eu-west-2")
        ...
        input_file = read_from_s3(s3, config['bucket_name'], config['file_name'])
        ...

It is where the read_from_s3 function is called that it tells me I have invalid credentials, which is the first instance of connecting to AWS. For reference, here is the read_from_s3 function:

def read_from_s3(s3, bucket_name, file_name):
    # Read from S3
    object = s3.Object(bucket_name, file_name)
    input_file = object.get()['Body'].read()

    return input_file

What was expected to happen Successfully mock AWS in tests.

What actually happens InvalidAccessKeyId error, suggesting that moto mocking isn’t working.

If I’ve missed any important details please let me know so I can update.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:13
  • Comments:25 (2 by maintainers)

github_iconTop GitHub Comments

6reactions
xiang-zhucommented, Sep 20, 2019

I’m seeing this same issue, had figured out what’s happening. It’s not caused by library versions, I just updated to use latest version for the libraries

  • boto: 2.49.0
  • boto3: 1.9.232
  • botocore: 1.12.232
  • moto: 1.3.13

moto is injecting the botocore_stubber to botocore.handlers.BUILTIN_HANDLERS, but for my case, there’s another import from code which includes a line of s3 = boto3.client('s3') call outside of functions, That import is before the import moto in my test, so it’s causing boto creating the DEFAULT_SESSION without the extra botocore_stubber handler, and it was used when the AWS call in test supposed to be mocked.

The workaround is to set boto3.DEFAULT_SESSION to None before the usage of moto mock, and this will cause a creation of new DEFAULT_SESSION with the botocore_stubber handler.

I did it with a pytest fixture

@pytest.fixture(scope="session")
def clear_default_boto3_session(): 
    boto3.DEFAULT_SESSION = None

def test_with_moto(clear_default_boto3_session):
    with mock_s3():
        s3 = boto3.resource('s3', region_name='us-east-1')
        assert [b for b in s3.buckets.all()] == []

Update: I also have to move the line of boto3.client in my code into a method to get the mock work with my code.

Update again: Instead of resetting the boto3.DEFAULT_SESSION and move my boto3.client call in code, I added import boto3 into my tests/__init__.py so it gets imported before anything else.

5reactions
whatnickcommented, Oct 12, 2021

Maintainer time is premium in open-source. If you need it merged, hop on the community and become maintainer. Otherwise, fork and host on a private PyPI.

On Mon, Oct 11, 2021, 03:02 Cyril Scetbon @.***> wrote:

It seems there’s been a PR opened for 3 months, any reason why it’s taking so long to get it reviewed, merged in etc… ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/spulec/moto/issues/2413#issuecomment-939512262, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADX7BAXQ2RKMRV6R6NUT53UGG52RANCNFSM4IVELS6A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Read more comments on GitHub >

github_iconTop Results From Across the Web

The AWS Access Key Id does not exist in our records
A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records....
Read more >
InvalidAccessKeyId - mocking no longer works - Bountysource
Successfully mock AWS in tests. ... InvalidAccessKeyId error, suggesting that moto mocking isn't working. If I've missed any important details ...
Read more >
Testing AWS Python code with moto - codecentric Blog
It is appended implicitly, when importing moto in your test code, but does not return (mock) anything by default. Mocking can be achieved...
Read more >
How To: Stub ".promise()" in AWS-SDK Node.js - Jeremy Daly
This works perfectly in production and the code is super compact and simple: ... In this case, the .promise() method no longer existed....
Read more >
S3 - The AWS Access Key Id you provided does not exist in ...
@body="<?xml version=\"1.0\" encoding=\"UTF-8\"?> <Error> <Code>InvalidAccessKeyId</Code>
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found