aws-lambda-nodejs: Socket timed out without establishing a connection

See original GitHub issue

Describe the bug

I have anywhere between 20-50 nodejs lambda functions in single stack and I update their dependencies and deploy with cdk.

But lately I am not able to deploy updates. I get following error when I deploy.

current credentials could not be used to assume 'arn:aws:iam::******:role/cdk-hnb659fds-lookup-role-******-us-east-1', but are for the right account. Proceeding anyway.
(To get rid of this warning, please upgrade to bootstrap version >= 8)
current credentials could not be used to assume 'arn:aws:iam::******:role/cdk-hnb659fds-file-publishing-role-******-us-east-1', but are for the right account. Proceeding anyway.
current credentials could not be used to assume 'arn:aws:iam::******:role/cdk-hnb659fds-file-publishing-role-******-us-east-1', but are for the right account. Proceeding anyway.
[9%] fail: Socket timed out without establishing a connection
[18%] fail: Socket timed out without establishing a connection

I keep trying again and again and sometimes it goes through and most of the time it doesn’t work. Only stack with lower number of lambda functions sometimes gets deployed. But stack with large number of lambda functions fails 100% of the time.

Expected Behavior

I expected it to deploy no matter number of lambda functions in the stack. It used to get deployed without any problem.

Current Behavior

current credentials could not be used to assume 'arn:aws:iam::******:role/cdk-hnb659fds-lookup-role-******-us-east-1', but are for the right account. Proceeding anyway.
(To get rid of this warning, please upgrade to bootstrap version >= 8)

I don’t know how to upgrade bootstrap version. I ran cdk bootstrap multiple times and it says no changes.

Reproduction Steps

const testSignUpFn = new NodejsFunction(this, 'testSignUpNodeJS', {
      runtime: Runtime.NODEJS_14_X,
      entry: `${__dirname}/../lambda-fns/sign-up/index.ts`,
      handler: 'signUp',
      architecture: Architecture.ARM_64,
      memorySize: 1024
    })

It was working before but suddenly stopped working.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.20.0 (build 738ef49)

Framework Version

No response

Node.js Version

v16.14.2

OS

Ubuntu 20.04 on WSL 2

Language

Typescript

Language Version

~3.9.7

Other information

No response

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:13 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
jscrobinsoncommented, Nov 28, 2022

This does appear to be related to the Asset Parallelism feature. Executing a deployment with –asset-parallelism=false resulted in a successful deployment.

When running without –asset-parallelism=false the stack failed on the following error:

Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-ACCOUNT_ID-eu-west-2","Prefix":"0936406e22fea26017ecca536fcbdc550936406e22fea26017ecca536fcbdc55.zip","MaxKeys":1}) => Socket timed out without establishing a connection (code=TimeoutError)

There are only four assets in the bucket and none of them are over 50KB.

System information: OS: Ubuntu 20.04 NodeJS Version: v16.3.0 CDK verison: 2.51.1

1reaction
viktorchukhantsevcommented, Aug 31, 2022

Facing with this issue regularly now on 2.39.1 When I enable vpn and deploy again this error disappear so looks like this is somehow related to connection establishing issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Node Lambda AWS TimeoutError: Socket timed out without ...
Connecting a function to a public subnet does not give it internet access or a public IP address. There are two common solutions...
Read more >
Troubleshoot Lambda function retry and timeout issues when ...
AWS SDK, Maximum retry count, Connection timeout, Socket timeout ... TimeoutError: Socket timed out without establishing a connection .
Read more >
Interesting story about how we solved a 'socket hang up' issue ...
A: Looks like we gave a bit higher maxSockets value to our https agent, but the error is not like a timeout error...
Read more >
AWS Lambda Timeout Best Practices - Lumigo
Easily identify Lambda timeouts to keep your serverless apps running fast and performing at their best. No code, 5-minute set up. Start Lumigo...
Read more >
Using sequelize in AWS Lambda
Set this value to 0 so connection pool eviction logic eventually cleans up all connections * in the event of a Lambda function...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found