Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Behavior of @aws-sdk/credential-providers fromTemporaryCredentials has changed and results in InvalidClientTokenId #6869

Open
4 tasks done
jandppw opened this issue Feb 6, 2025 · 12 comments
Assignees
Labels
bug This issue is a bug. potential-regression Marking this issue as a potential regression to be checked by team member queued This issues is on the AWS team's backlog

Comments

@jandppw
Copy link

jandppw commented Feb 6, 2025

Checkboxes for prior research

Describe the bug

When upgrading @aws-sdk/credential-providers from 3.721.0 to 3.742.0, our tests fail when fromTemporaryCredentials is called.

According to credential-providers/CHANGELOG.md, there are changes to fromTemporaryCredentials in 3.731.0 and 3.734.0.

Regression Issue

  • Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/credential-providers@3.742.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.13.1

Reproduction Steps

The relevant code is:


const fromTemporaryCredentialsOptions = {
  params: {
    RoleArn: `arn:aws:iam::${id}:role/${rolePathAndName}`,
    // Member must have length less than or equal to 64
    RoleSessionName: roleSessionName,
    DurationSeconds: 1800
  },
  clientConfig: { region, profile }
}
const credentialsProvider = fromTemporaryCredentials(fromTemporaryCredentialsOptions)
const credentials = await credentialsProvider()
return { region, credentials } // awsV3ClientConfig

This is general code, that has the intention to get temporary credentials where the "user" gets necessary permissions by assuming a role (RoleArn). The user‘s credentials should be found in ~/.aws/credentials under the given profile. The returned credentials are then used in different places to create a client to perform commands, e.g.,

const apiGatewayClient = new APIGatewayClient(awsV3ClientConfig)
instance = new S3Client(awsV3ClientConfig)

(but we don’t get there).

Observed Behavior

exception (STSServiceException):
InvalidClientTokenId: The security token included in the request is invalid.
    at throwDefaultError (…/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
    at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
    at de_CommandError (…/node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
    at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.base.js:108:33
    at async clientConfig (…/common/awsV3ClientConfig.js:45:27)

Expected Behavior

Return the correct credentials for the role as before.

Possible Solution

No response

Additional Information/Context

No response

@jandppw jandppw added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 6, 2025
@github-actions github-actions bot added the potential-regression Marking this issue as a potential regression to be checked by team member label Feb 6, 2025
@jandppw
Copy link
Author

jandppw commented Feb 6, 2025

  • The problem does not occur when all other updates are applied, and @aws-sdk/credential-providers@3.721.0 is used.
  • @aws-sdk/credential-providers@3.731.0 has the issue.
  • @aws-sdk/credential-providers@3.726.0 does not have the issue.
  • @aws-sdk/credential-providers@3.729.0 does not have the issue.

@aws-sdk/credential-providers@3.730.0 does have an issue, but here the error says

Error: `credentials` is missing
    at credentialsProvider (node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/core/dist-cjs/submodules/httpAuthSchemes/index.js:211:15)
    at boundCredentialsProvider (node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/core/dist-cjs/submodules/httpAuthSchemes/index.js:215:71)
    at …/node_modules/@smithy/core/dist-cjs/index.js:85:23
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async …/node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:50:33
    at async clientConfig (common/awsV3ClientConfig.js:45:27)

In @aws-sdk/credential-providers@3.731.0, the error changes to the InvalidClientTokenId described above.

@kuhe
Copy link
Contributor

kuhe commented Feb 6, 2025

Mocking of @aws-sdk/client-sts needs to be transferred to @aws-sdk/nested-clients/sts, if your test does this.

what is the output from the credential provider log?

import { fromTemporaryCredentials } from "@aws-sdk/credential-providers";


const provider = fromTemporaryCredentials({
  logger: console,
  params: {
    RoleArn: "arn:aws:iam::123...456:role/{role}",
  },
  clientConfig: {
    region: "us-west-2",
    profile: "profile_name",
  },
});

const credentials = await provider();

output:

DEBUG: @aws-sdk/credential-providers - fromTemporaryCredentials (STS)
DEBUG: @aws-sdk/credential-providers - fromTemporaryCredentials STS client init with options.clientConfig.region=us-west-2, AWS SDK default credentials, STS default requestHandler.

when I run this I get credentials successfully.

@kuhe kuhe added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Feb 6, 2025
@aBurmeseDev aBurmeseDev removed the needs-triage This issue or PR still needs to be triaged. label Feb 7, 2025
@jandppw
Copy link
Author

jandppw commented Feb 7, 2025

@kuhe Thanks for the response.

Mocking of @aws-sdk/client-sts needs to be transferred to @aws-sdk/nested-clients/sts, if your test does this.

I do not really understand what you are saying here. If you assume there is something mocked in this test: there isn’t.

There are new versions since yesterday. @aws-sdk/credential-providers@3.743.0 has the issue.

Thank you for the suggestion to add logger: console. I didn’t know about that.

Here is the complete code, simplified to zoom in on the issue:

…/common/awsV3ClientConfig.js:

const { fromTemporaryCredentials } = require('@aws-sdk/credential-providers')
const { v4: uuidv4 } = require('uuid')
const { STSClient, GetCallerIdentityCommand } = require('@aws-sdk/client-sts')

/**
 * @param {string} region
 * @param {string} profile
 * @returns {Promise<string>}
 */
async function accountId(region, profile) {
  console.info('Using AWS credentials from `~/.aws/credentials to determine account id`')
  /** @type {STSClient | undefined } */
  const stsClient = new STSClient({ region, profile })
  const command = new GetCallerIdentityCommand({})
  const response = await stsClient.send(command)
  const accountId = response.Account
  console.info('  AWS Account ID retrieved') // do not log accountId

  return accountId
}

/**
 * @param {string} region
 * @param {string} profile
 * @param {string} rolePathAndName
 * @returns {Promise<AWSv3ClientConfig>}
 */
async function clientConfig(region, profile, rolePathAndName) {
  console.info('Using AWS credentials from `~/.aws/credentials` to assume role.')
  const id = await accountId(region, profile)
  const roleSessionName = `XXXXXX-${uuidv4()}`
  /** @type {import('@aws-sdk/credential-providers').FromTemporaryCredentialsOptions} */
  const fromTemporaryCredentialsOptions = {
    logger: console, // NOTE: added as suggested by @kuhe
    params: {
      RoleArn: `arn:aws:iam::${id}:role/${rolePathAndName}`,
      // Member must have length less than or equal to 64
      RoleSessionName: roleSessionName,
      DurationSeconds: 1800
    },
    clientConfig: { region, profile }
  }
  const credentialsProvider = fromTemporaryCredentials(fromTemporaryCredentialsOptions)
  const credentials = await credentialsProvider()
  console.info(`  AWS credentials for role ${rolePathAndName} received (session name: ${roleSessionName}). Assuming.`)

  return { region, credentials }
}

module.exports = {
  clientConfig
}

…/test/unit/00.common/awsV3ClientConfig.test.js:

const { clientConfig } = require('../../../common/awsV3ClientConfig')
const { profile, automatedTestRoleNameAndPath } = require('../../../common/developerInfo')

const region = 'eu-west-1'

describe('awsV3ClientConfig, function () {
  describe('clientConfig', function () {
    it('works', async function () {
      const result = await clientConfig(region, profile, automatedTestRoleNameAndPath)
      console.log(result)
      result.should.be.an.Object()
    })
    
  })
})

The output when using @aws-sdk/credential-providers@3.729.0 is:

Using AWS credentials from `~/.aws/credentials` to assume role.
Using AWS credentials from `~/.aws/credentials to determine account id`
  AWS Account ID retrieved
@aws-sdk/credential-providers - fromTemporaryCredentials (STS)
  AWS credentials for role automated-test/automated-test-XXXXXX received (session name: XXXXXX-f89c3d0a-ebae-47a1-bfc0-7da67adab1a5). Assuming.
{
  region: 'eu-west-1',
  credentials: {
    accessKeyId: 'AXXXXXXXXXXXXXXXXXXXX',
    secretAccessKey: 'cBXXXXXXXXXXXXXXXXXXXX',
    sessionToken: 'IQXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXFA==',
    expiration: 2025-02-07T09:43:13.000Z,
    credentialScope: undefined
  }
}

The output when using @aws-sdk/credential-providers@3.743.0 is:

Using AWS credentials from `~/.aws/credentials` to assume role.
Using AWS credentials from `~/.aws/credentials to determine account id`
  AWS Account ID retrieved
@aws-sdk/credential-providers - fromTemporaryCredentials (STS)
@aws-sdk/credential-providers - fromTemporaryCredentials STS client init with options.clientConfig.region=eu-west-1, AWS SDK default credentials, STS default requestHandler.
endpoints Initial EndpointParams: {
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Region": "eu-west-1",
  "UseDualStack": false
}
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints assign: PartitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints Resolving endpoint from template: {
  "url": "https://sts.{Region}.{PartitionResult#dnsSuffix}",
  "properties": {},
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {},
  "url": "https://sts.eu-west-1.amazonaws.com/"
}
@smithy/property-provider -> Not found in config files w/ profile [<PROFILE>]: CONFIG_RETRY_MODE
@smithy/property-provider -> Not found in config files w/ profile [<PROFILE>]: CONFIG_MAX_ATTEMPTS
{
  clientName: 'STSClient',
  commandName: 'AssumeRoleCommand',
  input: {
    RoleArn: 'arn:aws:iam::XXXXXXXXXXXXX:role/automated-test/automated-test-XXXXXXX',
    RoleSessionName: 'XXXXXX-767fdf62-8b1f-4094-981e-285b55fa1125',
    DurationSeconds: 1800
  },
  error: InvalidClientTokenId: The security token included in the request is invalid.
      at throwDefaultError (…/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
      at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
      at de_CommandError (…/node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
      at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
      at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
      at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.base.js:108:33
      at async clientConfig (…/common/awsV3ClientConfig.js:49:23)
      at async Context.<anonymous> (…/test/unit/00.common/awsV3ClientConfig.test.js:9:22) {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 403,
      requestId: '058110da-de42-4413-975a-100a1582c15f',
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Type: 'Sender',
    Code: 'InvalidClientTokenId'
  },
  metadata: {
    httpStatusCode: 403,
    requestId: '058110da-de42-4413-975a-100a1582c15f',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

InvalidClientTokenId: The security token included in the request is invalid.
    at throwDefaultError (node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
    at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
    at de_CommandError (node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
    at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.base.js:108:33
    at async clientConfig (common/awsV3ClientConfig.js:49:23)
    at async Context.<anonymous> (test/unit/00.common/awsV3ClientConfig.test.js:9:22)

Note that getting the account id with accountId(region, profile) with the same credentials works.

The output when using @aws-sdk/credential-providers@3.730.0 is:

Using AWS credentials from `~/.aws/credentials` to assume role.
Using AWS credentials from `~/.aws/credentials to determine account id`
  AWS Account ID retrieved
@aws-sdk/credential-providers - fromTemporaryCredentials (STS)

Error: `credentials` is missing
    at credentialsProvider (node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/core/dist-cjs/submodules/httpAuthSchemes/index.js:211:15)
    at boundCredentialsProvider (node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/core/dist-cjs/submodules/httpAuthSchemes/index.js:215:71)
    at …/node_modules/@smithy/core/dist-cjs/index.js:85:23
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async …/node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:50:33
    at async clientConfig (common/awsV3ClientConfig.js:49:23)
    at async Context.<anonymous> (test/unit/00.common/awsV3ClientConfig.test.js:9:22)

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Feb 8, 2025
@kuhe kuhe added the needs-triage This issue or PR still needs to be triaged. label Feb 10, 2025
@kuhe kuhe assigned kuhe and unassigned kuhe Feb 10, 2025
@kuhe kuhe removed the needs-triage This issue or PR still needs to be triaged. label Feb 10, 2025
@kuhe
Copy link
Contributor

kuhe commented Feb 10, 2025

We usually need a more specific reproduction sample repository for issues with this many variables that are unseen, such as ENV and credentials file(s).

But, I have a theory as to what is happening. It is related to a fix in behavior for fromTemporaryCredentials, and interacts with the value you have set for region in two places: the clientConfig object and your configuration files.

In v3.729.0, the region set in the profile takes precedence and in the latest version of the SDK the clientConfig.region takes higher precedence. The latter is intended as the correct way to set the region, because the field clientConfig is intended as the highest precedence when configuring credential clients that are not instantiated by the user.

Your code-level configured region appears to be eu-west-1. Please compare with what region is set in the configuration file of the profile being used, and ensure that your AWS account enabled the intended regions and permission levels are set in those regions.

@kuhe kuhe added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Feb 10, 2025
Copy link

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

@github-actions github-actions bot added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 21, 2025
@jandppw
Copy link
Author

jandppw commented Feb 21, 2025

@kuhe

I have a theory as to what is happening. […]

In v3.729.0, the region set in the profile takes precedence and in the latest version of the SDK the clientConfig.region
takes higher precedence. […]

Thank you! That sounds like a sound theory based on a change after @aws-sdk/credential-providers@3.729.0. Looking into it now …

@jandppw
Copy link
Author

jandppw commented Feb 21, 2025

@kuhe

We usually need a more specific reproduction sample repository for issues with this many variables that are unseen, such
as ENV and credentials file(s).

I'm sorry, but I probably have little more to give. The point is that I can run the above code with @aws-sdk/credential-providers@3.729.0 without the issue, and switch to one of the later versions to reproduce the error, without changing anything else, back and forth. In my mind that rules out influence of any other variable. Note that GetCallerIdentity works in all cases.

FYI: @aws-sdk/credential-providers@3.750.0 also shows the issue. The output changed a bit:

Using AWS credentials from `~/.aws/credentials` to assume role.
Using AWS credentials from `~/.aws/credentials to determine account id`
  AWS Account ID retrieved
@aws-sdk/credential-providers - fromTemporaryCredentials (STS)
@aws-sdk/credential-providers - fromTemporaryCredentials STS client init with options.clientConfig.region=eu-west-1, AWS SDK default credentials, STS default requestHandler.
endpoints Initial EndpointParams: {
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Region": "eu-west-1",
  "UseDualStack": false
}
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints assign: PartitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints Resolving endpoint from template: {
  "url": "https://sts.{Region}.{PartitionResult#dnsSuffix}",
  "properties": {},
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {},
  "url": "https://sts.eu-west-1.amazonaws.com/"
}
@smithy/property-provider -> Not found in config files w/ profile [<PROFILE>]: CONFIG_RETRY_MODE
@smithy/property-provider -> Not found in config files w/ profile [<PROFILE>]: CONFIG_MAX_ATTEMPTS
{
  clientName: 'STSClient',
  commandName: 'AssumeRoleCommand',
  input: {
    RoleArn: 'arn:aws:iam::XXXXXXX:role/automated-test/automated-test-XXXXXXX',
    RoleSessionName: 'XXXXXX-5b70a339-4460-4897-b80f-74ea3b8724c6',
    DurationSeconds: 1800
  },
  error: InvalidClientTokenId: The security token included in the request is invalid.
      at throwDefaultError (…/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
      at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
      at de_CommandError (…/node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
      at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
      at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
      at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      at async …/node_modules/
InvalidClientTokenId: The security token included in the request is invalid.
    at throwDefaultError (node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
    at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
    at de_CommandError (node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
    at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.base.js:108:33
    at async clientConfig (common/awsV3ClientConfig.js:49:23)
    at async Context.<anonymous> (test/unit/00.common/awsV3ClientConfig.test.js:9:22)

@jandppw
Copy link
Author

jandppw commented Feb 21, 2025

@kuhe

I have a theory as to what is happening. […]

In v3.729.0, the region set in the profile takes precedence and in the latest version
of the SDK the clientConfig.region takes higher precedence. […]

Your code-level configured region appears to be eu-west-1.

Indeed. Confirmed. This is 'eu-west-1. This is used in parameter in STSClient
constructor in accountId, and in the parameter to fromTemporaryCredentials as
clientConfig.

[…] what region is set in the configuration file of the profile being used […]

In this setting, the only configuration is a ~/.aws/credentials file, that contains a
number of profiles.

  • there is a [default] profile
  • there are 8 other profiles, the <PROFILE> used next to the region in the places
    discussed above included
  • every one of these 9 profiles has only the aws_access_key_id and
    aws_secret_access_key set; no other properties are set in any of these profiles;
    notably, none of these profiles have the region set in the ~/.aws/credentials file
[<PROFILE>]
aws_access_key_id=AXXXXXXXXXXXXXXXXXXXXXXXXX
aws_secret_access_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If I combine this information with your theory, the following should be the case:

  • The call done inside fromTemporaryCredentials is done in the later versions of the SDK
    to region 'eu-west-1', which seems to be confirmed by the output shown in earlier
    comments
    (endpoints Resolved endpoint: { … "url": "https://sts.eu-west-1.amazonaws.com/" … }).
  • fromTemporaryCredentials executions that call region 'eu-west-1' fail.

When using @aws-sdk/credential-providers@3.729.0, there is no such output, although now
logger: console is added to the parameters to the call.

By you theory, fromTemporaryCredentials in @aws-sdk/credential-providers@3.729.0 would
contact another region, where it does work. Since no region is setup in any profile in
~/.aws/credentials, that would assume that the “other region” came from somewhere else.
There is this notion of "implicitGlobalRegion": "us-east-1" in the logs. Since getting
the accountId via the STSClient works in all versions, that would mean a different
logic is used there?

As an experiment, I used @aws-sdk/credential-providers@3.750.0 with the region
explicitly set in the profile in ~/.aws/credentials. As far as I can see, the output is
exactly the same as without the region set in the profile.

As an experiment, I used @aws-sdk/credential-providers@3.750.0 with the region
explicitly set to 'us-east-1' in the profile in ~/.aws/credentials. As far as I can
see, the output is exactly the same.

As an experiment, I ran the test with the region set in code explicitly to
'us-east-1'.

For @aws-sdk/credential-providers@3.750.0, the output now is:

Using AWS credentials from `~/.aws/credentials` to assume role.
Using AWS credentials from `~/.aws/credentials to determine account id`
  AWS Account ID retrieved
@aws-sdk/credential-providers - fromTemporaryCredentials (STS)
@aws-sdk/credential-providers - fromTemporaryCredentials STS client init with options.clientConfig.region=us-east-1, AWS SDK default credentials, STS default requestHandler.
endpoints Initial EndpointParams: {
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Region": "us-east-1",
  "UseDualStack": false
}
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "US East (N. Virginia)"
}
endpoints assign: PartitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "US East (N. Virginia)"
}
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints Resolving endpoint from template: {
  "url": "https://sts.{Region}.{PartitionResult#dnsSuffix}",
  "properties": {},
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {},
  "url": "https://sts.us-east-1.amazonaws.com/"
}
@smithy/property-provider -> Not found in config files w/ profile [<PROFILE>]: CONFIG_RETRY_MODE
@smithy/property-provider -> Not found in config files w/ profile [<PROFILE>]: CONFIG_MAX_ATTEMPTS
{
  clientName: 'STSClient',
  commandName: 'AssumeRoleCommand',
  input: {
    RoleArn: 'arn:aws:iam::XXXXXXXXXXX:role/automated-test/automated-test-XXXXXXXXX',
    RoleSessionName: 'XXXXXXXXXXXX-3b90fe18-db24-4e0a-85d3-2ee7d638dbad',
    DurationSeconds: 1800
  },
  error: InvalidClientTokenId: The security token included in the request is invalid.
      at throwDefaultError (…/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
      at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
      at de_CommandError (…/node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
      at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
      at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
      at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.base.js:108:33
      at async clientConfig (…/common/awsV3ClientConfig.js:49:23)
      at async Context.<anonymous> (…/test/unit/00.common/awsV3ClientConfig.test.js:9:22) {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 403,
      requestId: '65bbff34-7654-4353-87f6-eb315a77d533',
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Type: 'Sender',
    Code: 'InvalidClientTokenId'
  },
  metadata: {
    httpStatusCode: 403,
    requestId: '65bbff34-7654-4353-87f6-eb315a77d533',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

InvalidClientTokenId: The security token included in the request is invalid.
    at throwDefaultError (node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
    at …/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
    at de_CommandError (node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/nested-clients/dist-cjs/submodules/sts/index.js:299:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async …/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async …/node_modules/@smithy/core/dist-cjs/index.js:167:18
    at async …/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
    at async …/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async …/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.base.js:108:33
    at async clientConfig (common/awsV3ClientConfig.js:49:23)
    at async Context.<anonymous> (test/unit/00.common/awsV3ClientConfig.test.js:9:22)

As you can see, accountId still works, and the output for fromTemporaryCredentials and
the error are, mutatis mutandis, the same.

With @aws-sdk/credential-providers@3.729.0 the test succeeds. I get a token from
'us-east-1':

Using AWS credentials from `~/.aws/credentials` to assume role.
Using AWS credentials from `~/.aws/credentials to determine account id`
  AWS Account ID retrieved
@aws-sdk/credential-providers - fromTemporaryCredentials (STS)
  AWS credentials for role automated-test/automated-test-XXXXX received (session name: XXXXXXXXXX-27dcef03-1c52-4d2c-8a52-f8d97b902f69). Assuming.
{
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'AXXXXXXXXXXXX',
    secretAccessKey: 'ceXXXXXXXXXXXXXXXXXXXXXXX',
    sessionToken: 'IQoJXXXXXXXXXXXXXXXXXXXXXXXXXXXX==',
    expiration: 2025-02-21T12:25:20.000Z,
    credentialScope: undefined
  }
}

I believe that this is not in line with your theory?

@jandppw
Copy link
Author

jandppw commented Feb 21, 2025

@kuhe Is there any easy or less easy way you can suggest to have more logging? It would be interesting to see, e.g., what HTTP calls are actually finally made. There must be a difference there.

@github-actions github-actions bot removed closing-soon This issue will automatically close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. labels Feb 22, 2025
@jandppw
Copy link
Author

jandppw commented Feb 24, 2025

In a debugging session I found out that, using @aws-sdk/credential-providers@3.750.0:

node_modules/@smithy/smithy-client/dist-cjs/index.js, line 189:

request.request.headers.authorization === "AWS4-HMAC-SHA256 Credential=<YOUR_ACCESS_KEY_ID_FROM_CREDENTIALS_DEFAULT>/XXXXXX/eu-west-1/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=0e0c3fXXXXXXXXXXXXXX"

node_modules/@smithy/core/dist-cjs/index.js line 167:

identity === {
  "accessKeyId": "<YOUR_ACCESS_KEY_ID_FROM_CREDENTIALS_DEFAULT>",
  "secretAccessKey": "<YOUR_SECRET_ACCESS_KEY_FROM_CREDENTIALS_DEFAULT>",
  "$source": {
    "CREDENTIALS_PROFILE": "n",
    "CREDENTIALS_CODE": "e"
  }
}

node_modules/@aws-sdk/credential-providers/node_modules/@aws-sdk/core/dist-cjs/submodules/httpAuthSchemes/index.js, line 284, in the then:

credentials === {
  accessKeyId: "<YOUR_ACCESS_KEY_ID_FROM_CREDENTIALS_DEFAULT>",
  secretAccessKey: "<YOUR_SECRET_ACCESS_KEY_FROM_CREDENTIALS_DEFAULT>",
  sessionToken: undefined,
  $source: {
    CREDENTIALS_PROFILE: "n"
  }
}

In my ~/.aws/credentials file, the top entry is:

[default]
aws_access_key_id=<YOUR_ACCESS_KEY_ID_FROM_CREDENTIALS_DEFAULT>
aws_secret_access_key=<YOUR_SECRET_ACCESS_KEY_FROM_CREDENTIALS_DEFAULT>

In other words, the current code selects the default profile, instead of the profile that is given in the clientConfig.

@jandppw
Copy link
Author

jandppw commented Feb 24, 2025

When I hard code the profile in clientConfig to 'this-profile-does-not-exist':

  const fromTemporaryCredentialsOptions = {
    logger: console,
    params: {
      RoleArn: `arn:aws:iam::${id}:role/${rolePathAndName}`,
      // Member must have length less than or equal to 64
      RoleSessionName: roleSessionName,
      DurationSeconds: 1800
    },
    clientConfig: { region, profile: 'this-profile-does-not-exist' }
  }
  const credentialsProvider = fromTemporaryCredentials(fromTemporaryCredentialsOptions)

I get the exact same error.

In other words: the new implementation does not use the profile given in clientConfig of the parameter of fromTemporaryCredentials when looking up credentials in ~/.aws/credentials.

@jandppw
Copy link
Author

jandppw commented Feb 24, 2025

The following code, that uses STS AssumeRoleCommand directly, works around the issue in @aws-sdk/credential-providers@3.750.0:

async function clientConfig(region, profile, rolePathAndName) {
  console.info('Using AWS credentials from `~/.aws/credentials` to assume role.')
  const id = await accountId(region, profile)
  const roleSessionName = `XXXXXXXXXXXXX-${uuidv4()}`
  /** @type {import('@aws-sdk/client-sts').AssumeRoleCommandInput} */
  const assumeRoleRequest = {
    RoleArn: `arn:aws:iam::${id}:role/${rolePathAndName}`,
    // Member must have length less than or equal to 64
    RoleSessionName: roleSessionName,
    DurationSeconds: 1800
  }
  const stsClient = new STSClient({ region, profile })
  const command = new AssumeRoleCommand(assumeRoleRequest)
  
  const response = await stsClient.send(command)

  const credentials = {
	accessKeyId: response.Credentials.AccessKeyId,
	secretAccessKey: response.Credentials.SecretAccessKey,
	sessionToken: response.Credentials.SessionToken
  }
  console.info(`  AWS credentials for role ${rolePathAndName} received (session name: ${roleSessionName}). Assuming.`)

  return { region, credentials }
}

@kuhe kuhe added the queued This issues is on the AWS team's backlog label Feb 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. potential-regression Marking this issue as a potential regression to be checked by team member queued This issues is on the AWS team's backlog
Projects
None yet
Development

No branches or pull requests

3 participants