Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot use S3 endpoint in cn-north-1 region #5836

Open
liuweidadada opened this issue Mar 3, 2025 · 3 comments · May be fixed by #5840
Open

Cannot use S3 endpoint in cn-north-1 region #5836

liuweidadada opened this issue Mar 3, 2025 · 3 comments · May be fixed by #5840

Comments

@liuweidadada
Copy link

Bug report

When use region cn-north-1 and vpc endpoint for s3 in China, nextflow will use us-east-1 as aws client region not cn-north-1 configured

Expected behavior and actual behavior

Excepcted used the AWS_S3_ENDPOINT and AWS_REGION configured
Actual behavior is AWS_S3_ENDPOINT and us-east-1(default)

Steps to reproduce the problem

use S3 vpce in cn-north-1 like xxxx.s3.cn-north-1.vpce.amazonaws.com.cn

Program output

with an explicit deny in a resource-based policy (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

Problem source code

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsS3Config.groovy

    boolean isCustomEndpoint() {
        endpoint && !endpoint.contains(".amazonaws.com")
    }

In cn-north-1 region endpoint looks like 'xxxx.s3.cn-north-1.vpce.amazonaws.com.cn'
isCustomEndpoint will return false ,but actual it need return true

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java

		final String bucketName = S3Path.bucketName(uri);
		final boolean global = bucketName!=null;
		final AwsClientFactory factory = new AwsClientFactory(awsConfig, globalRegion(awsConfig));
		client = new S3Client(factory.getS3Client(clientConfig, global));

In cn-north-1 factory.getS3Client(clientConfig, global) global should be false, but in code "bucketName!=null" global must be true.

@pditommaso pditommaso linked a pull request Mar 3, 2025 that will close this issue
@pditommaso
Copy link
Member

May also be related to #5671

@liuweidadada
Copy link
Author

Thank you for resolving s3 endpoint's issue.
The "global" problem is another sssues that impede use.
I suggest that the following changes can be made.
nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3FileSystemProvider.java

                S3Client client;
		ClientConfiguration clientConfig = createClientConfig(props);

		//final String bucketName = S3Path.bucketName(uri);
		final boolean global =  !awsConfig.getRegion().startsWith("cn-");
		final AwsClientFactory factory = new AwsClientFactory(awsConfig, globalRegion(awsConfig));
		client = new S3Client(factory.getS3Client(clientConfig, global));

@pditommaso
Copy link
Member

This should be enough #5840

When using a custom endpoint (including AWS china) the global flag is always false

final boolean global = bucketName!=null && !awsConfig.getS3Config().isCustomEndpoint();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants