S3 default integrity protection change - version 2.30.0

[debora-ito]

In AWS SDK for Java 2.30.0, we released changes to the S3 client that adopts new default integrity protections. For more information on default integrity behavior, please refer to the official SDK documentation. In SDK releases from this version on, clients default to enabling an additional checksum on all Put calls and enabling validation on Get calls.

You can disable default integrity protections for S3. We do not recommend this because checksums are important to S3 integrity posture. Integrity protections can be disabled by setting the config flag to WHEN_REQUIRED, or by using the related AWS shared config file settings or environment variables.

Disclaimer: the AWS SDKs and CLI are designed for usage with official AWS services. We may introduce and enable new features by default, such as these new default integrity protections, prior to them being supported or otherwise handled by third-party service implementations. You can disable the new behavior with the WHEN_REQUIRED value for the request_checksum_calculation and response_checksum_validation configuration options covered in Data Integrity Protections for Amazon S3.

[steveloughran]

What's the measured performance hit of the new option? If the cost is negligible then I'm happy -but since we've been burned by the v2 default of checksum validation tangibly hurting benchmarks, we now set .checksumValidationEnabled(false) for S3 clients by default.

As these are only CRCs they should be very low cost -correct?

[joviegas]

Yeap you are correct this should be low cost.
Please refer our Bench mark test details in #5701

[zoewangg]

To add to what John mentioned, checksum calculation was enabled by default using MD5 in Java SDK even before 2.30.0, as part of this change, we switched to CRC32 from MD5, which means the performance should be better, if anything.

[joviegas]

Workaround to skip default checksum calculations for operations that don't require them and add MD5 checksums for operations that require checksums

1. Skip Default Checksum Calculation for operation which do not require checksum

Choose one of these options:

Option A: Add to AWS profile configuration

[YOUR_DEFAULT_PROFILE]
request_checksum_calculation = when_required

Or

Option B: Set system property

import software.amazon.awssdk.core.SdkSystemSetting;
import software.amazon.awssdk.services.s3.S3Client;

public class DemoClass {

    public static void main(String[] args) {
    
        System.setProperty(SdkSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION.property(), 
                                                                        "WHEN_REQUIRED");

        S3Client s3Client = S3Client.builder().build();

        s3Client.listBuckets();
    }

Adding MD5 checksum for operations which require checksum calculation

Step 1: Add Interceptor class as below

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.Optional;

import software.amazon.awssdk.checksums.DefaultChecksumAlgorithm;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.checksums.ChecksumSpecs;
import software.amazon.awssdk.core.interceptor.Context;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
import software.amazon.awssdk.core.interceptor.SdkExecutionAttribute;
import software.amazon.awssdk.core.interceptor.SdkInternalExecutionAttribute;
import software.amazon.awssdk.core.interceptor.trait.HttpChecksum;
import software.amazon.awssdk.core.internal.util.HttpChecksumUtils;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.Header;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.utils.Md5Utils;

public class Md5RequiredOperationInterceptor implements ExecutionInterceptor {

    @Override
    public SdkHttpRequest modifyHttpRequest(Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) {
        boolean isHttpChecksumRequired = isHttpChecksumRequired(executionAttributes);
        boolean requestAlreadyHasMd5 = context.httpRequest().firstMatchingHeader(Header.CONTENT_MD5).isPresent();

        Optional<RequestBody> syncContent = context.requestBody();
        Optional<AsyncRequestBody> asyncContent = context.asyncRequestBody();

        if (!isHttpChecksumRequired || requestAlreadyHasMd5) {
            return context.httpRequest();
        }

        if (asyncContent.isPresent()) {
            throw new IllegalStateException("This operation requires a content-MD5 checksum, " +
                    "but one cannot be calculated for non-blocking content.");
        }

        if (syncContent.isPresent()) {
            try {
                String payloadMd5 = Md5Utils.md5AsBase64(syncContent.get().contentStreamProvider().newStream());
                return context.httpRequest().copy(r -> r.putHeader(Header.CONTENT_MD5, payloadMd5));
            } catch (IOException e) {
                throw new UncheckedIOException(e);
            }
        }
        return context.httpRequest();
    }

    private boolean isHttpChecksumRequired(ExecutionAttributes executionAttributes) {
        return executionAttributes.getAttribute(SdkInternalExecutionAttribute.HTTP_CHECKSUM_REQUIRED) != null
                || isMd5ChecksumRequired(executionAttributes);
    }

    public static boolean isMd5ChecksumRequired(ExecutionAttributes executionAttributes) {
        ChecksumSpecs resolvedChecksumSpecs = getResolvedChecksumSpecs(executionAttributes);
        if (resolvedChecksumSpecs == null) {
            return false;
        } else {
            return resolvedChecksumSpecs.algorithm() == null && resolvedChecksumSpecs.isRequestChecksumRequired();
        }
    }

    public static ChecksumSpecs getResolvedChecksumSpecs(ExecutionAttributes executionAttributes) {
        ChecksumSpecs checksumSpecs = executionAttributes.getAttribute(SdkExecutionAttribute.RESOLVED_CHECKSUM_SPECS);
        return checksumSpecs != null ? checksumSpecs : resolveChecksumSpecs(executionAttributes);
    }

    public static ChecksumSpecs resolveChecksumSpecs(ExecutionAttributes executionAttributes) {
        HttpChecksum httpChecksumTraitInOperation = executionAttributes.getAttribute(SdkInternalExecutionAttribute.HTTP_CHECKSUM);
        if (httpChecksumTraitInOperation == null) {
            return null;
        } else {
            boolean hasRequestValidation = httpChecksumTraitInOperation.requestValidationMode() != null;
            String requestAlgorithm = httpChecksumTraitInOperation.requestAlgorithm();
            String checksumHeaderName = requestAlgorithm != null ? HttpChecksumUtils.httpChecksumHeader(requestAlgorithm) : null;
            return ChecksumSpecs.builder().algorithmV2(DefaultChecksumAlgorithm.fromValue(requestAlgorithm)).headerName(checksumHeaderName).responseValidationAlgorithmsV2(httpChecksumTraitInOperation.responseAlgorithmsV2()).isValidationEnabled(hasRequestValidation).isRequestChecksumRequired(httpChecksumTraitInOperation.isRequestChecksumRequired()).isRequestStreaming(httpChecksumTraitInOperation.isRequestStreaming()).requestAlgorithmHeader(httpChecksumTraitInOperation.requestAlgorithmHeader()).build();
        }
    }
}

Step 2: Use this interceptor in the client builder as below

import software.amazon.awssdk.services.s3.S3Client;

public class DemoChecksumRollback {

    public static void main(String[] args) {

        Md5RequiredOperationInterceptor md5RequiredOperationInterceptor =
                new Md5RequiredOperationInterceptor();

        S3Client s3Client = S3Client.builder()
                .overrideConfiguration(override ->
                        override.addExecutionInterceptor(md5RequiredOperationInterceptor))
                .build();

        s3Client.listBuckets();
    }
}

3. Doing Both: skipping Checksum for operations that do not require checksum and adding MD5 checksum for operations that require checksum

Add profile configuration as mentioned above or set System Property as below. Add Md5RequiredOperationInterceptor as below

import software.amazon.awssdk.services.s3.S3Client;

public class DemoChecksumRollback {
    public static void main(String[] args) {

        System.setProperty(SdkSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION.property(), "WHEN_REQUIRED");
        // Md5RequiredOperationInterceptor is same as class defined in above section
        Md5RequiredOperationInterceptor md5RequiredOperationInterceptor = 
                new Md5RequiredOperationInterceptor();

        S3Client s3Client = S3Client.builder()
                .overrideConfiguration(override ->
                        override.addExecutionInterceptor(md5RequiredOperationInterceptor))
                .build();

        s3Client.listBuckets();
    }
}

Note: The above workarounds will not prevent adding Default checksums for operations which require checksum like DeleteObjects etc.

[UdayBidikar]

Would be nice if AWS puts MD5 checksum also under the control of a new system option!

[joviegas]

Would be nice if AWS puts MD5 checksum also under the control of a new system option!

Thank you for the feedback. Your suggestions have been noted and forwarded to the appropriate internal teams.

[steveloughran]

Is there a single switch in the client builder to say "no request checksums?"

As that is all we should need to restore behaviour such that the SDK will work with third pary stores again?

Everyone seems to

• iceberg: do it for delete/put requests: S3: Disable strong integrity checksums apache/iceberg#12264
• trinio: do it for delete requests: Disable strong integrity checks for S3 FS  trinodb/trino#24954

these both force in a different signer for these requests.

Is that the only way to do it? I really want to decouple checksums from the signing, as our code supports people providing their own signers, and I don't want their code to break from us adding a workaround to what is effectively a regression "work with other stores" in the SDK".

[zoewangg]

There is no single switch to disable checksums altogether; you can only disable checksums for operations that do not require checksums such as GetObject and PutObject, and it will always be enabled for operations that require it (as defined in service model).