Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(middleware-flexible-checksums): buffer stream chunks to minimum required size #6882

Merged
merged 1 commit into from
Feb 17, 2025

Conversation

kuhe
Copy link
Contributor

@kuhe kuhe commented Feb 13, 2025

helps with #6859

Attempts to buffer input stream chunks to a minimum required size as required by S3, as described in https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html.

This is for input streams e.g. S3 PUT Object or Part.

With checksumming on, user streams go through a filter called AWS chunked encoding. This process annotates every stream chunk with its size and thus the chunks are sent to S3 as-is in whatever size the user provides them, instead of Node.js being able to automatically buffer smaller chunks.

S3 has always required a chunk size of 8kb, preferably 64kb, but due to automatic buffering this was rarely an issue prior to the introduction of default checksums.

As a mitigation / convenience feature, the JS SDK can automatically buffer streams for the user before undergoing chunked encoding. This incurs a small performance penalty that can be avoided by the user if they provide streams with larger chunks.

@kuhe kuhe requested a review from a team as a code owner February 13, 2025 18:54
@kuhe kuhe marked this pull request as draft February 13, 2025 18:54
@kuhe kuhe force-pushed the fix/flex-check branch 4 times, most recently from 1bbab18 to 7136583 Compare February 14, 2025 18:10
@kuhe kuhe marked this pull request as ready for review February 14, 2025 18:10
@kuhe kuhe force-pushed the fix/flex-check branch 3 times, most recently from 91533e2 to 0c38028 Compare February 17, 2025 16:12
@kuhe
Copy link
Contributor Author

kuhe commented Feb 17, 2025

  • unit
  • integ
  • e2e
  • cucumber (s3 tag especially)

@kuhe kuhe merged commit 1f1905e into aws:main Feb 17, 2025
4 checks passed
@kuhe kuhe deleted the fix/flex-check branch February 17, 2025 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants