Fixing Upload Failures with S3-Compatible Storage

What Happened

When trying to upload files to mdx object storage using AWS CLI v2, the following error occurred:

An error occurred (InvalidArgument) when calling the PutObject operation:
x-amz-content-sha256 must be UNSIGNED-PAYLOAD,
STREAMING-AWS4-HMAC-SHA256-PAYLOAD, or a valid sha256 value.

The failure happened consistently for both small and large files.

Root Cause

AWS CLI v2 computes a SHA256 hash of the file contents at upload time and sends it in the x-amz-content-sha256 request header as a data-integrity check.

[AWS S3 (native)]
AWS CLI → "x-amz-content-sha256: abc123def456..." → OK (accepted)

[S3-compatible storage (some implementations)]
AWS CLI → "x-amz-content-sha256: abc123def456..." → Error (rejected)

Some S3-compatible storage implementations only accept the constant values UNSIGNED-PAYLOAD or STREAMING-AWS4-HMAC-SHA256-PAYLOAD for this header, and reject an actual hash value.

Both behaviors are technically valid under the S3 specification, but the compatibility gap triggers the error.

Fix

Setting two environment variables is all it takes:

export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED

This tells AWS CLI to compute and validate checksums only when required, which prevents it from sending a hash value in a format the S3-compatible storage does not accept.

Example Usage

# Upload with the environment variables set inline
AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED \
AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED \
aws s3 cp ./myfile.tif s3://my-bucket/path/to/myfile.tif \
  --endpoint-url https://s3ds.mdx.jp

Making It Permanent

Add the following to ~/.aws/config:

[default]
request_checksum_calculation = when_required
response_checksum_validation = when_required

If you also use AWS S3 directly, it is safer to keep the profiles separate:

[default]
# For native AWS S3 (default)

[profile mdx]
request_checksum_calculation = when_required
response_checksum_validation = when_required

# Use with an explicit profile
aws s3 cp ./myfile.tif s3://my-bucket/path/ \
  --endpoint-url https://s3ds.mdx.jp \
  --profile mdx

Note: Multipart Uploads Have the Same Issue

When uploading large files (over 8 MB by default), AWS CLI automatically switches to multipart upload. Since x-amz-content-sha256 is sent for each part, the same error occurs at the UploadPart stage.

Raising multipart_threshold to avoid multipart mode is one workaround, but since the root cause is the checksum format, the environment variables above are the simpler fix.

Affected Storage Systems

This was observed with mdx object storage (s3ds.mdx.jp), but the same issue can occur with any S3-compatible storage. Known examples include:

  • mdx object storage
  • Some versions of MinIO
  • Ceph Object Gateway (RGW), depending on configuration

The issue does not occur with native AWS S3.