Skip to content

HDDS-16187. [STS] Fix Latent S3 DeleteObjects Issue #11019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChenSammi merged 2 commits into from
Aug 17, 2026
+108 −22
Merged

HDDS-16187. [STS] Fix Latent S3 DeleteObjects Issue #11019

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48643d0
fix delete-objects issue
Aug 14, 2026
fc80f77
pr review updates for chihsuan
Aug 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-sts.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1209,6 +1209,34 @@ STS session policy s3:* on * must allow ListAllMyBuckets, Create/ListBucket, and
${output} = Execute aws s3api --endpoint-url ${S3G_ENDPOINT_URL} delete-bucket --bucket ${bucket} --profile sts
Should Not Contain ${output} AccessDenied

STS session policy containing only GetObject must deny DeleteObjects
${bucket_suffix} = Generate Random String 8 [LOWER]
${bucket} = Set Variable sts-bucket-deleteobjects-${bucket_suffix}
${key_suffix} = Generate Random String 8 [LOWER]
${key} = Set Variable sts-deny-deleteobjects-${key_suffix}.txt
${local_path} = Set Variable ${TEMP_DIR}/${key}
Create File ${local_path} deleteobjects deny test content

# Create bucket and object with full STS temp-bucket role permissions.
Assume Role And Configure STS Profile perm_access_key_id=${PERMANENT_ACCESS_KEY_ID} perm_secret_key=${PERMANENT_SECRET_KEY} role_arn=${STS_TEMP_BUCKET_ROLE_ARN}
${output} = Execute aws s3api --endpoint-url ${S3G_ENDPOINT_URL} create-bucket --bucket ${bucket} --profile sts
Should Contain ${output} Location
${output} = Execute aws s3api --endpoint-url ${S3G_ENDPOINT_URL} put-object --bucket ${bucket} --key ${key} --body ${local_path} --profile sts
Should Contain ${output} "ETag"

# Restrict token to GetObject-only via session policy. DeleteObjects must return AccessDenied.
${session_policy} = Set Variable {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"s3:GetObject","Resource":"arn:aws:s3:::${bucket}/*"}]}
Assume Role And Configure STS Profile policy_json=${session_policy} perm_access_key_id=${PERMANENT_ACCESS_KEY_ID} perm_secret_key=${PERMANENT_SECRET_KEY} role_arn=${STS_TEMP_BUCKET_ROLE_ARN}
${output} = Execute And Ignore Error aws s3api --endpoint-url ${S3G_ENDPOINT_URL} delete-objects --bucket ${bucket} --delete 'Objects=[{Key=${key}}],Quiet=false' --profile sts
Run Keyword And Continue On Failure Should Contain ${output} AccessDenied

# Cleanup using a full-permission token.
Assume Role And Configure STS Profile perm_access_key_id=${PERMANENT_ACCESS_KEY_ID} perm_secret_key=${PERMANENT_SECRET_KEY} role_arn=${STS_TEMP_BUCKET_ROLE_ARN}
${output} = Execute aws s3api --endpoint-url ${S3G_ENDPOINT_URL} delete-object --bucket ${bucket} --key ${key} --profile sts
Should Not Contain ${output} AccessDenied
${output} = Execute aws s3api --endpoint-url ${S3G_ENDPOINT_URL} delete-bucket --bucket ${bucket} --profile sts
Should Not Contain ${output} AccessDenied

Revoking Permanent User Must Revoke Existing Session Token
# Create session tokens for both buckets, verify they work, then revoke permanent user secret and verify both fail.
Assume Role And Get Temporary Credentials perm_access_key_id=${PERMANENT_ACCESS_KEY_ID} perm_secret_key=${PERMANENT_SECRET_KEY} role_arn=${ICEBERG_ALL_ACCESS_ROLE_OBS_ARN}
Expand Down
16 changes: 14 additions & 2 deletions hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import org.apache.hadoop.ozone.audit.AuditEventStatus;
import org.apache.hadoop.ozone.audit.AuditMessage;
import org.apache.hadoop.ozone.audit.S3GAction;
Expand Down Expand Up @@ -351,7 +352,14 @@ public MultiDeleteResponse multiDelete(
throw newError(S3ErrorTable.MALFORMED_XML, bucketName);
}

OzoneBucket bucket = getVolume().getBucket(bucketName);
final OzoneBucket bucket;
try {
bucket = getVolume().getBucket(bucketName);
} catch (OMException ex) {
throw newError(bucketName, ex);
} catch (IOException ex) {
throw newError(S3ErrorTable.INTERNAL_ERROR, bucketName, ex);
}
MultiDeleteResponse result = new MultiDeleteResponse();
List<String> deleteKeys = new ArrayList<>();

Expand Down Expand Up @@ -380,8 +388,12 @@ public MultiDeleteResponse multiDelete(
}
getMetrics().updateDeleteKeySuccessStats(startNanos);
} catch (IOException ex) {
LOG.error("Delete key failed: {}", ex.getMessage());
getMetrics().updateDeleteKeyFailureStats(startNanos);
final OMException omEx = (OMException) HddsClientUtils.containsException(ex, OMException.class);

@chihsuan chihsuan Aug 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we only handle ACCESS_DENIED, and non-ACCESS_DENIED OM errors still return HTTP 200 with ALL/InternalError. Should we translate every contained OMException here so that errors like TOKEN_EXPIRED and BUCKET_NOT_FOUND retain their S3 response?

@fmorg-git fmorg-git Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated - fc80f77

if (omEx != null && S3ErrorTable.translateResultCode(omEx) == S3ErrorTable.ACCESS_DENIED) {
throw newError(S3ErrorTable.ACCESS_DENIED, bucketName, omEx);

@chihsuan chihsuan Aug 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This throw skips the audit block below, so rejected multi-delete requests leave no failure audit record. I wonder if we should audit the failure before propagating it.

@fmorg-git fmorg-git Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated - fc80f77

}
LOG.error("Delete key failed: {}", ex.getMessage());
result.addError(
new Error("ALL", "InternalError",
ex.getMessage()));
Expand Down