Question 1

Why does RustFS generate constant network traffic and CPU load on an idle cluster with no client requests?

Accepted Answer

This is caused by the tier free-version recovery scanner: even when no remote tier is configured (`admin/v3/tier` returns `[]`), the elected recovery leader recursively walks every bucket from the root, streaming `xl.meta` (including inline object data) from all peers. Symptoms: the leader node receives 10-15 MB/s from peers and uses ~1 CPU core, with repeated `GET /rustfs/rpc/walk_dir` calls (`recursive: true`, `base_dir: ""`) for each bucket. The load follows the `tier-free-version-recovery/leader.lock` election and moves to another node after pod restarts.

Workaround (verified on RustFS 1.0.3): disable the recovery sweep by setting the environment variable:

```
RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED=false
```

Measured result on the leader node: inbound traffic dropped from 9.3 MB/s to 0.28 MB/s and rustfs CPU from 0.77 to 0.05 cores. Note: `RUSTFS_HEAL_OBJECT_SELECT_PROB=0` does NOT stop this load, as it only affects healing. Only re-enable the sweep if you actually configure remote tiers, since free versions cannot exist without them.

Question 2

How to fix "invalid key name: 's3:ObjectCreationOperation'" error when applying an S3 bucket policy in RustFS?

Accepted Answer

This error occurs because RustFS versions before PR #8671 do not recognize the AWS condition key `s3:ObjectCreationOperation`, so any policy containing it fails to parse. Upgrade RustFS to a build that includes PR #8671, which adds the key as a server-derived Bool value on every `s3:PutObject` authorization: `false` for multipart staging operations (CreateMultipartUpload, UploadPart, UploadPartCopy) and `true` for actual object writes (PutObject, CopyObject, CompleteMultipartUpload, POST Object). Because it is server-derived, no request header or identity claim can supply or override it. After upgrading you can enforce conditional writes (`If-None-Match: *`, i.e. create-only, no overwrite) while still allowing multipart uploads, using the AWS-documented policy shape:

[
  {
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::bucket/*",
    "Condition": { "Null": { "s3:if-none-match": "false" } }
  },
  {
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::bucket/*",
    "Condition": { "Bool": { "s3:ObjectCreationOperation": "false" } }
  }
]

Note: the full conditional-write example also requires the `s3:if-none-match` condition key, which is added separately via PR #8615.

Question 3

Why does DeleteBucket fail with BucketNotEmpty after a successful Multi-Object Delete (DeleteObjects) on a versioned bucket?

Accepted Answer

Root cause: a bug where `DeleteObjects` with an explicit `VersionId` targeting a **delete marker** treated the removal as a marker write instead of a version deletion. The API returned success, but the marker persisted — and delete markers count as content, so `DeleteBucket` fails with `The bucket you tried to delete is not empty`. The bug triggers when (a) one batch contains multiple delete-marker versions of the same key (markers after the first are silently skipped), or (b) pending version-delete replication keeps the marker on the source.

**Fix:** Upgrade to a RustFS build that includes PR #8854, which routes explicit marker `VersionId` removals through the normal version-deletion path per the AWS S3 `DeleteObjects` spec. Regression tests (immediate listing + bucket deletion, duplicate IDs, crash recovery) pass after the fix.

**Workarounds on unpatched versions:**
1. Re-send the delete: retry `POST /bucket?delete` with the marker's `VersionId`, or issue single `DELETE /bucket/object?versionId=<marker-id>` requests.
2. Poll `ListObjectVersions` until no delete markers remain (markers may disappear after a delay due to async cleanup), then call `DeleteBucket`.
3. Avoid batches with duplicate delete-marker versions for the same key — split them into separate `DeleteObjects` calls.
4. On replicated setups, wait for version-delete replication to complete before deleting the bucket.

Question 4

Does DeleteObjects support S3 conditional deletes with per-object ETag matching (PreconditionFailed)?

Accepted Answer

Yes, ETag-based conditional bulk deletes for the S3 DeleteObjects API were implemented and merged into the main branch via PR #8535 on October 15. However, this feature is NOT included in the 1.0.4 stable release, which does not contain all changes from main. To use it now, build or deploy from the main branch. The maintainers plan to ship it in the next stable release (after 1.0.4) following a testing round. When available, objects are deleted only if their ETag matches the condition; otherwise the API returns PreconditionFailed, matching AWS S3 conditional delete behavior and enabling safe concurrent deletion.

Question 5

How to fix 'row-level snapshot operation requires deleted data files or added delete files' error when running Iceberg rewrite_manifests via REST catalog?

Accepted Answer

This is a server-side bug in the REST catalog v1.0.1-preview.5, which wrongly validates rewrite_manifests commits as row-level operations and rejects them. Fix: upgrade the catalog server to v1.0.2, where `CALL catalog.system.rewrite_manifests(table => 'db.table')` succeeds. Avoid v1.0.3 — a regression makes rewrite_manifests fail again with 'manifest referenced data file is missing' even when no data changed; stay on v1.0.2 if you rely on this procedure. As an alternative, use another REST catalog implementation such as Lakekeeper, which does not have this commit-validation bug.

rustfs

Core Features

What It Can't Do

Use Cases

Tags

Getting Started

Similar Projects