Medusa backup to GCS with customer managed encryption keys (CMEK) enabled fails

[Avi-Walerius]

Medusa backups to google cloud storage (GCS) fail when customer managed encryption keys (CMEK) are enabled on the GCS bucket.
ref - https://docs.cloud.google.com/kms/docs/cmek
The problem was identified in a k8ssandra-operator environment.

In the medusa.log, the backup part succeeds to send the files e.g.:

2026-08-04T09:06:35.552052061Z	[2026-08-04 09:06:35,551] INFO: Performing ASYNC backup medusa-04082026 (type=1)
...
2026-08-04T09:44:18.952785183Z	[2026-08-04 09:44:18,952] INFO: Backup done
2026-08-04T09:44:18.952836465Z	[2026-08-04 09:44:18,952] INFO: - Started: 2026-08-04 09:06:35
2026-08-04T09:44:18.952842928Z	                        - Started extracting data: 2026-08-04 09:06:38
2026-08-04T09:44:18.952848232Z	                        - Finished: 2026-08-04 09:44:18
2026-08-04T09:44:18.952854494Z	[2026-08-04 09:44:18,952] INFO: - Real duration: 0:37:40.733003 (excludes time waiting for other nodes)
2026-08-04T09:44:18.956190634Z	[2026-08-04 09:44:18,956] INFO: - 2320 files, 67.54 GB
2026-08-04T09:44:18.956219817Z	[2026-08-04 09:44:18,956] INFO: - 2320 files copied from host (2320 new, 0 reuploaded)

But later, we see this error:

2026-08-04T10:38:59.577719285Z	[2026-08-04 10:38:59,576] ERROR: Unexpected [AttributeError] raised by servicer method [/Medusa/GetBackups]
2026-08-04T10:38:59.577735418Z	Traceback (most recent call last):
  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
2026-08-04T10:38:59.577753376Z	             ^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577773179Z	  File "/home/cassandra/medusa/storage/abstract_storage.py", line 82, in list_blobs
2026-08-04T10:38:59.577776340Z	    objects = loop.run_until_complete(self._list_blobs(prefix))
2026-08-04T10:38:59.577779413Z	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577781727Z	  File "/usr/lib64/python3.12/asyncio/base_events.py", line 691, in run_until_complete
2026-08-04T10:38:59.577784095Z	    return future.result()
2026-08-04T10:38:59.577786395Z	           ^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577788971Z	  File "/home/cassandra/medusa/storage/google_storage.py", line 102, in _list_blobs
2026-08-04T10:38:59.577791471Z	    o['md5Hash'],
2026-08-04T10:38:59.577793737Z	    ~^^^^^^^^^^^
2026-08-04T10:38:59.577796054Z	KeyError: 'md5Hash'

2026-08-04T10:38:59.577800620Z	The above exception was the direct cause of the following exception:
2026-08-04T10:38:59.577802638Z	
2026-08-04T10:38:59.577804812Z	Traceback (most recent call last):
  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
2026-08-04T10:38:59.577811801Z	             ^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577814240Z	  File "/home/cassandra/medusa/storage/abstract_storage.py", line 73, in list_objects
2026-08-04T10:38:59.577816493Z	    objects = self.list_blobs(prefix=path)
2026-08-04T10:38:59.577819124Z	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577824853Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
2026-08-04T10:38:59.577827773Z	    return copy(f, *args, **kw)
2026-08-04T10:38:59.577830160Z	           ^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577833066Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
2026-08-04T10:38:59.577835868Z	    do = self.iter(retry_state=retry_state)
2026-08-04T10:38:59.577838204Z	         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577842273Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
2026-08-04T10:38:59.577844874Z	    result = action(retry_state)
2026-08-04T10:38:59.577847078Z	             ^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577849461Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 421, in exc_check
2026-08-04T10:38:59.577851825Z	    raise retry_exc from fut.exception()
2026-08-04T10:38:59.577853999Z	tenacity.RetryError: RetryError[<Future at 0x7a7a78625760 state=finished raised KeyError>]

2026-08-04T10:38:59.577858275Z	The above exception was the direct cause of the following exception:
2026-08-04T10:38:59.577860326Z	
2026-08-04T10:38:59.577868733Z	Traceback (most recent call last):
  File "/home/cassandra/medusa/service/grpc/server.py", line 267, in GetBackups
    backups = get_backups(connected_storage, self.config, True)
2026-08-04T10:38:59.577874178Z	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577875959Z	  File "/home/cassandra/medusa/listing.py", line 27, in get_backups
2026-08-04T10:38:59.577877648Z	    cluster_backups = sorted(
2026-08-04T10:38:59.577879381Z	                      ^^^^^^^
2026-08-04T10:38:59.577881201Z	  File "/home/cassandra/medusa/storage/__init__.py", line 363, in list_cluster_backups
2026-08-04T10:38:59.577882940Z	    node_backups = sorted(
2026-08-04T10:38:59.577884708Z	                   ^^^^^^^
2026-08-04T10:38:59.577886470Z	  File "/home/cassandra/medusa/storage/__init__.py", line 184, in list_node_backups
2026-08-04T10:38:59.577888457Z	    backup_index_blobs = self.list_backup_index_blobs()
2026-08-04T10:38:59.577890286Z	                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577892088Z	  File "/home/cassandra/medusa/storage/__init__.py", line 275, in list_backup_index_blobs
2026-08-04T10:38:59.577894298Z	    return self.storage_driver.list_objects(path)
2026-08-04T10:38:59.577896042Z	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577897894Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
2026-08-04T10:38:59.577899648Z	    return copy(f, *args, **kw)
2026-08-04T10:38:59.577901329Z	           ^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577903103Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
2026-08-04T10:38:59.577904825Z	    do = self.iter(retry_state=retry_state)
2026-08-04T10:38:59.577906600Z	         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577908495Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
2026-08-04T10:38:59.577911564Z	    result = action(retry_state)
2026-08-04T10:38:59.577913270Z	             ^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577915523Z	  File "/home/cassandra/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 421, in exc_check
2026-08-04T10:38:59.577917335Z	    raise retry_exc from fut.exception()
2026-08-04T10:38:59.577919922Z	tenacity.RetryError: RetryError[<Future at 0x7a7a787ce060 state=finished raised RetryError>]

2026-08-04T10:38:59.577925223Z	During handling of the above exception, another exception occurred:
2026-08-04T10:38:59.577927798Z	
2026-08-04T10:38:59.577930635Z	Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 689, in grpc._cython.cygrpc._handle_exceptions
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 831, in _handle_rpc
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 554, in _handle_unary_unary_rpc
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 415, in _finish_handler_with_unary_response
  File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
2026-08-04T10:38:59.577988043Z	             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-04T10:38:59.577990471Z	  File "/home/cassandra/medusa/service/grpc/server.py", line 276, in GetBackups
2026-08-04T10:38:59.577992703Z	    response.status = medusa_pb2.StatusType.UNKNOWN
2026-08-04T10:38:59.577998639Z	    ^^^^^^^^^^^^^^^
2026-08-04T10:38:59.578000530Z	AttributeError: Protocol message GetBackupsResponse has no "status" field.

K8ssandra-operator log has:

026-08-04T10:38:59.578682002Z	2026-08-04T10:38:59.578Z	ERROR	failed to list backups	{"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"medusa-04082026","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "medusa-04082026", "reconcileID": "e4a192bf-8a93-486e-a269-df2d7c862f46", "medusabackupjob": {"name":"medusa-04082026","namespace":"k8ssandra-operator"}, "CassandraPod": "cassandra-dev-dc1-r1-sts-0", "error": "failed to get backups: rpc error: code = Internal desc = Unexpected <class 'AttributeError'>: Protocol message GetBackupsResponse has no \"status\" field."}
2026-08-04T10:38:59.578719710Z	github.com/k8ssandra/k8ssandra-operator/controllers/medusa.(*MedusaBackupJobReconciler).getBackupSummary
2026-08-04T10:38:59.578724388Z		/workspace/controllers/medusa/medusabackupjob_controller.go:246
2026-08-04T10:38:59.578727122Z	github.com/k8ssandra/k8ssandra-operator/controllers/medusa.(*MedusaBackupJobReconciler).Reconcile
2026-08-04T10:38:59.578731352Z		/workspace/controllers/medusa/medusabackupjob_controller.go:191
2026-08-04T10:38:59.578734720Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-08-04T10:38:59.578737480Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216
2026-08-04T10:38:59.578759194Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-08-04T10:38:59.578762132Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461
2026-08-04T10:38:59.578764657Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-08-04T10:38:59.578768567Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421
2026-08-04T10:38:59.578770957Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
2026-08-04T10:38:59.578773521Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296
2026-08-04T10:38:59.578780154Z	2026-08-04T10:38:59.578Z	ERROR	Failed to get backup summary	{"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"medusa-04082026","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "medusa-04082026", "reconcileID": "e4a192bf-8a93-486e-a269-df2d7c862f46", "medusabackupjob": {"name":"medusa-04082026","namespace":"k8ssandra-operator"}, "error": "failed to get backups: rpc error: code = Internal desc = Unexpected <class 'AttributeError'>: Protocol message GetBackupsResponse has no \"status\" field."}
2026-08-04T10:38:59.578782504Z	github.com/k8ssandra/k8ssandra-operator/controllers/medusa.(*MedusaBackupJobReconciler).Reconcile
2026-08-04T10:38:59.578784804Z		/workspace/controllers/medusa/medusabackupjob_controller.go:193
2026-08-04T10:38:59.578787Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-08-04T10:38:59.578789245Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:216
2026-08-04T10:38:59.578791389Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-08-04T10:38:59.578793602Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:461
2026-08-04T10:38:59.578795809Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-08-04T10:38:59.578798047Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421
2026-08-04T10:38:59.578800262Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
2026-08-04T10:38:59.578813043Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296
2026-08-04T10:38:59.578817688Z	2026-08-04T10:38:59.578Z	ERROR	Reconciler error	{"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"medusa-04082026","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "medusa-04082026", "reconcileID": "e4a192bf-8a93-486e-a269-df2d7c862f46", "error": "failed to get backups: rpc error: code = Internal desc = Unexpected <class 'AttributeError'>: Protocol message GetBackupsResponse has no \"status\" field."}
2026-08-04T10:38:59.578819727Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-08-04T10:38:59.578821486Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:474
2026-08-04T10:38:59.578823247Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-08-04T10:38:59.578835266Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:421
2026-08-04T10:38:59.578837553Z	sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
2026-08-04T10:38:59.578839388Z		/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.5/pkg/internal/controller/controller.go:296

According to:
https://docs.cloud.google.com/storage/docs/json_api/v1/objects/list
https://docs.cloud.google.com/storage/docs/json_api/v1/objects#resource

When CMEK is used, GCS api list does not include the md5hash, which I think the cause of the medusa error:
"MD5 hash of the data, encoded using base64. This field is not present for composite objects and objects encrypted with customer-managed encryption keys (CMEK). For more information about using the MD5 hash, see Data validation."

In k8ssandra-operator environment the result is that the medusabackup object is never created, which means the backup is not fully complete from k8ssandra-operator perspective and can not be restored.

Im not sure if this can be fixed only at medusa level or needs something at k8ssandra-operator as well, so opening here for now.

[rzvoncek]

I checked with Avi and we conculded that the easies fix for this is a medusa patch that will allow it to degrade the checks (of a local file and a file in remote storage) to be size only, not size and hash.

[rzvoncek]

Plan: Fix GCS CMEK Missing md5Hash

Overview

When a GCS bucket has Customer-Managed Encryption Keys (CMEK) enabled, the GCS JSON API does not
include the md5Hash field in object list/stat responses. Medusa's GoogleStorage._list_blobs
and _stat_blob unconditionally access o['md5Hash'], raising a KeyError that crashes the
GetBackups gRPC call. The net result in k8ssandra-operator is that the MedusaBackup object is
never created even though the files were physically uploaded.

The fix is entirely within Medusa — no k8ssandra-operator changes are needed. The goal is to
gracefully handle a missing md5Hash field everywhere it is read from the GCS API response,
replacing it with None (falling back to size-only comparison during verify/restore) and
logging a one-time INFO message so operators know CMEK is in effect.

---

Sub-Tasks

---

Sub-Task 1 — Handle missing md5Hash in _list_blobs

Intent
The immediate crash site. _list_blobs uses a list-comprehension over paginated GCS objects and
directly subscripts o['md5Hash']. When CMEK is active the field is absent and a KeyError is
raised, which bubbles up as a tenacity RetryError and ultimately kills the GetBackups RPC.

Expected Outcomes

• _list_blobs no longer raises KeyError when md5Hash is absent.
• AbstractBlob.hash is set to None for CMEK-encrypted objects.
• An INFO-level log message is emitted (once, not once-per-object) when at least one object is
  missing its md5Hash, so operators are aware.

Todo List

1. In medusa/storage/google_storage.py, change the
   _list_blobs list-comprehension to use o.get('md5Hash') instead of o['md5Hash'].
2. After building the list, if any blob has hash is None, emit a single logging.info
   message explaining that md5Hash is absent (likely CMEK) and size-only comparison will be
   used.

Relevant Context

• medusa/storage/google_storage.py lines 105–119 —
  _list_blobs method.
• medusa/storage/abstract_storage.py line 39 —
  AbstractBlob namedtuple; the hash field is already typed as nullable (local storage sets
  it to None).

Status — [ ] pending

---

Sub-Task 2 — Handle missing md5Hash in _stat_blob and _upload_object / _upload_blob

Intent
_stat_blob (used by _download_blob and _get_object) also subscripts blob['md5Hash']
directly. Similarly, _upload_object and _upload_blob extract resp['md5Hash'] from the
upload response — GCS documentation does not guarantee md5Hash in the response when CMEK is
active. These must be made safe with the same .get() pattern.

Expected Outcomes

• _stat_blob returns an AbstractBlob with hash=None when md5Hash is absent.
• _upload_object returns an AbstractBlob with hash=None when md5Hash is absent.
• _upload_blob returns a ManifestObject with MD5=None when md5Hash is absent.
• No regressions in the normal (non-CMEK) upload/download flow.

Todo List

1. In medusa/storage/google_storage.py line 230 (_stat_blob)
   — change blob['md5Hash'] to blob.get('md5Hash').
2. In medusa/storage/google_storage.py line 176
   (_upload_object) — change resp['md5Hash'] to resp.get('md5Hash').
3. In medusa/storage/google_storage.py line 274
   (_upload_blob) — change resp['md5Hash'] to resp.get('md5Hash').

Relevant Context

• _stat_blob at lines 220–234.
• _upload_object at lines 152–177.
• _upload_blob at lines 236–275.
• ManifestObject.MD5 is already handled as nullable by compare_with_manifest (which returns
  size-only result when actual_hash is falsy — line 328).

Status — [ ] pending

---

Sub-Task 3 — Guard compare_with_manifest against None hash_in_manifest

Intent
compare_with_manifest (used during verify/restore) calls base64.b64decode(hash_in_manifest)
on lines 331–332. If hash_in_manifest is None (because the manifest was written with a
None MD5 under CMEK), this will raise a TypeError. The existing short-circuit (if not actual_hash: return sizes_match) only skips when actual_hash is falsy — it doesn't protect
the hash_in_manifest side. A symmetric guard is needed.

Expected Outcomes

• compare_with_manifest returns a size-only comparison result when either actual_hash OR
  hash_in_manifest is None/falsy.
• No change in behaviour for normal (non-CMEK) comparisons.

Todo List

1. In medusa/storage/google_storage.py lines 325–337
   (compare_with_manifest) — extend the early-return guard to also cover hash_in_manifest
   being falsy: if not actual_hash or not hash_in_manifest: return sizes_match.

Relevant Context

• compare_with_manifest at lines 325–337.
• Callers: blob_matches_manifest (line 307) and file_matches_storage (line 315).

Status — [ ] pending

---

Sub-Task 4 — Add unit tests for CMEK / missing md5Hash scenarios

Intent
Ensure the new behaviour is covered by automated tests so that future changes cannot silently
re-introduce the bug.

Expected Outcomes

• A test in tests/storage/google_storage_test.py
  verifies that _list_blobs correctly handles GCS object responses that have no md5Hash
  key (simulating CMEK), producing AbstractBlob objects with hash=None.
• A test verifies that compare_with_manifest returns a size-based result (True when sizes
  match) when hash_in_manifest is None.
• All existing tests continue to pass.

Todo List

1. Add a test method test_list_blobs_cmek_missing_md5hash to GoogleStorageTest that mocks
   _paginate_objects to yield a minimal GCS object dict without md5Hash and asserts that
   the returned AbstractBlob has hash=None.
2. Add a test method test_compare_with_manifest_none_hash_in_manifest to GoogleStorageTest
   that calls GoogleStorage.compare_with_manifest with matching sizes and
   hash_in_manifest=None and asserts True is returned.
3. Add a test method test_compare_with_manifest_none_actual_hash (covering the existing
   guard, for completeness) similarly.

Relevant Context

• Existing test patterns in tests/storage/google_storage_test.py
  — specifically _make_gcs_storage helper and the mock-based tests.
• _paginate_objects is an async for-based generator; it can be mocked as an AsyncMock
  that returns an async iterable.

Status — [ ] pending

---

Files Changed

File	Change
medusa/storage/google_storage.py	Use .get('md5Hash') in _list_blobs, _stat_blob, _upload_object, _upload_blob; guard compare_with_manifest
tests/storage/google_storage_test.py	Add unit tests for CMEK missing-hash scenarios