Tech: corrige des problèmes avec le soft delete des blobs openstack#13421
Merged
Conversation
LeSim
force-pushed
the
track_and_discard_purged_blob
branch
2 times, most recently
from
July 7, 2026 13:22
2050930 to
38c5798
Compare
LeSim
marked this pull request as ready for review
July 7, 2026 13:24
mfo
approved these changes
Jul 7, 2026
LeSim
force-pushed
the
track_and_discard_purged_blob
branch
2 times, most recently
from
July 8, 2026 09:08
185c9d7 to
b9a3745
Compare
LeSim
enabled auto-merge
July 8, 2026 09:09
LeSim
force-pushed
the
track_and_discard_purged_blob
branch
from
July 8, 2026 09:13
b9a3745 to
89d5847
Compare
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 8, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 9, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 13, 2026
… on openstack previously, it was base on if the whole storage_service was bind to openstack. But we are simultaneously using s3 and swift, and keeping the active_storage.service == :openstack
The Sentry call referenced an undefined `headers` variable, raising a NameError whenever copy_object returned a non-201 status. Extract the headers hash sent to copy_object and reuse it.
LeSim
force-pushed
the
track_and_discard_purged_blob
branch
from
July 13, 2026 08:01
89d5847 to
d01f1d0
Compare
The OpenStack soft delete only expires the file (X-Delete-At) and leaves the blob row in place. Stamp soft_delete_at so the leaked rows can be tracked and cleaned up later.
… job Soft-deleted blobs stay unattached with their row intact, so this job kept re-enqueuing them nightly and resetting X-Delete-At. Ignore blobs that already have soft_delete_at set.
`delete_multiple_objects` built its request body with `URI.encode`, removed in Ruby 3.0, raising NoMethodError on any non-empty list. Reopen the method to use `URI::DEFAULT_PARSER.escape` (same escaping, keeps the container/object slash literal). Canary specs fail if either Ruby or fog stops making the patch necessary.
LeSim
force-pushed
the
track_and_discard_purged_blob
branch
from
July 13, 2026 08:09
d01f1d0 to
1107820
Compare
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 13, 2026
LeSim
added a commit
that referenced
this pull request
Jul 16, 2026
Re-scans the whole blob stock by id range, targeting only openstack blobs, and hard-deletes the orphans: their variants, the parent blobs, and the corresponding files on the bucket via delete_multiple_objects. Ref #13421
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corrige plusieurs soucis :
Détection d'openstack dans delayed_purge
DelayedPurged regardait soit si la gem openstack était présente, soit si le service était openstack, pour décider si on pouvait utiliser les features openstack pour soft delete. Or nous utilisons en prod openstack + s3, le test n'était plus bon,
=> on se base maintenant directement sur le service utilisé renseigné sur le blob
Nombre de blob croissant
On ne supprime plus directement de blob : on supprime les attachments et on place un flag de suppression dans X jours sur le fichier openstack. Les blobs sont gardés en base et dit orphelins
PurgeUnattachedBlobsles reprends et les ...purge_later(retour dans DelayedPurged ...) 😿 .=> soluce :
soft_deleted_ata la table des blobs (2min de migration pour l'index sur la dev)PurgeUnattachedBlobsexclue les déjà soft deletedPurgeSoftDeletedBlobsJobqui supprimer les soft deleted blobs après le délai de retentionles variants éternels
Les variants ne sont jamais supprimés : delayed_purged utilise
service.delete_prefixed("variants/#{key}/")... hors les variants ne sont pas stockés sous la clé "variant", mais sous une entrée séparée dans la table active_storage_variants (conf track_variants)=> on mark aussi ces variants pour un soft delete
Stats:
RAF:
service.delete_prefixed("variants/#{key}/")fonctionne, possible bug de la gem voir (Ruby 3.0 support fog/fog-openstack#522 et Add partial support for ruby 3 / 3.1 fog/fog-openstack#527)PurgeSoftDeletedBlobsJobpour s'assurer qu'on a pas de fichier orphelin qui traineProchaine PR