Skip to content

Prune by either int or interval for all retention policies#8775

Open
Goddesen wants to merge 22 commits into
borgbackup:masterfrom
Goddesen:prune-timely-by-interval
Open

Prune by either int or interval for all retention policies#8775
Goddesen wants to merge 22 commits into
borgbackup:masterfrom
Goddesen:prune-timely-by-interval

Conversation

@Goddesen
Copy link
Copy Markdown
Contributor

@Goddesen Goddesen commented Apr 19, 2025

This PR adds optional interval handling for all retention filter flags of the prune command, previously only available on --keep-within. E.g. prune --keep-hourly=7d will keep hourly archives for the last 7 days regardless of count. Adds --keep which acts as a combination of --keep-last and --keep-within.

Implements #8637.

I opted to make the existing flags handle both ints and intervals instead of adding new flags as there are already so many flags for this command. Simplified some prune filtering: With the default filter function now also handling intervals, prune_within is no longer needed as a special case.

I added a library to freeze time in testing, let me know if that's not wanted and I'll figure out something else. It's such a hassle to deal with timestamps relative to now() in test. The tests should be fairly comprehensive in checking both their timely filter (hourly/yearly, etc.) and the new inclusive timestamp check. I did not add new helper tests for prune_split as this function is not used anywhere other than prune_cmd and isn't really a helper.

TODOs:

  • New complicated example using intervals, showing how they overlap and interact with simple ints. Test that confirms this example (currently no test at all for overlapping intervals).
  • Docs once exact implementation is decided.

TODOs from comments:

  • Figure out default values and their interaction with the must pass at least one flag check.

Notes:

  • New retention flag --keep: Merges functionality of --keep-within and --keep-last with new int-or-interval handling.
  • --keep-last is no longer an alias of --keep-secondly and thus keeps archives made on the same second. It now fits together with --keep-within and new flag --keep.
  • Intervals now support 0 seconds. While working with int_or_interval and creating timedelta objects it seemed weird to restrict input like this. Not extremely useful unless you want to prune on archives in the same second as they were created, but it seemed logical when setting up tests to verify new --keep-last behavior.. Technically breaking, but will likely not meaningfully affect any real scenarios.
  • Timestamp comparisons for retention intervals are now inclusive on seconds. Matches (my personal) human intuition: If it is currently xx:xx:10 and there's an archive at xx:xx:05 then --keep-within 5S should cover that archive. Easy change to make when already altering the filtering logic. Technically breaking, but will likely not meaningfully affect any real scenarios.

This has been a pet peeve of mine in the pruning command for a long time. In my mind the most clean backup regime keeps all backups for a short time (allows catching small errors quickly), then hourly backups for a reasonable time (say, 7 days), then daily backups for a little longer and then finally weekly/monthly as storage permits. This was not previously possible, requiring for example --keep-within 7d --keep-hourly 168 --keep-daily 14 --keep-weekly -1 for an approximation. But keeping around 168 archives for a machine that's only running a few hours a day seems mighty excessive. So here we are :)
With this implemented my ideal retention for my primary laptop with archives every 15m looks something like --keep 3d --keep-hourly 7d --keep-daily 30d --keep-weekly -1.

@PhrozenByte
Copy link
Copy Markdown
Contributor

This is loosely related to #8715 as well.

I like the idea very much (I didn't and can't review the code, but I'm happy to help with the docs if desired). Borg's current retention policy IMHO is rather hard to understand for beginners (but very safe, because it usually keeps more than what users expect) and an interval-based approach feels more intuitive to me. However, special care is necessary to not cause unexpected behaviour, especially with frequent backups (e.g. daily), combined with overlapping rules (e.g. including all of within, daily, weekly, monthly, and yearly), and some missing backups (and thus the need to sometimes keep the "next best" to later fulfil less frequent rules). As far as I remember there were some issues with this in the early days of Borg that at least caused multiple major docs overhauls (not sure whether the behaviour actually changed substantially, but the docs definitely did because many users understood things wrong). Maybe Thomas can give some insights about the challenges back then?

Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
@Goddesen Goddesen force-pushed the prune-timely-by-interval branch from 82c9e4e to da77550 Compare April 20, 2025 08:56
Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
Copy link
Copy Markdown
Member

@ThomasWaldmann ThomasWaldmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

Some minor stuff I found...

Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
@Goddesen Goddesen force-pushed the prune-timely-by-interval branch from 64c48ed to 38fbd43 Compare April 21, 2025 13:15
@Goddesen
Copy link
Copy Markdown
Contributor Author

It seems the GHA test runner did not like the previous tzinfo=None change in helpers_test.py. Replaced with an implementation in MockArchive more alike to the one for real archives, let's see if that plays nicely.

@PhrozenByte
Copy link
Copy Markdown
Contributor

New retention flag --keep: Merges functionality of --keep-within and --keep-last with new int-or-interval handling.

I'd vote for removing --keep-within and --keep-last in favour of the new --keep option with Borg 2.0 for consistency with other --keep-* options. If this gets backported to Borg 1.x, they should be deprecated. WDYT?

Timestamp comparisons for retention intervals are now inclusive on seconds. Matches (my personal) human intuition: If it is currently xx:xx:10 and there's an archive at xx:xx:05 then --keep-within 5S should cover that archive. Easy change to make when already altering the filtering logic.

IMHO this must be consistent with #8776. Question is what we agree on. I honestly believe differently: If it is xx:xx:10 now and what to cover the last five seconds, that's exclusive the xx:xx:05 second, because I also count archives that are created at xx:xx:10 (i.e. seconds 10, 9, 8, 7, and 6). WDYT?

Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
@ThomasWaldmann
Copy link
Copy Markdown
Member

If you want to be able to backport this to 1.4-maint, you must not break compatibility in this PR, but deprecating some options that can be replaced by a better new option can be done here.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 21, 2025

Codecov Report

❌ Patch coverage is 94.80519% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.68%. Comparing base (fea52ef) to head (ea867c4).
⚠️ Report is 127 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/archiver/prune_cmd.py 94.81% 3 Missing and 4 partials ⚠️
src/borg/helpers/parseformat.py 94.44% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8775      +/-   ##
==========================================
+ Coverage   83.62%   83.68%   +0.05%     
==========================================
  Files          90       93       +3     
  Lines       15539    15776     +237     
  Branches     2337     2364      +27     
==========================================
+ Hits        12995    13202     +207     
- Misses       1806     1832      +26     
- Partials      738      742       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@Goddesen
Copy link
Copy Markdown
Contributor Author

If you want to be able to backport this to 1.4-maint, you must not break compatibility in this PR, but deprecating some options that can be replaced by a better new option can be done here.

How do you do separation of functionality like this when a breaking 2.0 change is to be backported with deprecations instead? The code might turn out very different, especially with changes introduced on master.

@Goddesen
Copy link
Copy Markdown
Contributor Author

IMHO this must be consistent with #8776. Question is what we agree on. I honestly believe differently: If it is xx:xx:10 now and what to cover the last five seconds, that's exclusive the xx:xx:05 second, because I also count archives that are created at xx:xx:10 (i.e. seconds 10, 9, 8, 7, and 6). WDYT?

I don't think #8776 is relevant, as it matches absolute timestamps based on a pattern (like how the prune period grouping functions work) and does not deal with relative intervals where this consideration is needed. I am not familiar with the code being touched in that PR but I don't think there is overlap with the work here.

I had a whole comment here written out defending the inclusive check based on comparing timestamps only using seconds-granularity. In that scenario this makes sense. Having taken a closer look I see that archive.save explicitly saves timestamps with microseconds and the second-precision timestamps I have been seeing have all been test values. Good thing I checked. In that case the change from > to >= only makes for one microsecond of difference. I'm comfortable reverting this.

@Goddesen
Copy link
Copy Markdown
Contributor Author

Comments fixed, inclusive timestamp change reverted with tests made slightly more robust. If that's it for implementation comments I'll extend the documentation with an involved example like the retention policy I have described in the PR description and write a test for it.

I am still not sure how to go about doing breaking change on 2.0 and backporting change with deprecations to 1.*. @ThomasWaldmann ?

@ThomasWaldmann
Copy link
Copy Markdown
Member

ThomasWaldmann commented Apr 24, 2025

Easiest is to backport a non-breaking change and then do the breaking change in a separate PR afterwards (and not backport that).

There are some code structure differences between master and 1.4-maint though, so even backporting the first PR might not be trivial. As 1.4.x is a stable release, we need to be very careful to not break anything there. That sometimes can mean "no backport" if the change is too big or too risky.

Comment thread src/borg/archiver/prune_cmd.py Outdated
@Goddesen
Copy link
Copy Markdown
Contributor Author

Last explicit default removed.

There are some code structure differences between master and 1.4-maint though, so even backporting the first PR might not be trivial. As 1.4.x is a stable release, we need to be very careful to not break anything there. That sometimes can mean "no backport" if the change is too big or too risky.

The changiest change introduced here is allowing intervals to be of length 0. Since this is an extension of functionality I think it's a fairly safe backport.

Am I correct in assuming I just put entries for --keep-within and --keep-last in deprecations in preprocess_args in archiver/__init__.py?

@PhrozenByte
Copy link
Copy Markdown
Contributor

In regards to backporting this to 1.4-maint and breaking changes:

I kinda feel like I accidentally caused some confusion, so: Do we even have breaking changes right now?

The only "breaking" change is that --keep-within 0 works now. Since that is undocumented anyway, I personally wouldn't consider that a breaking change. The only breaking change would be to drop --keep-within and --keep-last in favour of the new --keep. I still strongly vote for doing that. However, just for 2.0: We can safely keep them for 1.4 and mark them as deprecated. They're just aliases for --keep with this PR, but don't really change functionality, do they?

@Goddesen
Copy link
Copy Markdown
Contributor Author

Goddesen commented Apr 24, 2025

The only "breaking" change is that --keep-within 0 works now.

Ah I forgot there is one more thing: --keep-last is no longer an alias for --keep-secondly. A little more breaky than the interval change I think, but I struggle to believe this behavior is heavily relied upon in the wild? That is, unless you read the docs about secondly pruning and then decide to just use --keep-last for some reason.

But if you want, it's pretty easy to just not introduce that change now, and do it in the breaking PR instead.

@PhrozenByte
Copy link
Copy Markdown
Contributor

Ah I forgot there is one more thing: --keep-last is no longer an alias for --keep-secondly.

There's only a difference if someone actually managed to create multiple archives within the same second, right? I'd say that's similar to 0 intervals: Sure, it's a breaking change in theory, but in practice?

Anyway, I don't think we really have a problem here. If we decide on dropping --keep-last (and --keep-within for this matter) with Borg 2.x anyway, we can simply let --keep-last stay an alias for --keep-secondly (whose behaviour hasn't changed, right?) with Borg 1.4. Users that want to try the "true" and new "last" with Borg 1.x can use the new --keep. Same for 0 intervals: Adding a check and explicitly rejecting 0 intervals with just Borg 1.4 isn't that much of a big deal, is it?

Not for me to decide, but I suggest dealing with that stuff in the backport PR.

I don't think #8776 is relevant, as it matches absolute timestamps based on a pattern (like how the prune period grouping functions work) and does not deal with relative intervals where this consideration is needed.

It does: It can also match all archives within e.g. a full month relative to "now" (and added just recently, also relative to arbitrary other timestamps). However, I agree that there's no practical difference due to the microseconds scale. I didn't consider that. Yet I'd still vote for consistency. I just skimmed through @c-herz's work in #8776 and noticed that matching exclusively causes some problems there as well (minor, yet there are some). IMHO the consistency question still somewhat matters, so how about we simply decide on always matching inclusively? I'm thus also tagging @c-herz, WDYT?

@Goddesen
Copy link
Copy Markdown
Contributor Author

After some consideration I think this new retention interval should also account for the oldest rule that was implemented for the retention count flags. I'll get back to this.

@ThomasWaldmann
Copy link
Copy Markdown
Member

before doing further changes, please rebase on current master branch to get the cython workaround. otherwise, builds will fail.

@ThomasWaldmann
Copy link
Copy Markdown
Member

Guess I'ld like to merge this for next beta, can we finish this until then?

@Goddesen
Copy link
Copy Markdown
Contributor Author

I'll endeavor to get something done soon then. Factorio has been a distraction 😄

@ThomasWaldmann
Copy link
Copy Markdown
Member

btw, i dissected the borg.testsuite.helpers_test monster module into a borg.testsuite.helpers package with modules corresponding to modules in borg.helpers.

@ThomasWaldmann
Copy link
Copy Markdown
Member

ping?

shall I help here a bit by rebasing this on current master branch and resolve the conflicts?

@Goddesen Goddesen force-pushed the prune-timely-by-interval branch from a87a0d9 to dc6f878 Compare June 3, 2025 21:59
@Goddesen
Copy link
Copy Markdown
Contributor Author

If all tests pass now I think I'm done on my end and ready for review. I'll double check early this week.

@Goddesen
Copy link
Copy Markdown
Contributor Author

Goddesen commented Jun 2, 2026

The failing test does not seem to be related to my changes, so I say this is now good for new review.

The final implementation is now the simple interval version discussed earlier. I've added a new full-scale example of a rolling backup scheme using interval retention. Fixed up relevant pruning docs/help a bit such that it all feels a bit more cohesive.

For a while I had introduced a new dependency to ease testing with exact timestamps, but that is now replaced with a new flag --since off of which all interval calculations are based. All later archives than --since are ignored for pruning purposes (and thus always kept). This ensures deterministic tests with time, and also enables an advanced user to implement their own "fuzzy" intervals on top if they should wish (such as the --since $(date +%F) to start from midnight in the new help output). There might be a better flag name, but I think --keep-weekly <interval> --since <date> reads well.

The [oldest] rule had to be enabled only for the coarsest retention rule to work properly with intervals. The solution affects count retention too, but behavior should be unchanged in practice (excepting attribution to different retention rules).

One thing I think left to decide on for this PR and not a follow-up: Is this the time to deprecate --keep-within and --keep-last in favor of just --keep?

Also: Want me to update the PR description for prosperity?


Various AI assistance was used throughout, but large scale text generation only used (and subsequently refined and rewritten) for tests and docs.

@Goddesen Goddesen requested a review from ThomasWaldmann June 2, 2026 21:42
Copy link
Copy Markdown
Member

@ThomasWaldmann ThomasWaldmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite a lot of changes. Curious how users will cope with the new possibilities.

If you do another rebase onto master, the windows test should also work.

Comment thread src/borg/archiver/prune_cmd.py Outdated
candidate_archives = archives

if since is not None:
# Prefilter: Archives from _after_ the `prune_since` time are skipped entirely.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean they are KEPT, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I'll improve the wording.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test for the prefiltered archives ignored for pruning checks.

Comment on lines +374 to +377
that time window. When ``--since`` is given together with an interval
retention, the interval is measured backwards from that timestamp
instead of from the current time. See ``Date and Time`` docs for exact
INTERVAL format.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somehow I find it confusing using "since" to give a timestamp that marks the end of a period. Isn't that rather "until" usually?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's probably good points to be made for both of these. "since" matches my own intuition better, with it acting as a base timestamp off of which the retention intervals are applied. My intuition is that the intervals go backwards in time, so "since" is the common starting point and they end at arbitrary different points in the past. With the count-based retention rules also starting their count from "right now" and scanning archives in reverse chronological order I felt this was the right way to go.

Comment thread src/borg/archiver/prune_cmd.py Outdated
Comment thread src/borg/archiver/prune_cmd.py Outdated
@ThomasWaldmann
Copy link
Copy Markdown
Member

One thing I think left to decide on for this PR and not a follow-up:
Is this the time to deprecate --keep-within and --keep-last in favor of just --keep?

Yes, guess that would be good.

@Goddesen
Copy link
Copy Markdown
Contributor Author

Goddesen commented Jun 5, 2026

Is this the time to deprecate --keep-within and --keep-last in favor of just --keep?

Yes, guess that would be good.

So just outright remove them here and mark them deprecated in a backport?

@ThomasWaldmann
Copy link
Copy Markdown
Member

Deprecation:

  • I guess we won't backport a change that big to 1.4.x, so deprecating it there would be pointless as users have no alternative.
  • For borg2, bigger changes are expected anyway and people must review/change their scripts, so guess we can just change it without deprecating it first.

@PhrozenByte
Copy link
Copy Markdown
Contributor

@Goddesen, great work ❤️

Just a quick note that I haven't forgotten about this PR after all the great and productive discussions we've had. Unfortunately, I've been a bit swamped with work lately, so I haven't had the chance to review it from an user's perspective yet. I'm aiming to get to it by the middle of next week. Hold me accountable if I don't 😉 Sorry for the delay!

@Goddesen
Copy link
Copy Markdown
Contributor Author

Goddesen commented Jun 6, 2026

If you do another rebase onto master, the windows test should also work.

For the benefit of reviewers I'll hold off on rebasing until ready for merge.

@ThomasWaldmann
Copy link
Copy Markdown
Member

ThomasWaldmann commented Jun 6, 2026

Review by Claude Opus 4.8:

Took a detailed pass over the code and the discussion. Overall this is a really nice improvement to a long-standing prune limitation, and I think narrowing the scope to the exact-interval MVP (deferring absolute-time / fuzzy / quarterly intervals to follow-ups) was the right call. A few findings below — one is a genuine crash, the rest are docs/behavior nits.

Bug — uncaught TypeError when mixing an interval rule with a coarser all/-1

In _validate_prune_args, the interval-redundancy check builds interval_args from values that are either a timedelta or the int -1, then compares pairs:

for (lo_arg, lo_val), (hi_arg, hi_val) in combinations(interval_args, 2):
    if lo_val == -1 or lo_val >= hi_val:   # lo_val timedelta, hi_val int -1
        raise CommandError(...)

When a finer rule has an interval and a coarser rule is -1/all, lo_val is a timedelta and hi_val is -1, so lo_val >= hi_val evaluates timedelta >= int and raises an uncaught TypeError (full traceback instead of a clean error). Minimal repro of the comparison:

>>> from datetime import timedelta
>>> timedelta(days=7) >= -1
TypeError: '>=' not supported between instances of 'datetime.timedelta' and 'int'

This is reachable with very ordinary input — e.g. borg prune --keep-daily 7d --keep-yearly all or --keep-hourly 14d --keep-daily -1. These combinations are not redundant (a finite finer window plus unlimited coarser retention), so the intent is to allow them — instead borg crashes. The existing test_prune_warns_on_redundant_interval_flags only parametrizes the reverse orientation (finer = -1), which is short-circuited by lo_val == -1 and never reaches the bad comparison, so the gap is uncovered. I've left a suggested fix + test in a separate comment below.

Worth addressing

  1. Help text overstates the oldest-keep rule. The new help says "The oldest archive is always kept." That isn't true in interval mode: keep_oldest is gated by can_retain(oldest_archive), so an oldest archive outside the coarsest window (e.g. a 30-day-old archive under --keep-daily 7d) is pruned. The next sentence ("...should survive until the next tier's interval naturally replaces it") also contradicts "always." The previous wording ("Borg will retain the oldest archive if any rule was not otherwise able to meet its retention target") was accurate.

  2. keep_oldest now only on the coarsest rule — worth one more test. Previously every rule that missed its target could retain the oldest; now only active_rules[-1] does. The claim that the kept set is unchanged (only attribution differs) matches my spot-checks, but this is the subtlest behavioral change here and would benefit from a dedicated test for the case where the coarsest rule meets its target while a finer one does not.

  3. Help wording "specified in increasing order of coarseness." CLI order is actually irrelevant — keep_args is rebuilt in PRUNING_RULES order and the check is purely on values. The real constraint is "a finer rule's interval must be strictly smaller than any coarser rule's." Suggest rewording so it doesn't imply argument order matters.

  4. Compatibility note. Removing --keep-within/--keep-last outright (plus --keep-last no longer being second-granularity) is a hard break for existing scripts. The thread settled on "borg2 only, no backport, no deprecation," which is defensible — just make sure the changelog spells out the migration (--keep-within Xd--keep Xd, --keep-last N--keep N).

Minor / nits

  • unique_period_func: the comment "values ... MUST be ordered the same as the input timestamp" is inaccurate — the counter runs opposite to the reverse-sorted input. Harmless (only equality is used, each archive is its own group), but misleading. Also max_digits = ceil(log10(MAX_ARCHIVES)) under-counts by one for exact powers of ten (harmless since zfill only pads).
  • int_args filter uses any((arg == r.key for r in PRUNING_RULES)), which is just arg in prune_keys (already computed).
  • Error message "...may have led to undefined behavior were it allowed" reads awkwardly.
  • --since vs --until: agree with @ThomasWaldmann that --since reading as the end of the look-back window is a bit counterintuitive; a one-line help clarification would help.
  • Confirmed the interval() int→timedelta return-type change is safe: the only non-test caller was the now-removed --keep-within.

Nice work overall — happy to help verify once the validation fix lands.

if arg in prune_keys and (isinstance(val, timedelta) or val == -1)
]
for (lo_arg, lo_val), (hi_arg, hi_val) in combinations(interval_args, 2):
if lo_val == -1 or lo_val >= hi_val:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a minimal fix for the TypeError described above. -1/all should only ever be handled by the equality branch; the magnitude comparison must be skipped whenever either side is the -1 sentinel:

Suggested change
if lo_val == -1 or lo_val >= hi_val:
if lo_val == -1 or (hi_val != -1 and lo_val >= hi_val):

This keeps the intended semantics:

  • lo == -1 (finer rule unlimited) → still flagged, since it covers any coarser rule.
  • finite finer + unlimited coarser (hi == -1) → allowed, as it should be (not redundant).
  • both finite → compared as before.

And a regression test (in src/borg/testsuite/archiver/prune_cmd_test.py) covering the previously-uncovered orientation — these combos should succeed, not error:

@pytest.mark.parametrize(
    "keep_args",
    [
        ("--keep-daily=7d", "--keep-yearly=all"),
        ("--keep-daily=7d", "--keep-yearly=-1"),
        ("--keep-hourly=14d", "--keep-daily=all"),
    ],
)
def test_prune_interval_with_unlimited_coarser(archivers, request, backup_files, keep_args):
    # A finite finer interval combined with an unlimited ("all"/-1) coarser
    # count is a valid, non-redundant combination and must not raise.
    # Previously crashed with:
    #   TypeError: '>=' not supported between instances of 'datetime.timedelta' and 'int'
    archiver = request.getfixturevalue(archivers)
    cmd(archiver, "repo-create", RK_ENCRYPTION)
    dt = datetime(2023, 12, 31, tzinfo=timezone.utc)
    _create_archive_dt(archiver, backup_files, "test-1", dt)
    output = cmd(archiver, "prune", "--list", "--dry-run", "--since", dt.isoformat(), *keep_args)
    assert re.search(r"Keeping archive .*test-1", output)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ also by Claude.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha. This is an oversight after a simplification midway through. Fixing.

Comment thread src/borg/archiver/prune_cmd.py Outdated
@Goddesen
Copy link
Copy Markdown
Contributor Author

Goddesen commented Jun 6, 2026

Int vs. timedelta comparison in granularity ordering check fixed.


1. **Help text overstates the oldest-keep rule.**

Reworded.


2. **`keep_oldest` now only on the coarsest rule — worth one more test.**

With the rework this is only applicable in a very specific case, and only for new behavior: A fine-grained count-based rule is unfulfilled while a coarser-grained interval rule has an interval that doesn't cover the archive that would be kept by the fine-grained oldest rule.

Example: --keep-hourly=24 --keep-daily 7d. We have 24 archives with the last two archives both being 8 days old and sharing the same hour-period (for some reason). One of them would be kept by hourly (no. 23) and the last one would previously be kept by the oldest rule (with something like --keep-daily=1 if all daily-candidates are taken) but is now pruned since the oldest-rule is only considered for the coarsest active retention rule and daily=7d does not cover the 24th archive. However, in steady state this archive would never have survived past 7 days anyway, so it being kept doesn't change much in my eyes.

I don't think this is worth spending much more time on -- only new behavior is affected here, the existing oldest-rule rolling backup test is green, and I cannot think of any test that could challenge this assertion for any behavior from Borg 1.


3. **Help wording "specified in increasing order of coarseness."**

Reworded.


4. **Compatibility note.** Removing `--keep-within`/`--keep-last` outright (plus `--keep-last` no longer being second-granularity) is a hard break for existing scripts.

Non-issue.


* `unique_period_func`: the comment "values ... MUST be ordered the same as the input timestamp" is inaccurate 

Outdated comment after refactor. Removed that final point.


* `int_args` filter uses `any((arg == r.key for r in PRUNING_RULES))`, which is just `arg in prune_keys` (already computed).

Fixed.


* Error message "...may have led to undefined behavior were it allowed" reads awkwardly.

Not really needed; removed.


* `--since` vs `--until`: agree with @ThomasWaldmann that `--since` reading as the _end_ of the look-back window is a bit counterintuitive; a one-line help clarification would help.

I still think --since is the intuitive choice considering all pruning mechanics work in reverse chronological order. Maybe I can reword the help output slightly to aid in this understanding for the user? If the help text always describes these operations in a backward-looking manner, maybe it all reads better with "since".

Up to you.


* Confirmed the `interval()` int→`timedelta` return-type change is safe: the only non-test caller was the now-removed `--keep-within`.

All relevant tests are green, I think this is fine.

@ThomasWaldmann
Copy link
Copy Markdown
Member

E AssertionError: Invalid final state '*' (This usually indicates unmatched */**)

Seems like there is some markup issue in the prune help (guess epilog).

@Goddesen
Copy link
Copy Markdown
Contributor Author

Goddesen commented Jun 7, 2026

Seems like there is some markup issue in the prune help (guess epilog).

You're right! I assumed workflow hiccups. Fixed.

@ThomasWaldmann
Copy link
Copy Markdown
Member

ThomasWaldmann commented Jun 7, 2026

Claude review:

Did another pass on the current revision (ea867c4). The interval-vs-all validation crash is handled now (nice, with the hi_val == -1 short-circuit + the test_prune_does_not_warn_on_normal_interval_flags coverage), and the epilog wording around the oldest-archive rule and flag ordering reads accurately. Two things stood out this time — one is a safety regression worth a decision before merge.

--keep-* 0 now soft-deletes all archives instead of erroring

_validate_prune_args gates on is not None:

keep_args = {rule.key: getattr(args, rule.key) for rule in PRUNING_RULES if getattr(args, rule.key) is not None}
if len(keep_args) == 0:
    raise CommandError('At least one of the "keep", ... settings must be specified.')

On master the equivalent guard used truthiness (if not any((args.secondly, ..., args.within))), so a lone --keep-daily 0 — all retention values falsy — was rejected with "At least one ... must be specified." Now 0 is not None, so it passes the gate, and prune() returns {} for a zero count/interval:

if len(archives) == 0 or n_or_interval in (0, timedelta(0)):
    return {}

The net effect: borg prune --keep-daily 0 (or --keep 0, --keep-secondly 0, --keep-daily 0S, …) keeps nothing and marks every matching archive for deletion. test_prune_keep_int_or_interval_zero actually locks this in (single archive → pruned), so "0 keeps nothing" is intentional — but the practical hazard is the classic scripted footgun: borg prune --keep-daily "$KEEP" where $KEEP resolves to 0 (unset/default/typo) used to be a safe no-op error and now wipes the series. It's recoverable via borg undelete until borg compact, but I'd rather not rely on that.

Suggested options (your call):

  • Treat an all-zero policy the same as "nothing specified" and raise the existing CommandError (keep the master safety net); still allow 0 when at least one other rule keeps something, if that combination is even useful.
  • Or, if --keep-* 0 deleting everything is genuinely intended, require an explicit confirmation / --force-style opt-in and call it out loudly in the changelog, since it's a behavior change from a previously-erroring invocation.

--since epilog says "Count-based retention is unaffected", but it is affected

The epilog states:

The --since option ... When --since is used together with interval-based --keep-* options ..., the interval is measured backwards from the given timestamp rather than from the current time. Count-based retention is unaffected.

But the pre-filter narrows candidate_archives for all rules, including count-based ones:

if since is not None:
    for archive in archives:
        if archive.ts <= since:
            break
        keep[archive] = KeepResult(rule=PRUNE_SINCE, idx=len(keep))
    candidate_archives = archives[len(keep):]
...
for rule, n_or_interval in active_rules:
    keep |= prune(archives=candidate_archives, ...)

So --since 2024-01-01 --keep-daily 3 keeps everything newer than the timestamp plus 3 daily archives from before it — the count rule only ever sees the older subset. That matches the flag's own help ("only consider archives older than this for pruning"), which is the correct behavior; it's just the epilog sentence that's misleading. What's unaffected is the count math (the N isn't measured from a timestamp), not the candidate set. Suggest rewording to something like: "Count-based rules are applied only to the archives older than --since (newer ones are kept unconditionally)."

Minor

  • prune()'s zero guard relies on timedelta(0) in (0, timedelta(0)) — works because the tuple membership falls through to the timedelta == timedelta comparison, but it's a slightly subtle mixed-type in; n_or_interval == 0 or n_or_interval == timedelta(0) reads more obviously.
  • do_prune passes since_timestamp=(since if since is not None else datetime.now().astimezone()) to every prune() call even for count rules, where it's unused. Harmless, just slightly noisy.

The interval feature itself looks solid and the test matrix is thorough. The --keep-* 0 behavior is the one I'd want resolved before this lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants