Skip to content

Codex/assertion text diff blocks#14425

Open
hamza-mobeen wants to merge 10 commits into
pytest-dev:mainfrom
hamza-mobeen:codex/assertion-text-diff-blocks
Open

Codex/assertion text diff blocks#14425
hamza-mobeen wants to merge 10 commits into
pytest-dev:mainfrom
hamza-mobeen:codex/assertion-text-diff-blocks

Conversation

@hamza-mobeen
Copy link
Copy Markdown
Contributor

@hamza-mobeen hamza-mobeen commented Apr 27, 2026

closes #6757

Description

This PR introduces a new configuration option, assertion_text_diff_style, to allow users to customize how multiline string assertion failures are displayed.

As highlighted in #6757, when pytest uses ndiff to display differences for multiline strings (like output from capsys), it can sometimes become an unreadable "soup", especially when changes involve indentation or leading/trailing whitespace. The current behavior often results in a confusing line-by-line diff that is difficult to parse.

This PR adds an optional block diff style that displays the exact contents of the "Left" and "Right" text blocks intact and sequentially, making it significantly easier to read the literal multiline strings being compared without the interspersed ndiff noise.

Key changes:

  • Added assertion_text_diff_style configuration option. Defaults to ndiff to preserve backwards compatibility.
  • Implemented the block diff style (_diff_text_block) for multiline string comparisons, solving the readability issue raised in assertion diffs for multiline-string can become unreadable soup #6757.
  • Added fallback logic so that single-line comparisons still use ndiff even if block is selected.
  • Added strict validation to ensure assertion_text_diff_style is either ndiff or block.
  • Included extensive test coverage in testing/test_assertion.py verifying block formatting, fallback logic, blank line handling, and configuration validation.

Checklist

  • Include documentation when adding new features.
  • Include new tests or update existing tests when applicable.
  • Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.
  • Add text like closes #XYZW to the PR description and/or commits (where XYZW is the issue number).
  • If AI agents were used, they are credited in Co-authored-by commit trailers.
  • Create a new changelog file in the changelog directory, with a name like <ISSUE NUMBER>.<TYPE>.rst.
  • Add yourself to AUTHORS in alphabetical order.

@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Apr 27, 2026
@hamza-mobeen
Copy link
Copy Markdown
Contributor Author

@RonnyPfannschmidt I'm still waiting on your feedback :)

Copy link
Copy Markdown
Member

@RonnyPfannschmidt RonnyPfannschmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This. Technically looks good but is also at odds with a number of comments on the original issue

As such id like to defer to @nicoddemus

Copy link
Copy Markdown
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hamza-mobeen for the PR, overall great work!

I left some comments, please take a look.

Comment thread doc/en/how-to/output.rst
Comment on lines +366 to +368
:confval:`assertion_text_diff_style`: Controls how pytest renders ``str == str`` failures. The default ``ndiff`` output
keeps the existing inline diff markers. Setting it to ``block`` prints multiline string comparisons as separate
``Left:`` and ``Right:`` blocks, which can be easier to read when whitespace or indentation differences dominate.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's clearly present the two options:

Suggested change
:confval:`assertion_text_diff_style`: Controls how pytest renders ``str == str`` failures. The default ``ndiff`` output
keeps the existing inline diff markers. Setting it to ``block`` prints multiline string comparisons as separate
``Left:`` and ``Right:`` blocks, which can be easier to read when whitespace or indentation differences dominate.
:confval:`assertion_text_diff_style`: Controls how pytest renders ``str == str`` failures.
* ``ndiff`` (the default) outputs the differences using inline diff markers.
* ``block`` prints multiline string comparisons as separate ``Left:`` and ``Right:`` blocks, which can be easier to read when whitespace or indentation differences dominate.
Note that it is possible to set this option (as any other configuration option) directly in the command line using ``-o assertion_text_diff_style=block``.


Supported values are:

* ``ndiff``: use the default inline diff rendering.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* ``ndiff``: use the default inline diff rendering.
* ``ndiff``: use the inline diff rendering markers.

Supported values are:

* ``ndiff``: use the default inline diff rendering.
* ``block``: render multiline string comparisons as separate ``Left:`` and ``Right:`` blocks.
Copy link
Copy Markdown
Member

@nicoddemus nicoddemus May 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* ``block``: render multiline string comparisons as separate ``Left:`` and ``Right:`` blocks.
* ``block``: render each string in separate ``Left:`` and ``Right:`` blocks.

Comment on lines +226 to +227
saved_config = util._config
util._config = config
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary? Is the config being passed here different from the one already in util._config?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, given this is the only place where util.assertrepr_compare is called, consider just passing the text diff style directly to util.assertrepr_compare.

right: object,
highlighter: _HighlightFunc,
verbose: int = 0,
assertion_text_diff_style: str = ASSERTION_TEXT_DIFF_STYLE_NDIFF,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets remove this default value and force callers to always pass it; it is an internal function, so breaking the API is actually a good thing because it will ensure every caller is actually passing the right thing.

Suggested change
assertion_text_diff_style: str = ASSERTION_TEXT_DIFF_STYLE_NDIFF,
assertion_text_diff_style: str,

) -> list[str]:
if (
assertion_text_diff_style == ASSERTION_TEXT_DIFF_STYLE_BLOCK
and _is_multiline_text(left, right)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we should special case being multiline text or not; I think it is reasonable to always honor the block configuration, even for single line texts.

right: str,
highlighter: _HighlightFunc,
verbose: int,
assertion_text_diff_style: str,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of typing this as str, let's use Literal["ndiff", "block"]. This way we are more explicit, plus we can use match below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

assertion diffs for multiline-string can become unreadable soup

3 participants