Terminal fills new line with background color when scrolling up

[Dmitry Sapozhnikov (o-sdn-o)]

Past discussions (discussions I didn't know about when I opened this thread)

• https://bugzilla.gnome.org/show_bug.cgi?id=754596
• https://stackoverflow.com/questions/32769876/sgr-reset-code-is-ignored-if-n-is-placed-on-a-new-line
• https://invisible-island.net/ncurses/ncurses.faq.html#bce_mismatches
• https://unix.stackexchange.com/questions/212933/background-color-whitespace-when-end-of-the-terminal-reached
• ansi terminal code reset breaks after scrolling #10556
• https://superuser.com/questions/1205861/gnome-terminal-vim-colors-not-updating
• Rendering issue with PowerShell Write-Host -BackgroundColor #19736

Context #19736 (comment)

This is a continuation of the discussion #19736

The default "... background color used when ... new text is scroll on to the screen" is documented at 5-48 of the DEC VT520 programmer's manual. It is also described in 2-32. (PDF link).
2-32

5-48

This default is in effect in any compliant terminal even if it does not support changing it with DECECM.

Following this rule will result in random side effects when displaying text with the specified background.

Example:

• With implicit line breaks, there is unavoidable side effect:
  

At this point, it's clear what side effects arise from following this rule and I'd like to discuss here what "breaks" in the terminal world if this rule is not followed—not implicitly filling the entire new line at the bottom of the viewport when scrolling text up during line feed.

[James Holderness (j4james)]

It breaks any scenario in which you actually want the background semi-permanently set to a different color. As one example, imagine a scrolling game where the background color is something other than the default. It may change as you progress between zones, but you're always going to want it scrolling with the active background color.

To get an idea of what I mean, paste something like this in powershell:

"`e[41m`e[2J"

Then press enter a few times until the screen starts to scroll. If the background color erase wasn't in effect, you'd suddenly find your background color changing back to the default as soon as the viewport started to scroll.

Nobody is disputing that there are scenarios when this behavior is not ideal, but that's precisely what the DECECM mode is for. If you're generating a log file that injects temporary background color changes in the output, you simply need to set that mode at the start and end of the log.

[Dmitry Sapozhnikov (o-sdn-o)]

Indeed, completely disabling newline fills isn't an option. If I understand correctly, the idea proposed in the gnome-terminal discussion solves this by disabling BCE only on implicit line breaks.

Comment 22 Egmont Koblinger 2017-04-21 20:22:59 UTC

Created attachment 350209 [details] [review]
bce followup fix

Out of the ideas presented above, this simple patch implements probably (hopefully) the simplest and the best. bce is respected and the new line is filled with the current background color if scrolling occurs because of an explicit escape sequence, but is still disregarded when the new line appears due to implicit autowrapping at the end of a line.

[Dustin L. Howett (DHowett)]

There was a discussion over in #10556 about this as well. To quote James Holderness (@j4james) (though I recommend reading through the rest of the thread; it has a similar trajectory to this one!)

Gnome Terminal (VTE) I know is different. A couple years back they decided they didn't like the way scrolling worked and figured they'd just change the "standard" behaviour (you can read more here). It's possible there are other terminals that have since copied them, but I suspect they are still in the minority, so I don't think we're likely to change unless XTerm does too (although I can't speak for the core devs - maybe they'll disagree).

The linked gnome-terminal bug has some fascinating discussion in it as well... including this snippet (from Egmont Koblinger) indicating a thing that did actually break in production.

Houston...

We broke the "le" text editor. Just open any file, and scroll down with the Down arrow.

It uses ncursesw, so unfortunately chances are that other ncurses-based apps are broken too.

It switches to alternate screen (no surprise from ncurses), and sets up restricted scrolling for the first N-1 lines (e.g. "\e[1;23r") while scrolling the contents upwards.

What could we do?

1. Try to convince Thomas to modify ncurses to explicitly set the background color of those lines

2. Disable this new behavior on the alternate screen

3. Disable this new behavior if restricted scrolling is in effect

4. Revert this new behavior entirely (I wouldn't like to, since it fixes many use cases)

[Dmitry Sapozhnikov (o-sdn-o)]

Comment 19 Egmont Koblinger 2016-03-31 21:10:50 UTC

Almost forgot I already had this idea in comment 7:

5. Legacy behavior on expicit newline, new behavior on implicit overflow to the next line.

Comment 22 Egmont Koblinger 2017-04-21 20:22:59 UTC

...
Out of the ideas presented above, this simple patch implements probably (hopefully) the simplest and the best. bce is respected and the new line is filled with the current background color if scrolling occurs because of an explicit escape sequence, but is still disregarded when the new line appears due to implicit autowrapping at the end of a line.

Comment 25 Egmont Koblinger 2017-04-23 22:09:01 UTC

The original risky patch has been submitted for 1.5 years. Since then, we've received two bugreports: the le editor which is now fixed, and dosbox (in the Ubuntu bug) which cannot be reproduced (and is also probably fixed by this followup patch).

As such, let me be brave enough to close this bug.

[Dmitry Sapozhnikov (o-sdn-o)]

I've added a list of past historical discussions to the description of this discussion. The source of the issues being discussed is crystal clear, and a solution has been found that provides a workaround for previously unavoidable issues. Thank you all.