webserver: stop the file editor buffering whole files into internal RAM

[kezarjg]

The web file editor at /api/file serves any readable path under /store or /sd, so the body is arbitrarily large. On a GET, HandleFile passed the whole body to PageContext::print(), which appends it to the connection send buffer, and that buffer cannot drain while the handler is still running. So the file exists twice for the duration: once in the extram::string it was loaded into, which is SPIRAM, and once as an mbuf copy in internal 8-bit RAM, which is the scarce pool.

This streams the body straight out of the buffer it was loaded into, so the second copy never happens.

Evidence

Browsing stored logs in the editor failed reproducibly once a session had served roughly 660 KB across about two dozen files. The failing allocation was usually somewhere unrelated, because what ran out was internal RAM in general, so the visible symptom moved between runs. After the change the same walk completed 72 files and about 20 MB cleanly.

Measured on an OVMS v3 module reading SD-card logs over the local network. Those runs were on my own tree, which carries this change plus unrelated work: the code path is identical, but the module hours are not attributable to this branch alone.

The change

HandleFile's GET path hands ownership of the loaded body to a chunked sender and returns. The sender frees the body and itself after the last chunk and emits the terminating zero-length chunk, which is why this path deliberately does not fall through to c.done(). An empty body, meaning a directory path, is handled: the sender terminates on its first event.

HttpStringSender is generalised to HttpStringSenderT<StringType>:

typedef HttpStringSenderT<std::string>      HttpStringSender;
typedef HttpStringSenderT<extram::string>   HttpExtRamStringSender;

The old name survives as a typedef, so this is source compatible, and nothing else in the tree referenced the class, so HandleFile is the only behavioural change. The implementation moves into the header because it is now a template, which is a mechanical move with the ESP_EARLY_LOGV traces preserved verbatim (the log tag is the literal "webserver"). The one functional edit inside the sender is computing the chunk length with a comparison rather than the MIN macro, which is not visible in the header.

Testing

Builds clean in CI with no vehicle modules enabled, so the result is attributable to this change alone. That covers compilation; the figures above are the behavioural evidence.

Worth knowing

The streaming path is roughly 1.6x slower than serving the same file statically. Irrelevant when editing a script, but a client pulling a multi-megabyte file through /api/file needs a timeout above about 150 seconds, and a default-timeout client may now give up where it previously succeeded.

I could not find an existing issue for this. It surfaced while chasing an unrelated memory problem rather than from a user report.

Three decisions I would happily revisit

Only HandleFile is converted. The other c.print() sites on an extram::string are PluginHandler, which holds a reference to registry-owned content and so cannot pass ownership without a copy, and PluginCallback, which emits a fragment into a composed page where a stream-and-return sender would truncate everything after it. Plugin content is also bounded in practice, unlike an arbitrary path under /sd. Happy to add a non-owning variant if you want those covered.

Template versus a second class. Duplicating forty lines to change one type seemed the worse trade, but this does touch code that already works. Easy to switch to a standalone extram sender alongside the concrete class if you prefer.

Streaming is unconditional. Small files pay the slowdown for a problem they do not have. A size threshold avoids that at the cost of two paths to keep correct, and I erred toward one.