Better (and faster) compression with zstd/brotli

[Glandos]

Since 58c3c6f (that fixed #384), compression was done with gzip.

Nowadays, gzip is completely outperformed by zstd and brotli regarding speed and efficiency. As an example:

zstd -b1 -e22 f68f580fc2f4ac1a3176b1673b7a8c402c719a5d4ada7fd9efae8cf693
 1#ada7fd9efae8cf693 :   4622059 ->    705138 (x6.555),  957.6 MB/s, 1869.9 MB/s
 2#ada7fd9efae8cf693 :   4622059 ->    579052 (x7.982),  276.5 MB/s, 2028.8 MB/s
 3#ada7fd9efae8cf693 :   4622059 ->    445631 (x10.37),  248.6 MB/s, 2394.3 MB/s
 4#ada7fd9efae8cf693 :   4622059 ->    444517 (x10.40),  363.6 MB/s, 2311.9 MB/s
 5#ada7fd9efae8cf693 :   4622059 ->    391150 (x11.82),  234.5 MB/s  2336.4 MB/s
 6#ada7fd9efae8cf693 :   4622059 ->    372034 (x12.42),  188.3 MB/s  2659.3 MB/s
 7#ada7fd9efae8cf693 :   4622059 ->    358079 (x12.91),  164.8 MB/s, 2746.9 MB/s
 8#ada7fd9efae8cf693 :   4622059 ->    351268 (x13.16),  139.2 MB/s, 2796.8 MB/s
 9#ada7fd9efae8cf693 :   4622059 ->    266755 (x17.33),  167.7 MB/s, 3463.0 MB/s
10#ada7fd9efae8cf693 :   4622059 ->    261443 (x17.68),  130.6 MB/s, 3346.0 MB/s
11#ada7fd9efae8cf693 :   4622059 ->    257953 (x17.92),   90.3 MB/s, 3605.4 MB/s
12#ada7fd9efae8cf693 :   4622059 ->    257860 (x17.92),   88.2 MB/s  3526.9 MB/s
13#ada7fd9efae8cf693 :   4622059 ->    258150 (x17.90),   24.1 MB/s, 3591.9 MB/s
14#ada7fd9efae8cf693 :   4622059 ->    254147 (x18.19),   15.5 MB/s, 3653.4 MB/s
15#ada7fd9efae8cf693 :   4622059 ->    251304 (x18.39),   8.87 MB/s  3566.1 MB/s
16#ada7fd9efae8cf693 :   4622059 ->    244460 (x18.91),   16.4 MB/s, 3518.5 MB/s
17#ada7fd9efae8cf693 :   4622059 ->    239654 (x19.29),   14.9 MB/s, 3524.2 MB/s
18#ada7fd9efae8cf693 :   4622059 ->    238287 (x19.40),   10.5 MB/s, 3515.5 MB/s
19#ada7fd9efae8cf693 :   4622059 ->    235423 (x19.63),   6.43 MB/s, 3532.7 MB/s
20#ada7fd9efae8cf693 :   4622059 ->    235423 (x19.63),   6.64 MB/s, 3518.8 MB/s
21#ada7fd9efae8cf693 :   4622059 ->    235057 (x19.66),   4.92 MB/s, 3444.4 MB/s
22#ada7fd9efae8cf693 :   4622059 ->    235019 (x19.67),   4.12 MB/s, 3420.9 MB/s

The original chunk in the datastore was compressed with gzip at 889249 bytes. So even with the poorest zstd settings, file is 79.3% of the gzip one, and (de)compression speed is far better.
Brotli is also a good candidate, since it embeds a dictionary specialized for HTML, although performance are roughly the same.

Of course, it introduces a breaking change, that's not easy to manage, but it's better sooner than later :)

[asciimoo]

Good suggestion, thanks! Probably it makes sense to keep the gzip for the favicons, because we transmit it directly over HTTP, but for the HTML content it would be a big improvement.

We should also consider a better storage format than using the file system directly (perhaps parquet?) where we could apply storage level compression and/or we could avoid block size overheads at least.

Of course, it introduces a breaking change, that's not easy to manage, but it's better sooner than later :)

It isn't that bad to manage the changes. We use an indexer versioning, exactly for these cases. We can increment the indexer version when we introduce the new compression (which automatically warns the user to run a reindex) and depending on the indexer version we can decide which compression we use.

Are you also interested in working on the implementation?

[Glandos]

Are you also interested in working on the implementation?

Thanks for the quick answer!

Unfortunately, I have no development time to dedicate to hister :(

[asciimoo]

No worries, your contribution is already valuable. Thanks again and let me know if you have other suggestions.