URLs preprocessing (filtering, cleanup, deduplication, ...) and default skip-list

[drzraf]

URLs can:

1. ✅ be simply unsuitable for indexing (rules.json skip)
2. [?] Contain a hash (url/#some-hash)
3. ❌ convey unwanted tracking parameters (utm_*) [but the resource is still of interest]
4. ❌ convey unwanted private values (token, oauth code, ...) [but the resource is still of interest]
5. ❌ have the parameter in distinct order but still represent the same resource (?a=b&c=d vs ?c=d&a=b) [https://github.com/Indexing rules documentation #277#issuecomment-4187306327]
6. ❌ unwanted domains/private path segments (/logout) or subdomains (auth.domain.com) deserve a default list provided by the indexer to make the first import smoother and safer.

For 3 and 4, some parameters are user-specific. But many are just a problem of the modern, impacting equally everyone. Like for cookies, there are database of known query-string parameters (associated with the domains).

Anyone who already dealt with cache deduplication probably compiled a list of dozens of such parameters. There are used in http frontend, in anti-tracking browser extension (even hardcoded in browser themselves, like in Firefox or Brave

It's logical that this problem (and corresponding solutions) fall onto content indexers too.

Providing something "safe" out-of-box is of uttermost importance to avoid users having the bad surprise to see very private content in their search results.

Regarding Point 3 and 4, just an initial command/regexp to strip common session-ids:
sed -r 's/#.*//;s/([?&])(g_ep|__mk|s?client|utm_[a-z0-9_]*|sca_esv|hl|biw|bih|ved|gs_lp|__cf_chl[a-z0-9_]*|usg|ust|uauid|skid|sfmc_[a-z0-9_]*|tid|fbs|s?ei|gs_lcp|gs_l|dpr|browser_session_id|iflsig|sg_ss|qs|hs|gclid|code|zx|ai|vssid|refinements|rh|pvid|crid|arko|pd_rd_[a-z0-9_]*|visitId|_gl|content-id|_x_ns_[a-z0-9_]*|ds|qid|purchaseId|pdp_|curPageLogUid|spc|srsltid|sstk|state|tab|share_id|token)=[^&]+/\1/ig;s/&&+/\&/g;s/[?&]$//;s/\?&/?/g'

Regarding Point 6:

account/
/your-account
/auth/
/booking/
/checkout/
/dashboard
/connexion/
/dolibarr/
/(login|o?auth2?)
/edit
/new
/settings
https?://(passport|connect|admin|mail|dashboard|idpsesion|myprofile|oauth|oidc|auth|authenticate|authenticator|login|authorize|moncompte|accounts|account|myaccount|espace-client|idp|billetterie)\.
/order/
/manager
/preferences/
/traduction/
/signin
/signup
/roundcube
/unsubscribe
/user/login
/messages
/am/

I used various heuristics to end-up with this list and it complied with my expectations : 100% of the URL matched by these were content I definitely wouldn't want indexed and users who would want them to be are probably not the larger share and could actively broaden a default skip-list. I think it's important to provide something like this out of the box (config generator ?)

[asciimoo]

@drzraf I absolutely agree, both of these would increase the quality of our index and all of these are valid future enhancement ideas. Thanks for the provided examples/collections as well.

Quick note: we already remove utm_ parameters: https://github.com/asciimoo/hister/blob/master/server/indexer/document.go#L113

100% of the URL matched by these were content I definitely wouldn't want indexed and users who would want them

What was the sample size you've checked?

I think it's important to provide something like this out of the box (config generator ?)

We do have a default config. The user provided config only overwrites values in the default config. It could be added to the default config similarly to the sensitive content filter: https://github.com/asciimoo/hister/blob/master/config/config.go#L350

[drzraf]

Quick note: we already remove utm_ parameters: https://github.com/asciimoo/hister/blob/master/server/indexer/document.go#L113

Oh ! Good to know !

100% of the URL matched by these were content I definitely wouldn't want indexed and users who would want them

What was the sample size you've checked?

~ 60k URLs (mostly EN/FR)

[drzraf]

Once the txt/json file is in place in the repository + infrastructure for defaults URLs + possibly query-string substitution, I could suggest these additions (and many more I still have):

    "(mail|my).salesforce.com",
    "stripe.com",
    "GOOGLE_ABUSE_EXEMPTION", // Google captcha/verification pages
    "=eyJ", // oauth tokens

and tons of others: Email provider, payment providers, oauth providers, services providers, administrations, shop, banks, airlines, chats/meeting webapp, CRM, dashboards, ...

[asciimoo]

These are super valuable insights and data. Thanks a lot for sharing them. There are several other high priority issues around the project, so I can't guarantee that I'm immediately implementing all these, but I'd definitely like to refine the indexing part of Hister on the long run.

Btw, if anyone wants to work on these, all contributions are highly appreciated. =)

[bjeanes]

=eYJ is used in non-sensitive contexts too. The first example that comes to mind doesn't have the leading = to be fair, but Metabase encodes the SQL query into the hash fragment as a JWT: https://metabase.example.com/question#eyJ...

[drzraf]

Wasn't 100% after all. Here are some URLs wrongly filtered by the above:

https://authjs.dev/
https://docs.retool.com/data-sources/guides/authentication/api
https://elevenlabs.io/docs/api-reference/authentication
https://firebase.google.com/docs/cloud-messaging/auth-server
https://github.com/knpuniversity/oauth2-client-bundle
https://packagist.org/packages/google/auth
https://oauth2-client.thephpleague.com/
https://api.stackexchange.com/docs/authentication

But nothing a good regexp couldn't fix :)

[drzraf]

But ... there is no Go regexp negative look-behind. So no (?:(?!doc).)*(auth) (unless using regexp2 and in the event this kind of anywhere/variable length negative look-behind are supported)

[wont-work]

for the subset of canonicalizing tracking parameters, uBlock Origin (and i believe AdGuard as well?) also have a removeparam function, so pre-proccessing their filterlists into a format hister can use could be worth it to bootstrap a list. the ClearURLs addon also has a list which seems to be reasonably up to date, and easier to ingest: https://github.com/ClearURLs/Rules

additionally using ad blocker filterlists before or as an alternative to readability can also be worth it, especially ones that might otherwise be "too heavy" for general purpose use, such as, https://github.com/LanikSJ/webannoyances.

all that said, i'm not exactly sure how easy that'd be given the bespoke nature of filterlist syntax. i know brave has a rust implementation with python bindings, and uBO of course has a javascript implementation, but i'm unsure if there's any implementation for go for the CLI indexer

unrelatedly: query sorting is something several caches can already do (here's the docs for vinyl cache) and may be worth implementing, but query syntax isn't strict so this could break websites which use nonstandard query param syntax (or otherwise cut corners)

[asciimoo]

These are useful addition indeed, thanks!

additionally using ad blocker filterlists before or as an alternative to readability can also be worth it, especially ones that might otherwise be "too heavy" for general purpose use, such as, https://github.com/LanikSJ/webannoyances.

This can be achieved by having adblock in your browser that runs the addon or indexes the websites.

query sorting is something several caches can already

This is a difficult question. It would be good to measure how often websites scramble GET parameters VS how often websites provide different results when you change the order.

[drzraf]

The most borderline case I can think of is ?list[]=z&list[]=a, aka when a single parameter is provided multiple times and the app' pathetically relies upon the order being preserved.
Any other case hoping for the order to stay untouched would be soon broken by so many caching proxies (inc. Cloudflare query-string optimization/reordering feature).

[drzraf]

(LLM-coded) tested implementation of the ClearURLs/uBlockOrigin filter parameters syntax (JSON/txt).

$ time sed -n 1,32000p 100k-urls.txt | parallel --pipe -N8000 -j4 './clean-urls -c data.min.json -u uBo-removeparam.txt' 1>/dev/null 
real	0m24,318s
user 1m9,788s
sys 0m0,644s

main.go.txt
go.mod.txt