feat: return objets counts in orgs suggest for search facets

[ThibaudDauce]

GET /organizations/suggest/ can now annotate each suggested organization with the
number of objects it owns that match a search, so a frontend can build an organization
filter with counts.

Parameters (all optional, on top of the existing q / size):

• count_for (dataset | reuse | dataservice): annotate each suggestion with a
  matching_count. Without it the endpoint stays a plain name suggest.
• count_filter.*: the search the count is scoped to (e.g. count_filter.tag=transport).
  Namespaced so the dataset full-text query (count_filter.q) never collides with the
  organization name query (q).
• topic: restrict candidate organizations to those owning a count_for object in this
  topic (the "universe").
• count_facet_ids: organization ids from the caller's current search facet, kept as
  candidates so organizations that actually have results show up even when they are not
  the most followed. They go through the same name + topic constraints as the rest.

How it works:

• MongoDB selects the candidates: most-followed organizations matching the name
  (restricted to topic when given), merged with the matching count_facet_ids.
• Elasticsearch computes the counts in a single size: 0 aggregation (include on the
  candidate ids, scoped by count_filter.*) — no result fetching.
• Results are ranked by followers, organizations with no match pushed to the end. Counts
  are never used as a sort key.

matching_count is null when count_for is absent, 0 when counted with no match,
>0 otherwise.

Replaces #3804. Covers #3738's organization-filter need through topic + count_facet_ids
without exposing Elasticsearch facet sizes to API consumers.

[Samuelfaure]

LGTM (not approving cause I dont feel confident enough in my skills on this repo)

[abulte]

@ThibaudDauce congrats on your creativity 🧠

As per being a replacement of #3738, from what I understand it suffers from the following limitations:

• I'm capped at 100 items being "counted", when feat(search): configurable facets size #3738 lets me adjust the facet size to my needs (I know roughly the number of orgs in my universe, let's say 200).
• Among the results, I'm not sure the whole list is indeed related to my count_filter.* query. I can get popular organisations that are not part of that result set (eg an organization without a dataset in my universe Topic). I can filter them out client side (which is not great) but they're eating into my already sparse 100 cap/quota.

I think your solution could work well in a purely "query suggest" context, I guess that's what you have a mind for data.gouv.fr's search. But for our current requirement on ecologie.data.gouv.fr (replicate the organisation list filter here https://ecologie.data.gouv.fr/datasets), it would fall a bit short.

[ThibaudDauce]

@abulte I reworked the PR to cover your case. Two additions:

• topic: restricts candidate organizations to those that actually own a count_for
  object in the topic (MongoDB-side, no facet-size cap involved). It also works on its
  own, without count_for — a plain name suggest scoped to a topic's organizations —
  so you can reuse it elsewhere if you need that.
• count_facet_ids: the organization ids from your current facet (the ones with results),
  added to the candidate pool under the same topic + name filtering as the rest (in your
  case the facet already comes from that topic, so none get dropped). This surfaces
  organizations with results ahead of empty ones — without it, if your most-followed
  organizations had no match you could get a page of zeros. Ranking among results is still
  by followers.

For your org list filter on a topic, e.g.:

/organizations/suggest/?count_for=dataset&topic=<topic_id>&count_filter.topic=<topic_id>&size=20

→ organizations of the topic, ranked by followers, each with its dataset count in the
topic; empty ones pushed to the end. When the user types, add q=... to search within
that same universe. No 100-cap, no out-of-universe organizations eating slots.

This keeps facet sizes internal (so it also replaces the need from #3738). I'm not assuming
it covers everything on your side — does this fit the ecologie.data.gouv.fr filter, or is
there a case it still misses?

[abulte]

@ThibaudDauce Thanks!

It does seem to cover some important use cases: organisations lists for the current universe in datasets and dataservices.

It would even cover things #3738 didn't:

• List all the orgs related to a Topic, whatever the linking element type. Nice for the organisations list!
• Suggest capabilities (not sure we'd use that right now, but good to have)

What it would not cover and #3738 did: list the orgs for a list of Topics (organisations list here https://ecologie.data.gouv.fr/bouquets). /topics?facet_size__organization_id_with_name=200 could do that. I don't think there's a way to shove that in here, it wouldn't make sense.

Thinking ahead, it would also fail to cover precise results for a query like count_for=dataset&count_filter.tag=xxx (we'd fall into the trap you fixed with the new topics args), where /datasets?tag=xxx&facet_size__organization_id_with_name=200 would yield precise results, with a large enough facet size. We don't currently have this use case, but I feel it might be useful someday, just flagging it.

Overall, I fell this might be over-engineered for topic-specific needs compared to a
controlled exposure of ES capabilities that can be extended easily over time. I would have concerns re performance too, directly querying a specialised system like ES feels safer. How does Dataset.objects(id__in=element_ids).distinct("organization") perform on a Topic with tens of thousands of datasets, fired with a suggest pattern?

If exposing ES internals really is a problem, maybe we could introduce /api/2/topics/{id}/organizations?filter_for=(datasets|...) endpoint. It would cover organisations, datasets and dataservices lists quite naturally, with a "real" endpoint that wouldn't have to be tied to count if we don't want to. The topics list would still need a separate solution. This would avoid conflating suggest concerns (which your original PR does really well) with topic concerns.

[ThibaudDauce]

On "over-engineered for topic-specific needs": the topic part is actually an add-on;
the core value is the global, unbounded case. This single endpoint also solves the APIste
filter (count_for=dataservice) and, importantly, the global data.gouv search filter,
which your alternatives don't cover: a facet can't sweep ~6300 orgs at a sane facet_size,
it's ranked by count (not by followers, so a 14k-dataset no-name org wins), and it isn't
searchable by name. The suggest is the only thing that does name search + scoped count +
follower ranking on an unbounded universe — that's its core, not over-engineering.

So it's really two problems: the unbounded one (data.gouv), which only the suggest solves,
and the bounded topic one (yours), where several shapes work.

On performance — I benchmarked the MongoDB derivation
(Dataset.objects(id__in=element_ids).distinct("organization")):

	latency
topic→orgs derivation — 1,000 datasets (200 orgs)	88 ms
topic→orgs derivation — 10,000 datasets (200 orgs)	146 ms
topic→orgs derivation — 50,000 datasets (200 orgs)	366 ms
full endpoint — topic with 50,000 datasets	367 ms
full endpoint — no topic (data.gouv path)	2.2 ms

Local test MongoDB, ~200 unique orgs; cost is driven by the number of datasets in the
topic, not the number of orgs. Directional, not production-absolute.

The cost is isolated to the topic path (data.gouv stays at ~2ms, no derivation) and only
grows with very large topics. It's also trivially cacheable: the topic→orgs set is stable
across keystrokes (only the name query changes), so a short per-topic TTL cache (say 5 min)
means it's computed at most once per window instead of per keystroke — a filter list
tolerates minutes of staleness. A client-side debounce would absorb it too. So I don't
think perf is a blocker.

On the dedicated endpoint: I'd rather not nest it under /topics/{id}/organizations.
If we expose org-of-a-topic lists, it fits more naturally as a filter on the existing
collection — /organizations?topic=xxx — consistent with the other org filters and reusing
the same topic→orgs logic. It's not wired yet but it's a one-liner now that the logic
exists, so we can add it whenever the list use case needs it (optionally with an
element-type param to mean "orgs having a dataset/reuse/dataservice in the topic").

On the multi-topic bouquets list: agreed, neither this nor a single-topic filter covers
"orgs across a list of topics" — that's a separate concern and I'm not trying to fold it in
here.

Does this address your concerns, or is there a case you still feel is missed?

Maybe @maudetes have an opinion too?