sitemap a11y improvements

[StephDriver]

closes #5170

[StephDriver]

Work Done:

• updated the CMS pages to have drafts (with previews) and tidied up the CMS interface a bit. Then we can use CMS published pages as source of truth for additional pages for sitemaps.
• changed utils/tests.py into a package and separated out the sitemap tests.
• rewrote generate_sitemaps from ground up, to use a different structure that would validate with sitemaps schema, and satisfy WCAG requirements for link deduplication and disambiguation.
• updated the progress bars while running the generator to show number of sitemaps per object.
• created a new template_tag for linking to sitemap from each front of house footer.
• wrote a new set of tests for this new generate_sitemaps.
• split the original sitemaps-and-robots.md and then updated the sitemaps part, with tables to explain which pages are included in sitemaps and when (lots and lots of logic to cover). I have reconciled these back to the tests.
• manual tests of new sitemaps alongside automated tests
  • loading them up and moving around the structure.
  • git tracked the sitemaps output, so that I could confirm they were consistently generated.
  • checked back and forth from the new sitemap footer links.

Notes:

1. a11y-mode work will also need to be updated to include this footer link too. see Add sitemap footer link to Accessibility mode templates clarity#7
2. Original docs recommended setting chron to run generate_sitemap every 30 min. Is this still wise? This is a lot of processing. Do we need to consider some kind of incremental update command.
3. There's a lot of logic. I have been over it line by line several times until my eyes blurred - good luck with it.
4. This moved beyond what was described in the original issues because we needed a source of truth for which CMS pages to include, and having explored using the Nav, it was clear the better solution was to introduce draft/publish flags for CMS. Further, the structure had to be amended to pass the validation, as we could only have either URLs or site-indexes listed on any one sitemap, we couldn't mix them.

[StephDriver]

rebased onto r-v1.9.x

[ajrbyers]

@StephDriver passed this back as tests are failing here.

[StephDriver]

@ajrbyers which tests? All of mine are passing.

...

Well never make an assumption. Its actually ruff formatting see: https://github.com/openlibhums/janeway/actions/runs/27752395486/job/82105825907?pr=5312

ooh, that's interesting, because I was running a formatter with a pre-commit hook - which won't let me commit until it passes, so that means my formatter is out of sync with the one testing here. I'll have to dig into it more on Monday as to what and why.

[StephDriver]

I haven't worked out why my existing hook wasn't firing for that file, when it fired for others - the most likely explanation is that it the difference crept in during a rebase. But the two missing empty lines have now been duly added.

[StephDriver]

Unpicking the test failures.

• sorted out the migrations
• updated the ruff formatting and added new guard as it was testing disc versions not staged versions pre commit. ruff pre-commit git-hook targets disk not staged files #5360
• went through all my new tests, updated them - found an error in my code relating to how draft access codes where handled (it allowed reuse) - fixed that, and updated tests around this new behaviour.
• tested locally with python 3.11 (as that is the one that keeps failing here) - I usually work in python 3.12, and I couldn't reproduce the errors here with that. Can't actually reproduce the problem with either.

Failures that are on the base branch too

• ERROR: test_pdf_to_text (src.core.tests.test_files.TestFilesHandler.test_pdf_to_text)
• FAIL: test_search_includes_article (src.journal.tests.test_journal_frontend.TestJournalSite.test_search_includes_article)
• FAIL: test_section_editor_cannot_see_pii_when_enabled

I cannot reproduce the test errors from github locally. They seem to come down to submission_article.primary_issue_id which I refer to here, but I didn't create. So my best guess is some kind of missing migration - but that's only affecting GH and not local?