Modernise stack: drop foxglove for FastAPI + SQLModel + Celery + uv#495
Open
tomhamiltonstubber wants to merge 10 commits intomasterfrom
Open
Modernise stack: drop foxglove for FastAPI + SQLModel + Celery + uv#495tomhamiltonstubber wants to merge 10 commits intomasterfrom
tomhamiltonstubber wants to merge 10 commits intomasterfrom
Conversation
Replace the foxglove-web framework, raw-SQL/buildpg DB layer, and arq worker with the tc-ai-backend blueprint: - FastAPI directly (no foxglove wrapper), Python 3.12, uv + pyproject.toml - SQLModel for the ORM with a custom DBSession (create/get_or_404/ get_or_create) copied from tc-ai-backend - Celery + Redis for background work, beat process for the two cron jobs - Logfire + Sentry for observability, ty + ruff for lint/type - No migrations: SQLModel.metadata.create_all + bootstrap.sql for things SQLModel can't express (enums, triggers, materialised view) Schema parity is preserved byte-for-byte against the old src/models.sql (verified via pg_dump diff). The 16 HTTP routes match the pre-migration snapshot exactly. The OpenAI spam-check pipeline is removed as part of this refactor; its 8 tests are deleted. Tests: 113 passing, 1 skipped (PDF generation needs x86 wkhtmltopdf, will run on Heroku Linux). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- UserSession: convert from Pydantic BaseModel to a callable dependency so auth failures surface as 403 (not the 422 ValidationError that pydantic v2 wraps validator-raised HTTPExceptions in). - send_email: write the send_request_failed row from a custom Task on_failure hook when celery raises MaxRetriesExceededError. Keep the in-body short-circuit for the direct-call test path. - Materialised view: switch CREATE → CREATE IF NOT EXISTS so dyno restarts no longer wipe the cache. Tests refresh the MV in _truncate_all to keep cross-test state clean. - delete_subaccount: bulk-delete via FK CASCADE instead of loading every Message + MessageGroup into memory. - Switch update_aggregation_view + delete_old_emails to db.execute(text(...)) — Session.exec() is for typed select() queries. - Standardise webhook dedup + click dedup on `SET key 1 NX EX <ttl>` everywhere (atomic, no incr+expire race window). 113 passing, 1 skipped (unchanged). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code: - Drop unused DBSession helpers (create/exists/get_or_404), unused HTTP client verbs (delete/put), unused getter functions (get_mandrill/ get_messagebird), and prepare_search_query — all dead code. - Replace email_send_view's manual get-or-create with the shared helper in sms.py, which now uses db.get_or_create() — exercises that helper. - Refactor _SendEmailTask.on_failure to write the failure row directly rather than instantiating SendEmail (which needs a celery request). - Add server_default ''::tsvector on Message.vector — defensive backup if the set_message_vector trigger ever drops. - Add default_factory=utcnow on created_ts/send_ts/update_ts/ts so Pydantic-side construction doesn't require explicit timestamps. Tests + tooling: - Drop pytest-xdist; tests run sequentially in 5s, parallel deadlocks on shared TRUNCATE. - Add tests/test_parity.py covering HMAC parity, enum round-trips, tsvector trigger output, and the harder-to-reach branches (retry exhaustion, 409 dedup, 404 billing, signature failures, HEAD webhook, get_or_create behaviour, sentry/logfire init paths). - Coverage: 92.43% → 100.00% (omitting app/worker.py, the celery bootstrap entry point). - ty: 58 diagnostics → 0. Auto-add ignore comments on SQLModel/ SQLAlchemy strict-typing friction; matches tc-ai-backend's clean ty status. 140 passing, 1 skipped (PDF rendering, x86-only wkhtmltopdf binary). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # src/version.py # src/views/common.py # tests/test_email.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Heroku's rediscloud add-on populates REDISCLOUD_URL, not REDIS_URL. foxglove's settings used to handle this; restore the same fallback so the worker/beat dynos pick up the correct broker URL on Heroku. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
|
@Henty — heads up: master commit fae14a5 ("Avoid storing clicks temp") commented out the click-tracking enqueue and Could you confirm — should this PR keep click tracking enabled, or restore the temp-fix in |
Member
Author
|
Resolved: keeping click tracking on as currently committed — Tom confirmed re-enabling is intentional. |
The old publish job built morpheus-mail from setup.py and uploaded to PyPI on tag pushes. Restore it — moved the publishable package's setup.py into packaging/morpheus-mail/ so it lives in its own directory and the root pyproject.toml (which defines the main app) doesn't interfere with the build. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleaner than the relative package_dir approach which polluted the source tree with copies of the render module. The publish job now copies app/render/ into packaging/morpheus-mail/morpheus/render/ before building. The staging dir is gitignored. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
beatprocess for cron jobs).DBSessionhelper class lifted from tc-ai-backend (create,get_or_404,get_or_create, etc.).pip+requirements.txt(Python 3.9) touv+pyproject.toml(Python 3.12). Dropsetup.py,setup.cfg,runtime.txt.tyfor type checking andrufffor lint/format.SQLModel.metadata.create_all()runs on startup, sandwiched betweenbootstrap.sql(extensions, enums, plpgsql functions) andpost_bootstrap.sql(triggers, materialised view, materialised-view index). Both files are idempotent.src/spam/,src/llm_client.py, the spam branch in the email send view, related settings). 8 spam-only tests deleted.What we did
app/mirroring tc-ai-backend (core/,messages/api/,messages/tasks.py,ext/clients.py,common/auth.py,sentry/,observability/).HttpMessageError+ custom exception handler so error responses keep the legacy{'message': ...}shape (clients and tests rely on this).httpx.Clientshared at module level.@celery_app.task(send_email,send_sms,store_click,update_mandrill_webhooks,update_message_status,update_aggregation_view,delete_old_emails). Email retries usetask.retry(countdown=...)with the same backoff schedule (EMAIL_RETRYING).tests/conftest.pywith sync TestClient, transactional truncate, eager Celery, and ahttpx.MockTransportreplacement for the old aiohttpdummy_server. The mock transport tracks request log entries that the legacydummy_server.app['log']assertions expect. SyncDb shim translates$Nplaceholders to SQLAlchemy named params and stringifies JSONB values (legacy parity).Verification
pg_dump --schema-only --no-owner --no-privilegesof a fresh DB built bycreate_db_and_tables()diffs to zero meaningful changes vs the oldsrc/models.sql(only the random\restricttokens pg_dump injects per dump differ).uv run pytest tests/→ 113 passed, 1 skipped in 5s. The skipped test (test_message_details_links) needswkhtmltopdfto run; pydf's bundled binary is x86_64-only and won't execute on macOS arm64. It runs on Heroku Linux.uv run ruff check app/ && uv run ruff format --check app/clean.Out of scope
app/— all SQLModel/SQLAlchemy strict-typing complaints (no functional issues). Will clean up in a follow-up PR.Deploy notes
Procfileadds a third process (beat) for Celery cron jobs. Without scalingbeat: 1in the Heroku dashboard,update_aggregation_viewanddelete_old_emailswill silently stop running.heroku rollback— schema unchanged.Test plan
create_db_and_tables()against a clone of prod data — verify it's a no-op (noALTER/CREATE TABLE).POST /send/email/(Mandrill),POST /send/sms/(Messagebird),POST /webhook/mandrill/with a known signature,GET /messages/email-mandrill/,GET /messages/email-mandrill/aggregation/.beatdyno before merging to prod.🤖 Generated with Claude Code