Inconsistency regarding which states DM.abort() may be called in

[ctheune]

BUG/PROBLEM REPORT / FEATURE REQUEST

I think there's a consistency issue here between how abort() in data managers is documented and how it is actually handled.

Exhibit 1: docs

https://github.com/zopefoundation/transaction/blob/master/src/transaction/interfaces.py#L454

Exhibit 2: internal use of abort() after tpc_begin

https://github.com/zopefoundation/transaction/blob/master/src/transaction/_transaction.py#L460

transaction/src/transaction/_transaction.py

Line 428 in 5c0e211

rm.tpc_begin(self)

Exhibit 3: repoze.sendmail prohibiting abort() after tpc_begin

https://github.com/repoze/repoze.sendmail/blob/master/src/repoze/sendmail/delivery.py#L162

Exhibit 4: ZODB support of abort() without paying attention to TPC phase

https://github.com/zopefoundation/ZODB/blob/1079a96cd8e27d6e1333e03239013cb03467b8cf/src/ZODB/Connection.py#L407

I'm puzzled what the correct solution to this is, though. Update the docs? Allow abort() always? Allow abort only after tpc_begin but before tpc_vote? Fix our use of abort() after tpc_begin?

What I did:

We're running zope.sqlalchemy, with sqlalchemy, with psycopg together with a couple of custom data manager but also including repoze.sendmail.

We regularly see this happening:

ERROR [txn.135338525337280][Thread-3 (process_queue)] Error in abort() on manager <repoze.sendmail.delivery.MailDataManager object >
Traceback (most recent call last):
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1969, in >
    self.dialect.do_execute(
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, i>
    cursor.execute(statement, parameters)
psycopg2.errors.SerializationFailure: could not serialize access due to read/write dependencies among transactions
DETAIL:  Reason code: Canceled on identification as a pivot, during write.
HINT:  The transaction might succeed if retried.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/transaction/_transaction.py", line 428, in>
    rm.tpc_begin(self)
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/zope/sqlalchemy/datamanager.py", line 140,>
    self.session.flush()
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4353, in >
    self._flush(objects)
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4488, in >
    with util.safe_reraise():
        ^^^^^^^^^^^^^^^^^^^
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 122,>
    raise exc_value.with_traceback(exc_tb)
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4449, in >
    flush_context.execute()
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 465, i>
    rec.execute(self)
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 641, i>
    util.preloaded.orm_persistence.save_obj(
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 86, i>
    _emit_update_statements(
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 913, >
    c = connection.execute(
        ^^^^^^^^^^^^^^^^^^^
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1421, in >
    return meth(
        ^^^^^
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in >
    return connection._execute_clauseelement(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1643, in >
    ret = self._execute_context(
        ^^^^^^^^^^^^^^^^^^^^^^
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1848, in >
    return self._exec_single_context(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1988, in >
    self._handle_dbapi_exception(
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2365, in >
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1969, in >
    self.dialect.do_execute(
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, i>
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.errors.SerializationFailure) could not serialize access due to read/write dependencies a>
DETAIL:  Reason code: Canceled on identification as a pivot, during write.
HINT:  The transaction might succeed if retried.

[SQL: UPDATE maintenance_windows SET announced=%(announced)s WHERE maintenance_windows.id = %(maintenance_windows_id)s]
[parameters: {'announced': True, 'maintenance_windows_id': 13730475}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/transaction/_transaction.py", line 465, in>
    rm.abort(self)
File "/srv/dir2/deployment/work/instanceasyncrunner/.venv/lib/python3.12/site-packages/repoze/sendmail/delivery.py", line 168, in>
    raise TPC_InProgress()
repoze.sendmail.delivery.TPC_InProgress: TPC in progress```

### What I expect to happen:

This should not lead to an abortion error.

### What actually happened:

It led to an abortion error. ;)

### What version of Python and Zope/Addons I am using:

Linux, Python 3.12,  sqlalchemy 2.0.50, repoze.sendmail 4.5, transaction 5.1