INCIDENT 030 | Publishing interrupted on COOP node

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-04-16

Summary

The publication workflow was interrupted on the COOP node during the final stages of publishing on-chain.

Status

Under Review

Assessment

It is unclear as to what happened in this instance; it is likely an exception was thrown but swallowed by the Python tenacity retry library that we use in mainnet.

Additional Notes

Changes have been running in preproduction which allow us to remove the tenacity retry library and monitor events through a systemd service. While an exception will still be thrown in some circumstances, it will be easier to diagnose the problem. Running publishing as a service may also provide some more flexible options for recovery from such errors.

Technical improvements

Preproduction changes will be migrated to mainnet as part of the Orcfax network upgrade anticipated in May/June 2024.

Documentation improvements

N/A