INCIDENT 025 | Error in median selection precision

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-04-06

Summary

Orcfax heartbeat was missed at 0200 UTC on 6 April due to an error in median selection precision between collectors and the
validator.

Status

Under Review

Assessment

Orcfax collectors are written in Golang, and the validator in Python. With an even number of collector values, the team has discovered that median is occasionally being calculated to a different precision between the collector and the validator. This causes a median selection error.

Additional Notes

For consumers, the 0200 UTC heartbeat was missed and the problem corrected itself at the next regular heartbeat.

At the time the incident was first noticed, the Orcfax team did not have enough information to determine the cause and to satisfactorily complete the incident report.

However, additional observations since have allowed the team to better understand and reflect on the issues being seen.

Technical improvements

We are investigating:

1. More efficient methods for determining the level of precision in the validator node.
2. Temporarily reducing the number of sources in collectors from six to five.
3. Identify the correct retry mechanism between the validator and coop that increases publishing reliability.

Documentation improvements

1. Updating the documentation for devops teams looking at diagnosing validation issues.