Otel testing

[Mirko-von-Leipzig]

Figure out a reasonable way to test our open-telemetry spans and attributes.

Some investigation was done in #678 (but was reverted as out-of-scope):

• chore: add Open Telemetry attributes to grpc spans #698 (comment)
• chore: add Open Telemetry attributes to grpc spans #698 (comment)

Concerns

Ideally we would:

1. Attach a test span exporter (which collects the spans in memory)
2. Run some code
3. Assert on the resulting spans

The main issue iiuc is that the exporter is registered globally which makes running such tests.. a bit difficult.

We also don't want these otel tests to interfere with other tests which don't care about otel (but do create traces).

We could separate out these tests entirely, and force them to run sequentially. This might be a good use case for cargo xtask. Or we treat them as pure integration tests and create a full setup. The downside of this is complexity and inability to specify very niche scenarios.