Hydra explorer supporting multiple hydra versions

[ch1bo]

Why

The Hydra explorer should be able to report Hydra heads from multiple different protocol versions on all public networks. The explorer can be used to give usage information about each version.

What

• The hydra-explorer lists heads from all networks and supported versions according to networks.json

• The hydra-explorer does not need to be updated on every script change (e.g. when we switch plutus-tx/GHC versions)

• The hydra-explorer reports a protocol version per head.

• Can run multiple "(chain) observers", one for each network and hydra protocol version, feeding information into one hydra-explorer instance.

  • The hydra-explorer is decoupled from a specific hydra version
  • The interface between observers and explorer is specified and tests will ensure we don't break it accidentally

• The hydra-explorer should be deployed from https://github.com/cardano-scaling/hydra-explorer using multiple versions of the main hydra repository

  • We need to do this because, say, the script hashes are not enough to distinguish the version; i.e. we can't make hydra-node/src/Hydra/Chain/Direct/Tx.hs generic across versions; the easiest way is just to depend on the entire codebase 🥲

• We should delete the hydra-explorer folder from the main hydra repo

How

• The endpoint is a POST to /v1/observations/<network>/<version> with request body of some JSON object enumerating the observations

• Add an observer-side API definition to hydra-explorer OpenAPI schema

• The interface should be JSON-based simpler HTTP requests; so that it's easy reason about backwards compatibility and debug, and test

• hydra-chain-observers are HTTP clients sending requests to hydra-explorer observer-side API whenever a head transition is observed

• Observer-side API is exposed on separate port so we can host and port configurable on the command-line of hydra-explorer

• Both hydra-chain-observer and hydra-explorer are tested agains the observer-side API schema

TBD

• Is specifying an interface + testing against enough to keep this maintainable?
• Make the explorer understand any "old versions" of observer data?
  • Eventually drop too old observer versions
  • For how long should be support older versions? Will we have some separate notion of LTS versions for Hydra?
  • Might be older versions that we don't support but keep an eye on
• Can this support protocol changes like
  • additional transitions (incremental commit)
  • not having a commit phase anymore
• Protocol version is basically:
  • scripts, their hashes and addresses
  • datums and redeemers as contained in transactions (their format)
  • structure of protocol transactions

[noonio]

Having looked into this a bit as part of cardano-scaling/hydra-explorer#5 I think it's worth making some notes:

1. I don't think it's best to invoke hydra-explorer multiple times; let's just make an option for the chain-observer to look for multiple "versions"
2. A version is defined by the Tx Id in networks.json and the script outputs from there; from this a ScriptInfo datatype can be reconstructed
3. This can be used by the chain-observer to send ticks

I.e. the real limitation is with the chain-observer; so we just need to make that parameterisable by a specific ScriptInfo parameter; then maybe we just run multiple chain-observers, or (probably better) run a single one that attempts to find all the ScriptInfo things that it is provided.

-- Edit:

As discussed with @ffakenz and @ch1bo , it is not possible to "de-version" the relevant parts of the chain-observer code (i.e. to parameterise it by the version), because all the version information is essentially hard-coded in the code itself; so it's easier just to depend on different revisions of the code and adapt them to write to a generic (?JSON) format consumable by the hydra-explorer.

[ch1bo]

We discussed again the intended architecture of this and drew this picture today. It shows multiple versions of the chain observing code (in hydra-chain-observer) and also running on multiple networks / against different cardano-nodes, all aggregating data into a single hydra-explorer instance.

[ch1bo]

When working on this I realized that a lot of our dependencies could be improved. For example:

• The hydra-explorer now depends on hydra-tx (which depends on hydra-plutus scripts), only for the "interface" types like HeadId, Snapshot etc. Having a hydra-api, that contains Hydra specific entities and (hydra-)cardano-api that contains cardano specific entities would be more suitable.
• Similarly, the hydra-explorer depends on hydra-node mainly because of the OnChainTx type in Hydra.Chain. Either, we deliberately sever that dependency by replicating a compatible type into hydra-explorer, or we could move that (and other chain interface things) into a hydra-chain package. This would be used by all applications, the hydra-chain-observer, hydra-explorer and hydra-node.