Document DataStore exported ledger metadata file format

[tamirms]

Background

galexie is a service which publishes LedgerCloseMeta XDR objects bundled into a serialized file format LedgerCloseMetaBatch to a DataStore. Allowing subsequent queries from consumers for ledgers as a faster/easier alternative to acquiring the same from a captive core.

DataStore is the Go interface which defines all storage operations required by galexie. Currently, there is only 1 concrete implementation of the DataStore interface which is backed by GCS. However, we anticipate that there will be other DataStore implementations backed by other storage engines. For example, a DataStore implementation could be backed by S3, a local file system, or even a relational database like postgres.

Problem

External systems are not able to easily interoperate with GCS datastore exported bundled files because there is no external facing documentation on the export format:

Acceptance Criteria

Create a new ecosystem SEP which defines specification on Datastore export formats:

• Document the exported serialized bundled file format, it should adhere to LedgerCloseMetaBatch.
• Document the requirements for GCS Datastore exported bucket schema.
  • Define how the GCS bucket name and path should be encoded for a given exported bundled file based on the LedgerMetaArchive.ObjectKey which is an encoded string over partition, ledger range, and sort prefix.

[urvisavla]

There are a couple of open items that could influence the datastore specification:

• Co-locate schema settings on datastore (Galexie: co-locate schema settings on datastore #5507)
• Simplifying the append functionality (resumability) (services/galexie: Simplify resumability logic #5547)

Co-locating the schema on the datastore might not be strictly necessary for the formal specification but I think it could be useful. However, the proposed changes in #5547 to simplify the append logic will definitely require the datastore to support listing objects in a specific order.

Imo, tackling these issues will help create a more robust interface that enables seamless datastore interchangeability in Galexie.

[sreuland]

The pluggable datastore design, #5526, would be a related effort that is pending, if that comes to reality, then it would redefine what constitutes the interface of a Datastore.