Enable RPC as a source of LCM in go/ingest

[mollykarcher]

What problem does your feature solve?

RPC is the recommended starting point for newcomers to Stellar, and we'd like to eventually migrate most people Horizon -> RPC. However, RPC (intentionally) does not have the same indexing capabilities or API niceties that Horizon does, which forces users to index their own data based on their own use cases. This means parsing and traversing raw XDR, as well as becoming intimately familiar with Stellar's underlying data model, which isn't the easiest for a lot of users. We want these kinds of users to be able to use the parsing/transforming componentry we've already built in go/ingest and go/processors so they don't need to reinvent the wheel or port over some parsing code.

What would you like to see?

An RPC service can be swapped in as the source of LCM that can be parsed by the ingest library. Idea being, a golang service/project can:

• Run/deploy RPC
• Import go/ingest (+optionally go/processors) into their project
• Stream LCM from RPC via the same interaction model they would if they were pulling from any other LCM source (captive-core, Galexie-lake)

Something to keep is mind is that we are building a golang client for RPC - stellar/stellar-rpc#326 - and it might be worthwhile to utilize that when building this. It's undecided/unclear where that client will live, but should it live in the stellar-rpc repo, this would create a circular dependency between stellar/go and stellar/stellar-rpc, so this perhaps could influence our current debate around repo organization.

Alternatives

Using a Galexie-exported data lake is an alternative. However, we're specifically thinking about use-cases where the operator/developer will still need to submit transactions, which is not possible via Galexie/data-lake. So while they could use Galexie-lake as their txmeta source, that means they would need to still run both RPC and Galexie. Think wallet developers (ex. freighter), asset issuers (ex. circle), centralized exchanges (ex. coinbase), etc.

[sreuland]

can the new RPCLedgerBackend implementation reside in the stellar-rpc repo directly or maybe it goes in the new rpc client sdk, wondering if this alleviates circular dependency aspects.

in either of those cases,RPCLedgerBackend should implement all methods defined on go/ingest/backend/LedgerBackend but doesn't have to reference the go/ingest package unless wanting to add an assert which confirms it matches the interface. it does need to ref go/xdr in all cases for LCM definitions.

[edit] assuming the rpc client sdk goes into a new/dedicated repo other than as a sub-package in stellar-rpc