Milestone: Decentralisation and payments specification

[tim-hm]

This is the first iteration of the nil-db decentralisation and payment specification.

Background

The initial nil-db POC was a single node running on ECS backed by an Atlas-based mongodb cluster. Given some initial traction, we are looking to evolve the project by:

1. Decentralising nil-db api nodes
2. Adding a subscription model for payments
3. Improving observability

Design aims and non-goals

Where ever possible the design should prioritise:

• Keep the cluster simple
• Push complexity to clients
• Requests should be stateless
• Keep minimal features until there is a clear requirement for it

Design goals / non-goals / assumptions

1. On topology:
   1. No requirement for internode communication and therefore there is currently no requirement for a leader node.
   2. There will be 3 nodes in the cluster, where a node is a publicly accessible nil-db api backed by persistence (e.g. mongodb).
   3. Fixed topology (e.g. nodes cannot join/leave the dynamically).
   4. The POC will run on vanilla EC2 instances using a docker-compose and future operators can use this as their starting point for running a own node.
2. On data:
   1. Plain text and deterministically encrypted data is replicated across all nodes.
   2. If a value is secret shared, then one share per node.
   3. Warning: If a node losses all data, then secret shared values are lost until threshold secret sharing is supported
3. On clients:
   1. Communicating with the cluster is the client's responsibility (e.g. retries, failures, communicating with each node, etc).
4. On payments:
   1. A subscription-based system
   2. Payment made on chain with the client's cluster id as payload
   3. Nodes then validate the subscription transaction independently

Architecture

Milestones

Milestone 1 (17 Dec 24) - decentralisation

• feat(node): Add public/private keypair and node id (use secp256k1 and accept private key as env var) #49
• feat(node): Add cluster descriptor endpoint (eg GET /api/v1/cluster) #50
• feat(node): Add metrics support and endpoint #55
• feat(payments): Add api key telemetry (e.g. count requests) #59
• feat: restrict endpoint access to roles #57
• feat(node): Add admin endpoints to disable api during migrations (eg everything returns Unavailable)
• cicd: Add example docker-compose.yaml for running node with local or atlas based persistence
• test: Add cluster test suite (allow tests to run against 1 - 3 nodes)
• cicd: Create dashboards for monitoring cluster
• cicd: Deploy POC cluster on ec2 with atlas-based persistence

Milestone 2 (9 Jan 25) - nilql / encryption (tbc)

• feat: validate secret shares
• feat: validate deterministic queryable encryption
• feat: validate aggregations
• feat: package and publish nilql to npm and pypi

Milestone 3 (23 Jan 25) - payments

• feat(nilchain): Document/support the payment tx payload (e.g. website)
• feat(payments): Add payment validation endpoint with tx validation and cache/db for subscription status
• feat(payments): Add api-key-based rate limiting (eg 10k are free then require subscription)

Milestone other

• feat(db): support migrations as part of upgrades
• feat: persist api request counters

Open questions

1. When would an external node operator be looking to run nil-db?
2. For decentralisation do web want fixed or client-selected clusters?