Skip to content

Commit

Permalink
chore(Gossip KV): Update main README.md (hydro-project#1546)
Browse files Browse the repository at this point in the history
  • Loading branch information
rohitkulshreshtha authored Nov 7, 2024
1 parent 33c2de2 commit f7b9559
Showing 1 changed file with 57 additions and 26 deletions.
83 changes: 57 additions & 26 deletions datastores/gossip_kv/README.md
Original file line number Diff line number Diff line change
@@ -1,56 +1,87 @@
# Gossip Key-Value Store

# Architecture
A gossip-based key-value store library.
The Gossip Key-Value Store is a distributed key-value store that uses a gossip protocol to replicate data across all
members of the cluster. The key-value store is eventually consistent and is designed to be used in scenarios where
strict consistency is not a hard requirement.

## Data Model
Data stored in the key-value store is modelled as three level hierarchy `Namespace > Table > Rows`.

```
┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│ Process │ │ Process │ │ Process │ │ Process │
│ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │
│ │ User Application │ │ │ │ User Application │ │ │ │ User Application │ │ │ │ User Application │ │
│ └────────▲──┬─────────┘ │ │ └─────────▲─┬─────────┘ │ │ └─────────▲─┬─────────┘ │ │ └─────────▲─┬─────────┘ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ ┌────────┴──▼─────────┐ │ │ ┌─────────┴─▼─────────┐ │ │ ┌─────────┴─▼─────────┐ │ │ ┌─────────┴─▼─────────┐ │
│ │ Gossip KV Store │ │◄──►│ │ Gossip KV Store │ │◄──►│ │ Gossip KV Store │ │◄──►│ │ Gossip KV Store │ │
│ └─────────────────────┘ │ │ └─────────────────────┘ │ │ └─────────────────────┘ │ │ └─────────────────────┘ │
└─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
┌──────────────┐
│ Namespace │
└─┬────────────┘
│ ┌──────────────┐
└────┤ Table │
└─┬────────────┘
│ ┌──────────────┐
└────┤ Row │
└──────────────┘
```

## Data Model
TODO: Elaborate
* User Application manipulate data using client library
* Replicated to all members of the gossip cluster
* Eventually consistent
Represented as JSON, the data model looks like this:

```json
{
"sys": {
"members": {
"member_id": {
"port": 1234,
"protocol": "v1"
"protocols": [
{
"name": "gossip",
"endpoint": "..."
},
{
"name": "...",
"endpoint": "..."
}
]
}
}
},
"usr": {
"key 1": "value 1",
"key 2": "value 2"
"table_A": {
"key_1": "value_1",
"key_2": "value_2"
},
"table_B": {
"key_1": "value_1",
"key_2": "value_2"
}
}
}
```
Data in divided into two sections: A `sys` section that contains system data used by the key-value store itself. The
`usr` section contains user-defined data.
Note that JSON is used here for illustrative purposes only - it isn't the actual data format used by the key-value
store.

### Namespace
A namespace is a group of tables. There are only two namespaces in the key-value store: `sys` and `usr`. The `sys`
namespace is reserved for system data and is used by the key-value store itself. The `usr` namespace is used for user
data.

### `sys` Data
The `sys` data section contains system data / state that is required by the key-value store to do it's work.

#### Fields
### Members
The members table contains information about all the members in the cluster. Each member is identified by a unique
`member_id`.

#### Protocols
Each member exposes a set of protocols and endpoint that can be used to communicate with it. For example, every member
exposes the `gossip` protocol which is used by the key-value store to replicate data across the cluster.

### `usr` Data
The `usr` namespace is used to store user data. The data is stored in tables and rows. Each table is a collection of rows
where each row is a key-value pair. Values are stored as strings and are not interpreted by the key-value store.

# Unit Testing
A deterministic unit test suite is provided for the key-value store and the replication protocols. See the unit tests
on [server.rs](./kv/server.rs) for more.

# Deployment
## Local (Minikube) Deployment
See [Minikube Deployment](./deployment/local/README.md) for more information.
For local development and testing, the key-value store can be deployed on Minikube. See [Minikube Deployment](./deployment/local/README.md)
for more information.

## AWS Deployment
See [AWS Deployment](./deployment/aws/README.md) for more information.
See [AWS Deployment](./deployment/aws/README.md) for more information.

0 comments on commit f7b9559

Please sign in to comment.