forked from hydro-project/hydro
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore(Gossip KV): Update main README.md (hydro-project#1546)
fixes hydro-project#1531.
- Loading branch information
1 parent
33c2de2
commit f7b9559
Showing
1 changed file
with
57 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,56 +1,87 @@ | ||
# Gossip Key-Value Store | ||
|
||
# Architecture | ||
A gossip-based key-value store library. | ||
The Gossip Key-Value Store is a distributed key-value store that uses a gossip protocol to replicate data across all | ||
members of the cluster. The key-value store is eventually consistent and is designed to be used in scenarios where | ||
strict consistency is not a hard requirement. | ||
|
||
## Data Model | ||
Data stored in the key-value store is modelled as three level hierarchy `Namespace > Table > Rows`. | ||
|
||
``` | ||
┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐ | ||
│ Process │ │ Process │ │ Process │ │ Process │ | ||
│ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │ | ||
│ │ User Application │ │ │ │ User Application │ │ │ │ User Application │ │ │ │ User Application │ │ | ||
│ └────────▲──┬─────────┘ │ │ └─────────▲─┬─────────┘ │ │ └─────────▲─┬─────────┘ │ │ └─────────▲─┬─────────┘ │ | ||
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ | ||
│ ┌────────┴──▼─────────┐ │ │ ┌─────────┴─▼─────────┐ │ │ ┌─────────┴─▼─────────┐ │ │ ┌─────────┴─▼─────────┐ │ | ||
│ │ Gossip KV Store │ │◄──►│ │ Gossip KV Store │ │◄──►│ │ Gossip KV Store │ │◄──►│ │ Gossip KV Store │ │ | ||
│ └─────────────────────┘ │ │ └─────────────────────┘ │ │ └─────────────────────┘ │ │ └─────────────────────┘ │ | ||
└─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘ | ||
┌──────────────┐ | ||
│ Namespace │ | ||
└─┬────────────┘ | ||
│ ┌──────────────┐ | ||
└────┤ Table │ | ||
└─┬────────────┘ | ||
│ ┌──────────────┐ | ||
└────┤ Row │ | ||
└──────────────┘ | ||
``` | ||
|
||
## Data Model | ||
TODO: Elaborate | ||
* User Application manipulate data using client library | ||
* Replicated to all members of the gossip cluster | ||
* Eventually consistent | ||
Represented as JSON, the data model looks like this: | ||
|
||
```json | ||
{ | ||
"sys": { | ||
"members": { | ||
"member_id": { | ||
"port": 1234, | ||
"protocol": "v1" | ||
"protocols": [ | ||
{ | ||
"name": "gossip", | ||
"endpoint": "..." | ||
}, | ||
{ | ||
"name": "...", | ||
"endpoint": "..." | ||
} | ||
] | ||
} | ||
} | ||
}, | ||
"usr": { | ||
"key 1": "value 1", | ||
"key 2": "value 2" | ||
"table_A": { | ||
"key_1": "value_1", | ||
"key_2": "value_2" | ||
}, | ||
"table_B": { | ||
"key_1": "value_1", | ||
"key_2": "value_2" | ||
} | ||
} | ||
} | ||
``` | ||
Data in divided into two sections: A `sys` section that contains system data used by the key-value store itself. The | ||
`usr` section contains user-defined data. | ||
Note that JSON is used here for illustrative purposes only - it isn't the actual data format used by the key-value | ||
store. | ||
|
||
### Namespace | ||
A namespace is a group of tables. There are only two namespaces in the key-value store: `sys` and `usr`. The `sys` | ||
namespace is reserved for system data and is used by the key-value store itself. The `usr` namespace is used for user | ||
data. | ||
|
||
### `sys` Data | ||
The `sys` data section contains system data / state that is required by the key-value store to do it's work. | ||
|
||
#### Fields | ||
### Members | ||
The members table contains information about all the members in the cluster. Each member is identified by a unique | ||
`member_id`. | ||
|
||
#### Protocols | ||
Each member exposes a set of protocols and endpoint that can be used to communicate with it. For example, every member | ||
exposes the `gossip` protocol which is used by the key-value store to replicate data across the cluster. | ||
|
||
### `usr` Data | ||
The `usr` namespace is used to store user data. The data is stored in tables and rows. Each table is a collection of rows | ||
where each row is a key-value pair. Values are stored as strings and are not interpreted by the key-value store. | ||
|
||
# Unit Testing | ||
A deterministic unit test suite is provided for the key-value store and the replication protocols. See the unit tests | ||
on [server.rs](./kv/server.rs) for more. | ||
|
||
# Deployment | ||
## Local (Minikube) Deployment | ||
See [Minikube Deployment](./deployment/local/README.md) for more information. | ||
For local development and testing, the key-value store can be deployed on Minikube. See [Minikube Deployment](./deployment/local/README.md) | ||
for more information. | ||
|
||
## AWS Deployment | ||
See [AWS Deployment](./deployment/aws/README.md) for more information. | ||
See [AWS Deployment](./deployment/aws/README.md) for more information. |