Branch

A streaming toolkit with bottomless storage

• Embedded
• Single node
• Clustered
• HTTP API
• Web UI
• Kafka gateway
• S3
• GCS
• Azure blob storage
• MinIO
• R2
• Local
• Python bindings
• Elixir bindings
• Zig bindings
• C bindings

Usage

CLI

> branch --help

 Usage: branch [OPTIONS] COMMAND [ARGS]...

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                        │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation. │
│ --help                        Show this message and exit.                                                      │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ init                                                                                                         │
│ produce                                                                                                        │
│ consume                                                                                                        │
│ topics                                                                                                         │
│ server                                                                                                         │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Create your configuration file by following the prompts

> branch init
...

Create a topic

> branch topics create \
    --topic=my-topic.v1 \
    --partitions=3

Produce a new record every second

> watch -n 1 \
    'branch produce --topic=my-topic.v1 --partition=0 --message=\'{"hello": "cli!", "now": "$(date --iso-8601=ns)"}\''
...

Consume records produced to my-topic.v1

> branch consume --topic=my-topic.v1
...

Python

# my-branch.py
import branch
from datetime import datetime

msg = '{"hello": "python!", "now": datetime.now()}'
producer_record = branch.ProducerRecord(topic="my-topic.v1", value=msg)
branch.produce(producer_record)
branch.consume("my-topic.v1")

> python my-branch.py
...

Single node

> branch server start --port=9000 --ui-port=8999
...

Clustered

> branch server start --host=0.0.0.0 --port=9001 --name=my-branch-1
...
> branch server start --host=0.0.0.0 --port=9002 --name=my-branch-2
...

HTTP

> curl -X POST -d '{...}' http://localhost:9000/api/v1/produce
...

> curl http://localhost:9000/api/v1/consume
...

Web UI

> open http://localhost:8999

Development

This repository manages the dev environment as a Nix flake and requires Nix to be installed

nix develop -c $SHELL

make all

make deps.install
make deps.install/dev

make deps.freeze
make deps.freeze/dev

make deps/reqs
make deps/outdated
make deps/check

make run
make run/server

make format
make format/check

make typecheck

make lint

make test
make test/watch

make coverage

make build
make build/pyinstaller
make build/nuitka

make exec

make clean

make compose.build
make compose.up

make docker.build/deps
make docker.build/run

License

branch is released under the MIT license