Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: rename CH_ENGINE to CH_MIGRATIONS_ENGINE for consistency #7

Merged
merged 2 commits into from
Jun 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 19 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
<div align="center">

# clickhouse-migrations

![NPM Version](https://img.shields.io/npm/v/%40shini4i%2Fclickhouse-migrations)
![NPM Downloads](https://img.shields.io/npm/dm/%40shini4i%2Fclickhouse-migrations)
[![codecov](https://codecov.io/gh/shini4i/clickhouse-migrations/graph/badge.svg?token=8QWRD6EAQJ)](https://codecov.io/gh/shini4i/clickhouse-migrations)
![GitHub License](https://img.shields.io/github/license/shini4i/clickhouse-migrations)

</div>

## Install

Expand All @@ -12,7 +19,7 @@ npm install -g @shini4i/clickhouse-migrations

Create a directory, where migrations will be stored. It will be used as the value for the `--migrations-home` option (or for environment variable `CH_MIGRATIONS_HOME`).

In the directory, create migration files, which should be named like this: `1_some_text.sql`, `2_other_text.sql`, `10_more_test.sql`. What's important here is that the migration version number should come first, followed by an underscore (`_`), and then any text can follow. The version number should increase for every next migration. Please note that once a migration file has been applied to the database, it cannot be modified or removed.
In the directory, create migration files, which should be named like this: `01_some_text.sql`, `02_other_text.sql`, `10_more_test.sql`. What's important here is that the migration version number should come first, followed by an underscore (`_`), and then any text can follow. The version number should increase for every next migration. Please note that once a migration file has been applied to the database, it cannot be modified or removed.

For migrations' content should be used correct SQL ClickHouse queries. Multiple queries can be used in a single migration file, and each query should be terminated with a semicolon (;). The queries could be idempotent - for example: `CREATE TABLE IF NOT EXISTS table ...;` Clickhouse settings, that can be included at the query level, can be added like `SET allow_experimental_object_type = 1;`. For adding comments should be used `--`, `# `, `#!`.

Expand All @@ -22,39 +29,40 @@ If the database provided in the `--db` option (or in `CH_MIGRATIONS_DB`) doesn't
Usage
$ clickhouse-migrations migrate <options>

Required options
--url=<url> Clickhouse URL
(ex. https://clickhouse:8123)
Required flags
--url=<url> Clickhouse URL (ex. https://clickhouse:8123)
--user=<name> Username
--password=<password> Password
--db=<name> Database name
--migrations-home=<dir> Migrations' directory
--engine=<engine> The engine to use for DB creation (optional)


Optional flags
--password=<password> Password
--engine=<engine> The engine to use for DB creation

Environment variables
Instead of options can be used environment variables.
CH_MIGRATIONS_URL Clickhouse hostname (--url)
CH_MIGRATIONS_USER Username (--user)
CH_MIGRATIONS_PASSWORD Password (--password)
CH_MIGRATIONS_DB Database name (--db)
CH_MIGRATIONS_HOME Migrations' directory (--migrations-home)
CH_ENGINE The engine to use for DB creation (optional)
CH_MIGRATIONS_ENGINE The engine to use for DB creation (optional)

CLI examples
clickhouse-migrations migrate --url=http://localhost:8123
--user=default --password='' --db=analytics
--user=default -db=analytics
--migrations-home=/app/clickhouse/migrations

clickhouse-migrations migrate
```

Migration file example:
```
-- an example of migration file 1_init.sql
-- an example of migration file 01_init.sql

SET allow_experimental_object_type = 1;

CREATE TABLE IF NOT EXISTS events (
event JSON
);
```
```
14 changes: 7 additions & 7 deletions src/migrate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ const log = (type: 'info' | 'error' = 'info', message: string, error?: string) =
}
};

const connect = (url: string, username: string, password: string, db_name?: string): ClickHouseClient => {
const connect = (url: string, username: string, password?: string, db_name?: string): ClickHouseClient => {
const db_params: ClickhouseDbParams = {
url,
username,
Expand All @@ -28,7 +28,7 @@ const connect = (url: string, username: string, password: string, db_name?: stri
return createClient(db_params);
};

const create_db = async (url: string, username: string, password: string, db_name: string, engine?: string): Promise<void> => {
const create_db = async (url: string, username: string, db_name: string, password?: string, engine?: string): Promise<void> => {
const client = connect(url, username, password);

let q = `CREATE DATABASE IF NOT EXISTS "${db_name}"`;
Expand Down Expand Up @@ -220,13 +220,13 @@ const migration = async (
migrations_home: string,
url: string,
username: string,
password: string,
db_name: string,
password?: string,
engine?: string
): Promise<void> => {
const migrations = get_migrations(migrations_home);

await create_db(url, username, password, db_name, engine);
await create_db(url, username, db_name, password, engine);

const client = connect(url, username, password, db_name);

Expand All @@ -247,12 +247,12 @@ const migrate = () => {
.description('Apply migrations.')
.requiredOption('--url <url>', 'Clickhouse URL (ex: http://clickhouse:8123)', process.env.CH_MIGRATIONS_URL)
.requiredOption('--user <name>', 'Username', process.env.CH_MIGRATIONS_USER)
.requiredOption('--password <password>', 'Password', process.env.CH_MIGRATIONS_PASSWORD)
.requiredOption('--db <name>', 'Database name', process.env.CH_MIGRATIONS_DB)
.requiredOption('--migrations-home <dir>', "Migrations' directory", process.env.CH_MIGRATIONS_HOME)
.option('--engine <name>', 'Engine name', process.env.CH_ENGINE)
.option('--password <password>', 'Password', process.env.CH_MIGRATIONS_PASSWORD)
.option('--engine <name>', 'Engine name', process.env.CH_MIGRATIONS_ENGINE)
.action(async (options: CliParameters) => {
await migration(options.migrationsHome, options.url, options.user, options.password, options.db, options.engine);
await migration(options.migrationsHome, options.url, options.user, options.db, options.password, options.engine);
});

program.parse();
Expand Down
4 changes: 2 additions & 2 deletions src/types/cli.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ type ClickhouseDbParams = {
max_open_connections?: number;
compression?: { response?: boolean; request?: boolean };
username: string;
password: string;
password?: string;
application?: string;
database?: string;
clickhouse_settings?: ClickHouseSettings;
Expand All @@ -31,8 +31,8 @@ type CliParameters = {
migrationsHome: string;
url: string;
user: string;
password: string;
db: string;
password?: string;
engine?: string;
};

Expand Down
2 changes: 1 addition & 1 deletion tests/db.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ describe('Migration tests', () => {
const execSpy = jest.spyOn(createClient1, 'exec');
const insertSpy = jest.spyOn(createClient1, 'insert');

await migration('tests/migrations/one', 'http://sometesthost:8123', 'default', '', 'analytics');
await migration('tests/migrations/one', 'http://sometesthost:8123', 'default', 'analytics');

expect(execSpy).toHaveBeenCalledTimes(3);
expect(querySpy).toHaveBeenCalledTimes(1);
Expand Down