From 7bc034a1899d005f3587f29583dd0671f0ba691d Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 11:37:43 +0100 Subject: [PATCH 01/31] chore: space to validate initial build. --- api/add_compression_policy.md | 1 + 1 file changed, 1 insertion(+) diff --git a/api/add_compression_policy.md b/api/add_compression_policy.md index 74210e4142..520cd4f885 100644 --- a/api/add_compression_policy.md +++ b/api/add_compression_policy.md @@ -89,3 +89,4 @@ SELECT add_compression_policy('cpu_weekly', INTERVAL '8 weeks'); [compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ [set_integer_now_func]: /api/:currentVersion:/hypertable/set_integer_now_func [informational-views]: /api/:currentVersion:/informational-views/jobs/ + From b7e966e98770afb134a8b561f660e8a2dfce1342 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 12:12:03 +0100 Subject: [PATCH 02/31] chore: add first hypercore files. --- api/hypercore/add_columnstore_policy.md | 106 ++++++++++++++++++++ api/hypercore/index.md | 122 ++++++++++++++++++++++++ api/page-index/page-index.js | 12 +++ 3 files changed, 240 insertions(+) create mode 100644 api/hypercore/add_columnstore_policy.md create mode 100644 api/hypercore/index.md diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md new file mode 100644 index 0000000000..4ea13cd029 --- /dev/null +++ b/api/hypercore/add_columnstore_policy.md @@ -0,0 +1,106 @@ +--- +api_name: add_columnstore_policy() +excerpt: Set a policy to automatically move chunks in a hypertable to the columnstore when they reach a given age. +topics: [columnstore, hypercore, jobs] +keywords: [columnstore, hypercore, policies] +tags: [scheduled jobs, background jobs, automation framework] +products: [cloud, self_hosted] +api: + license: community + type: function +--- + +# add_columnstore_policy() + +Create a [job][job] that automatically moves chunks in a hypertable to the columnstore after a +specific time interval. + +You enable the columnstore a hypertable or continuous aggregate before you create a columnstore policy. +You do this by calling `ALTER TABLE` for hypertables and `ALTER MATERIALIZED VIEW` for continuous aggregates. + +To view the policies that you set or the policies that already exist, +see [informational views][informational-views], to remove a policy, see [remove_columnstore_policy][remove_columnstore_policy]. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +To create a columnstore job: + + + +1. **Enable columnstore** + + * [Use `ALTER TABLE` for a hypertable][compression_alter-table] + ```sql + ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol'); + ``` + * [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate] + ```sql + ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ``` + +1. **Add a policy to move chunks to the columnstore at a specific time interval** + + For example: + + * 60 days after the data was added to the table: + ``` sql + SELECT add_columnstore_policy('stocks_real_time', after => INTERVAL '60d'); + ``` + * 3 months prior to the moment you run the query: + + ``` sql + SELECT add_columnstore_policy('stocks_real_time', created_before => INTERVAL '3 months'); + ``` + * With an integer-based time column: + + ``` sql + SELECT add_columnstore_policy('table_with_bigint_time', BIGINT '600000'); + ``` + * Older than eight weeks: + + ``` sql + SELECT add_columnstore_policy('cpu_weekly', INTERVAL '8 weeks'); + ``` + +1. **View the policies that you set or the policies that already exist** + + ``` sql + SELECT * FROM timescaledb_information.jobs + WHERE proc_name='policy_compression'; + ``` + See [timescaledb_information.jobs][informational-views]. + + + +## Arguments + +Calls to `add_columnstore_policy` require either `after` or `created_before`, but cannot have both. + + + + +| Name | Type | Default | Required | Description | +|--|--|--|--|--| +| `hypertable` |REGCLASS| - | ✔ | Name of the hypertable or continuous aggregate to run this [job][job] on.| +| `after` |INTERVAL or INTEGER|- | ✖ | Add chunks containing data older than `now - {after}::interval` to the columnstore.
Use an object type that matchs the time column type in `hypertable`:
  • TIMESTAMP, TIMESTAMPTZ, or DATE: use an INTERVAL type.
  • Integer-based timestamps : set an integer type using the [integer_now_func][set_integer_now_func].
`after` is mutually exclusive with `created_before`. | +| `created_before` |INTERVAL| NULL | ✖ | Add chunks with a creation time of `now() - created_before` to the columnstore.
`created_before` is
  • Not supported for continuous aggregates.
  • Mutually exclusive with `after`.
| +| `schedule_interval` |INTERVAL| 12 hours when [chunk_time_interval][chunk_time_interval] >= `1 day` for `hypertable`. Otherwise `chunk_time_interval` / `2`. | ✖ | Set the interval between the finish time of the last execution of this policy and the next start.| +| `initial_start` |TIMESTAMPTZ| The interval from the finish time of the last execution to the [next_start][next-start].| ✖| Set the time this job is first run. This is also the time that `next_start` is calculated from.| +| `timezone` |TEXT| UTC. However, daylight savings time(DST) changes may shift this alignment. | ✖ | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`.| +| `if_not_exists` |BOOLEAN| `false` | ✖ | Set to `true` so this job fails with a warning rather than an error if a columnstore policy already exists on `hypertable` | + + + + + + +[compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/ +[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ +[set_integer_now_func]: /api/:currentVersion:/hypertable/set_integer_now_func +[informational-views]: /api/:currentVersion:/informational-views/jobs/ +[chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/ +[next-start]: /api/:currentVersion:/informational-views/jobs/#arguments +[job]: /api/:currentVersion:/actions/add_job/ +[remove_columnstore_policy]: /api/:currentVersion:/hypercore/remove_columnstore_policy/ diff --git a/api/hypercore/index.md b/api/hypercore/index.md new file mode 100644 index 0000000000..61fe9a4218 --- /dev/null +++ b/api/hypercore/index.md @@ -0,0 +1,122 @@ +--- +title: Hypercore +excerpt: Reference information about the TimescaleDB hybrid row-columnar storage engine +keywords: [hypercore] +tags: [hypercore] +products: [cloud, self_hosted] +api: + license: community +--- + +# Hypercore + +Hypercore is the $TIMESCALE_DB hybrid row-columnar storage engine, designed specifically for +real-time analytics and powered by time-series data. The advantage of hypercore is its ability +to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables +$CLOUD_LONG to deliver the best of both worlds, solving the key challenges in real-time analytics. + +Hypercore’s hybrid approach combines the benefits of row-oriented and column-oriented formats +in each $CLOUD_LONG service: + +- **Fast ingest with rowstore**: new data is initially written to the rowstore, which is optimized for + high-speed inserts and updates. + +- **Efficient analytics with columnstore**: you create [columnstore_policies][hypercore_workflow] + that automatically move your data to the columnstore as it _cools_. In columstore conversion, hypertable + chunks are compressed and organized for efficient, large-scale queries more suitable for analytics. + +- **Full mutability with transactional semantics**: regardless of where data is stored, + hypercore provides full ACID support. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Hypercore workflow + +Best practice for using Hypercore is to: + + + +1. **Enable columnstore** + + * [Use `ALTER TABLE` for a hypertable][alter_table_hypercore] + ```sql + ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol'); + ``` + * [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate] + ```sql + ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ``` + +1. **Add a policy to move chunks to the columnstore at a specific time interval** + + For example, 60 days after the data was added to the table: + ``` sql + SELECT add_columnstore_policy('older_stock_prices', after => INTERVAL '60d'); + ``` + See [add_columnstore_policy][add_columnstore_policy]. + +1. **View the policies that you set or the policies that already exist** + + ``` sql + SELECT * FROM timescaledb_information.jobs + WHERE proc_name='policy_compression'; + ``` + See [timescaledb_information.jobs][informational-views]. + +1. **Pause a columnstore policy** + + ``` sql + SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'stocks_real_time' + + -- Select the JOB_ID from the results + + SELECT alter_job(JOB_ID, scheduled => false); + ``` + See [alter_job][alter_job]. + +1. **Restart a columnstore policy** + ``` sql + SELECT alter_job(JOB_ID, scheduled => true); + ``` + See [alter_job][alter_job]. + +1. **Remove a columnstore policy** + ``` sql + SELECT remove_columnstore_policy('older_stock_prices'); + ``` + See [remove_columnstore_policy][remove_columnstore_policy]. +1. **Disable columnstore** + + If your table has chunks in the columnstore, you have to + [convert the chunks back to the rowstore][convert_to_rowstore] before you disable the columnstore. + ``` sql + ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = false); + ``` + See [alter_table_hypercore][alter_table_hypercore]. + + + +You can also [convert_to_columnstore][convert_to_columnstore] and [convert_to_rowstore][convert_to_rowstore] manually +for more fine-grained control over your data. + +## Limitations + +Chunks in the columnstore have the following limitations: + +* `ROW LEVEL SECURITY` is not supported on chunks in the columnstore. +* To add unique constraints on chunks in the columnstore [convert_the chunk to rowstore][convert_to_rowstore], + add the constraints to your data, then [convert the chunk back to the rowstore][convert_to_columnstore]. +* [SkipScan][skipscan] does not currently work on chunks in the columnstore. + + +[alter_table_hypercore]: /api/:currentVersion:/hypercore/alter_table/ +[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ +[convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ +[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ +[convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ +[informational-views]: /api/:currentVersion:/informational-views/jobs/ +[skipscan]: /use-timescale/:currentVersion:/query-data/skipscan/ +[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ +[hypercore_workflow]: /api/:currentVersion:/hypercore/#hypercore-workflow +[alter_job]: /api/:currentVersion:/actions/alter_job/ +[remove_columnstore_policy]: /api/:currentVersion:/hypercore/remove_columnstore_policy/ diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 408913ed06..c301d6a965 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -116,6 +116,18 @@ module.exports = [ }, ], }, + { + title: "Hypercore", + excerpt: "Seamlessly switch between fast row-oriented storage and efficient column-oriented storage", + href: "hypercore", + children: [ + { + title: "add_columnstore_policy", + href: "add_columnstore_policy", + excerpt: "Automatically move chunks in a hypertable to the columnstore after a specific time interval.", + }, + ], + }, { title: "Distributed hypertables", type: "directory", From 3899a6ee59013ae44195c28295d950afb6ec9919 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 12:26:44 +0100 Subject: [PATCH 03/31] chore: add api ref files and alter_table to the build. --- api/hypercore/alter_table.md | 77 +++++++++++++ api/hypercore/chunk_columnstore_settings.md | 52 +++++++++ api/hypercore/chunk_columnstore_stats.md | 106 ++++++++++++++++++ api/hypercore/convert_to_columnstore.md | 52 +++++++++ api/hypercore/convert_to_rowstore.md | 82 ++++++++++++++ .../hypertable_columnstore_settings.md | 59 ++++++++++ api/hypercore/hypertable_columnstore_stats.md | 79 +++++++++++++ api/hypercore/remove_columnstore_policy.md | 46 ++++++++ api/page-index/page-index.js | 5 + 9 files changed, 558 insertions(+) create mode 100644 api/hypercore/alter_table.md create mode 100644 api/hypercore/chunk_columnstore_settings.md create mode 100644 api/hypercore/chunk_columnstore_stats.md create mode 100644 api/hypercore/convert_to_columnstore.md create mode 100644 api/hypercore/convert_to_rowstore.md create mode 100644 api/hypercore/hypertable_columnstore_settings.md create mode 100644 api/hypercore/hypertable_columnstore_stats.md create mode 100644 api/hypercore/remove_columnstore_policy.md diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md new file mode 100644 index 0000000000..ea55c24dcb --- /dev/null +++ b/api/hypercore/alter_table.md @@ -0,0 +1,77 @@ +--- +api_name: ALTER TABLE (Hypercore) +excerpt: Enable the columnstore for a hypertable. +topics: [columnstore, hypercore] +keywords: [columnstore, hypercore] +tags: [settings, hypertables, alter, change] +api: + license: community + type: command +products: [cloud, self_hosted] +--- + +# ALTER TABLE (Hypercore) + +Enable the columnstore for a hypertable. + +After you have enabled the columnstore, either: +- [add_columnstore_policy][add_columnstore_policy]: create a [job][job] that automatically moves chunks in a hypertable to the columnstore at a + specific time interval. +- [convert_to_columnstore][convert_to_columnstore]: manually add a specific chunk in a hypertable to the columnstore. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +- **Configure a hypertable that ingests device data to use the columnstore**: + + In this example, the `metrics` hypertable is often queried about a specific device or set of devices. + Segment the hypertable by `device_id` to improve query performance. + + ```sql + ALTER TABLE metrics SET (timescaledb.enable_columnstore, timescaledb.orderby = 'time DESC', timescaledb.segmentby = 'device_id'); + ``` + +- **Specify the chunk interval without changing other columnstore settings**: + + - Set the time interval when chunks are added to the columnstore: + + ```sql + ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '24 hours'); + ``` + + - To disable the option you set previously, set the interval to 0: + + ```sql + ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '0'); + ``` + +## Arguments + +The syntax is: + +``` sql +ALTER TABLE SET (timescaledb.enable_columnstore, + timescaledb.orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', + timescaledb.segmentby = ' [, ...]', + timescaledb.compress_chunk_time_interval='interval' +); +``` + +| Name | Type | Default | Required | Description | +|--|--|------------------------------------------------------|--|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|`table_name`|TEXT| - | ✖ | The hypertable to enable columstore for. | +|`timescaledb.enable_columnstore`|BOOLEAN| true | ✖ | Enable columnstore. | +|`timescaledb.orderby`|TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | +|`timescaledb.segmentby`|TEXT| No segementation by column.| ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | +|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | +|`timescaledb.compress_chunk_time_interval`|TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | +|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | + + + +[chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/ +[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ +[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ +[convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ +[job]: /api/:currentVersion:/actions/add_job/ diff --git a/api/hypercore/chunk_columnstore_settings.md b/api/hypercore/chunk_columnstore_settings.md new file mode 100644 index 0000000000..6fb484788f --- /dev/null +++ b/api/hypercore/chunk_columnstore_settings.md @@ -0,0 +1,52 @@ +--- +api_name: timescaledb_information.chunk_columnstore_settings +excerpt: Get information about settings on each chunk in the columnstore +topics: [information, columnstore, hypercore, chunk] +keywords: [columnstore, hypercore, chunk, information] +tags: [chunk, columnstore settings] +api: + license: community + type: view +--- + +# timescaledb_information.chunk_columnstore_settings + +Retrieve information about each chunk in the columnstore. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +* Show settings for all chunks in the columnstore: + + ```sql + SELECT * FROM timescaledb_information.chunk_columnstore_settings + ``` + Returns: + ```sql + hypertable | chunk | segmentby | orderby + ------------+-------+-----------+--------- + measurements | _timescaledb_internal._hyper_1_1_chunk| | "time" DESC + ``` + +* Find all chunk columnstore settings for a specific hypertable: + + ```sql + SELECT * FROM timescaledb_information.chunk_columnstore_settings WHERE hypertable::TEXT LIKE 'metrics'; + ``` + Returns: + ```sql + hypertable | chunk | segmentby | orderby + ------------+-------+-----------+--------- + metrics | _timescaledb_internal._hyper_2_3_chunk | metric_id | "time" + ``` + +## Returns + +| Name | Type | Default | Required | Description | +|--|--|--|--|--| +|`hypertable`|`REGCLASS`|-|✖| The name of a hypertable in the columnstore | +|`chunk`|`REGCLASS`|-|✖| The name of a chunk in `hypertable` | +|`segmentby`|`TEXT`|-|✖| A list of columns used to segment `hypertable` | +|`orderby`|`TEXT`|-|✖| A list of columns used to order data in `hypertable`. Along with ordering and NULL ordering information. IAIN, I don't understand the second sentence. | + diff --git a/api/hypercore/chunk_columnstore_stats.md b/api/hypercore/chunk_columnstore_stats.md new file mode 100644 index 0000000000..a0a37ccc14 --- /dev/null +++ b/api/hypercore/chunk_columnstore_stats.md @@ -0,0 +1,106 @@ +--- +api_name: chunk_columnstore_stats() +excerpt: Get statistics about chunks in the columnstore +topics: [columnstore, hypercore] +keywords: [columnstore, hypercore, statistics, chunks, information] +tags: [disk space, schemas, size] +api: + license: community + type: function +--- + +# chunk_columnstore_stats() Community + +Get chunk-specific statistics related to hypercore. + +`chunk_columnstore_stats` returns the size of chunks in the columnstore, these values are computed when you call either: +- [add_columnstore_policy][add_columnstore_policy]: create a [job][job] that automatically moves chunks in a hypertable to the columnstore at a + specific time interval. +- [convert_to_columnstore][convert_to_columnstore]: manually add a specific chunk in a hypertable to the columnstore. + + +Inserting into a chunk in the columnstore does not change the chunk size. For more information about how to compute +chunk sizes, see [chunks_detailed_size][chunks_detailed_size]. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +- **Show the status of the first two chunks in the `conditions` hypertable**: + ```sql + SELECT * FROM chunk_columnstore_stats('conditions') + ORDER BY chunk_name LIMIT 2; + ``` + Returns: + ```sql + -[ RECORD 1 ]------------------+---------------------- + chunk_schema | _timescaledb_internal + chunk_name | _hyper_1_1_chunk + compression_status | Uncompressed + before_compression_table_bytes | + before_compression_index_bytes | + before_compression_toast_bytes | + before_compression_total_bytes | + after_compression_table_bytes | + after_compression_index_bytes | + after_compression_toast_bytes | + after_compression_total_bytes | + node_name | + -[ RECORD 2 ]------------------+---------------------- + chunk_schema | _timescaledb_internal + chunk_name | _hyper_1_2_chunk + compression_status | Compressed + before_compression_table_bytes | 8192 + before_compression_index_bytes | 32768 + before_compression_toast_bytes | 0 + before_compression_total_bytes | 40960 + after_compression_table_bytes | 8192 + after_compression_index_bytes | 32768 + after_compression_toast_bytes | 8192 + after_compression_total_bytes | 49152 + node_name | + ``` + +- **Use `pg_size_pretty` to return a more human friendly format**: + + ```sql + SELECT pg_size_pretty(after_compression_total_bytes) AS total + FROM chunk_columnstore_stats('conditions') + WHERE compression_status = 'Compressed'; + ``` + Returns: + ```sql + -[ RECORD 1 ]--+------ + total | 48 kB + ``` + + +## Arguments + +| Name | Type | Default | Required | Description | +|--|--|--|--|--| +|`hypertable`|`REGCLASS`|-|✖| The name of a hypertable | + + +## Returns + +|Column|Type| Description | +|-|-|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|`chunk_schema`|TEXT| Schema name of the chunk. | +|`chunk_name`|TEXT| Name of the chunk. | +|`compression_status`|TEXT| Current compression status of the chunk. | +|`before_compression_table_bytes`|BIGINT| Size of the heap before compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_index_bytes`|BIGINT| Size of all the indexes before compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_toast_bytes`|BIGINT| Size the TOAST table before compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_total_bytes`|BIGINT| Size of the entire chunk table (`before_compression_table_bytes` + `before_compression_index_bytes` + `before_compression_toast_bytes`) before compression. Returns `NULL` if `compression_status` == `Uncompressed`.| +|`after_compression_table_bytes`|BIGINT| Size of the heap after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`after_compression_index_bytes`|BIGINT| Size of all the indexes after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`after_compression_toast_bytes`|BIGINT| Size the TOAST table after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`after_compression_total_bytes`|BIGINT| Size of the entire chunk table (`after_compression_table_bytes` + `after_compression_index_bytes `+ `after_compression_toast_bytes`) after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`node_name`|TEXT| **DEPRECATED**: nodes the chunk is located on, applicable only to distributed hypertables. | + + +[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ +[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ +[job]: /api/:currentVersion:/actions/add_job/ +[chunks_detailed_size]: /api/:currentVersion:/hypertable/chunks_detailed_size/ diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md new file mode 100644 index 0000000000..d3111b9990 --- /dev/null +++ b/api/hypercore/convert_to_columnstore.md @@ -0,0 +1,52 @@ +--- +api_name: convert_to_columnstore() +excerpt: Manually add a chunk to thee columnstore +topics: [columnstore, rowstore, hypercore] +keywords: [columnstore, rowstore, hypercore] +tags: [chunks, hypercore] +api: + license: community + type: function +--- + +# convert_to_columnstore() Community + +Manually control the exact time you move a specific chunk in a hypertable to the columnstore. + +Although `convert_to_columnstore` gives you more more fine grained control, best practice is to use +[`add_columnstore_policy`][add_columnstore_policy]. You can also add chunks to the columnstore at a specific time +[running the job associated with your columnstore policy][run-job] manually. + +To move a chunk from the columnstore back to the rowstore, use [`convert_to_rowstore`][convert_to_rowstore]. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +Compress a single chunk. + +``` sql +SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); +``` + +To retrieve the chunks belonging to a hypertable, call [`show_chunks`](/api/latest/hypertable/show_chunks/). + + +## Arguments + +| Name | Type | Default | Required | Description | +|----------------------|--|---------|--|----------------------------------------------------------------------------------------------------------------| +| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the columnstore. | +| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the columnstore. | +| `recompress` | BOOLEAN | `false` |✖| Set to `true` to add a chunk that had more data inserted after being added to the columnstore. | + +## Returns + +|Column| Type | Description | +|-|-------------------------------------------------|-------------------------------------------------| +| `convert_to_columnstore` | REGCLASS | The name of the chunk added to the columnstore. | + + +[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ +[run-job]: /api/:currentVersion:/actions/run_job/ +[convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md new file mode 100644 index 0000000000..99f2e760fd --- /dev/null +++ b/api/hypercore/convert_to_rowstore.md @@ -0,0 +1,82 @@ +--- +api_name: convert_to_rowstore() +excerpt: Move a chunk from the columnstore to the rowstore +topics: [columnstore, hypercore] +keywords: [columnstore, hypercore, rowstore, chunks, backfilling] +api: + license: community + type: function +--- + +# convert_to_rowstore() Community + +Move a chunk of data from the columnstore to the rowstore. + +If you need to modify or add a lot of data to a chunk in the columnstore, best practice is to stop +any [jobs][job] moving chunks to the columnstore, convert the chunk back to the rowstore, then modify the +data. After the update, [convert the chunk to the columnstore][convert_to_columnstore] and restart the jobs. +This workflow is especially useful if you need to backfill old data. + + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +To modify or add a lot of data to a chunk: + + + +1. **Stop the [jobs][alter_job] that are automatically adding chunks to the columnstore** + ``` sql + SELECT alter_job(JOB_ID, scheduled => false); + ``` + You retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view. + +1. **Convert the chunks to update back to the rowstore** + + - Convert single chunk: + + ``` sql + SELECT convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); + ``` + + - Convert all chunks in a hypertable named `metrics`: + + ``` sql + SELECT convert_to_rowstore(c, true) FROM show_chunks('metrics') c; + ``` + +1. **[Update the data][insert] in the chunk you added to the rowstore** + + Best practice is to structure your INSERT statement to include appropriate + partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk: + + ``` sql + INSERT INTO metrics (time, value) + VALUES ('2025-01-01T00:00:00', 42); + ``` + +1. **Convert the updated chunks back to the columnstore** + ``` sql + SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); + ``` + +1. **Restart the [jobs][alter_job] that are automatically adding chunks to the columnstore** + ``` sql + SELECT alter_job(JOB_ID, scheduled => true); + ``` + + + +## Arguments + +| Name | Type | Default | Required | Description | +|--|--|--|--|------------------------------------------------------------------------------------------------------------| +|`chunk`|`REGCLASS`|-|✖| Name of the chunk to be moved to the rowstore. | +|`if_compressed`|`BOOLEAN`|`true`|✔| Set to `false` so this job fails with an error rather than an warning if `chunk` is not in the columnstore | + +[job]: /api/:currentVersion:/actions/ +[alter_job]: /api/:currentVersion:/actions/alter_job/ +[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ +[informational-views]: /api/:currentVersion:/informational-views/jobs/ +[insert]: /use-timescale/:currentVersion:/write-data/insert/ diff --git a/api/hypercore/hypertable_columnstore_settings.md b/api/hypercore/hypertable_columnstore_settings.md new file mode 100644 index 0000000000..2dd6598f48 --- /dev/null +++ b/api/hypercore/hypertable_columnstore_settings.md @@ -0,0 +1,59 @@ +--- +api_name: timescaledb_information.hypertable_columnstore_settings +excerpt: Get information about columnstore settings for all hypertables +topics: [information, hypercore, columnstore, hypertable] +keywords: [columnstore, hypercore, hypertable, information] +tags: [hypertable columnstore, columnstore settings] +api: + license: community + type: view +--- + +# timescaledb_information.hypertable_columnstore_settings + +Retrieve information about the settings for each hypertable in the columnstore. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +- **Show columnstore settings for all hypertables**: + + ```sql + SELECT * FROM timescaledb_information.hypertable_columnstore_settings' + ``` + Returns: + ```sql + hypertable | measurements + segmentby | + orderby | "time" DESC + compress_interval_length | + ``` + +- **Retrieve columnstore settings for a specific hypertable**: + + ```sql + SELECT * FROM timescaledb_information.hypertable_columnstore_settings WHERE hypertable::TEXT LIKE 'metrics'; + ``` + Returns: + ```sql + hypertable | metrics + segmentby | metric_id + orderby | "time" + compress_interval_length | + ``` + +## Returns + +|Name|Type| Description | +|-|-|---------------------------------------------------------------------------------------------------------------------| +|`hypertable`|`REGCLASS`| A hypertable which has the [columnstore enabled][compression_alter-table]. | +|`segmentby`|`TEXT`| The list of columns used to segment data | +|`orderby`|`TEXT`| List of columns used to order the data, along with ordering and NULL ordering information | +|`compress_interval_length`|`TEXT`| Interval used for [rolling up chunks during compression][rollup-compression] IAIN, update when main doc is written. | + + + +[rollup-compression]: /use-timescale/:currentVersion:/compression/manual-compression/#roll-up-uncompressed-chunks-when-compressing +[compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/ + diff --git a/api/hypercore/hypertable_columnstore_stats.md b/api/hypercore/hypertable_columnstore_stats.md new file mode 100644 index 0000000000..e8806e1f6e --- /dev/null +++ b/api/hypercore/hypertable_columnstore_stats.md @@ -0,0 +1,79 @@ +--- +api_name: hypertable_columnstore_stats() +excerpt: Get columnstore statistics related to the columnstore +topics: [columnstore, hypercore] +keywords: [columnstore, hypercore, hypertables, information] +tags: [statistics, size] +api: + license: community + type: function +--- + +# hypertable_columnstore_stats() Community + +Get statistics related to compression in the columnstore. All sizes are in bytes. + +For more information about using hypertables, including chunk size partitioning, +see the [hypertable section][hypertable-docs]. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +- **Show the compression status of the `conditions` hypertable**: + + ```sql + SELECT * FROM hypertable_columnstore_stats('conditions'); + ``` + Returns: + ```sql + -[ RECORD 1 ]------------------+------ + total_chunks | 4 + number_compressed_chunks | 1 + before_compression_table_bytes | 8192 + before_compression_index_bytes | 32768 + before_compression_toast_bytes | 0 + before_compression_total_bytes | 40960 + after_compression_table_bytes | 8192 + after_compression_index_bytes | 32768 + after_compression_toast_bytes | 8192 + after_compression_total_bytes | 49152 + node_name | + ``` + +- **Use `pg_size_pretty` get the output in a more human friendly format**: + + ```sql + SELECT pg_size_pretty(after_compression_total_bytes) as total + FROM hypertable_columnstore_stats('conditions'); + ``` + Returns: + ```sql + -[ RECORD 1 ]--+------ + total | 48 kB + ``` + +## Arguments + +|Name|Type|Description| +|-|-|-| +|`hypertable`|REGCLASS|Hypertable to show statistics for| + +## Returns + +|Column|Type|Description| +|-|-|-| +|`total_chunks`|BIGINT|The number of chunks used by the hypertable. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`number_compressed_chunks`|INTEGER|The number of chunks used by the hypertable that are currently compressed. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_table_bytes`|BIGINT|Size of the heap before compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_index_bytes`|BIGINT|Size of all the indexes before compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_toast_bytes`|BIGINT|Size the TOAST table before compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`before_compression_total_bytes`|BIGINT|Size of the entire table (`before_compression_table_bytes` + `before_compression_index_bytes` + `before_compression_toast_bytes`) before compression. Returns `NULL` if `compression_status` == `Uncompressed`.| +|`after_compression_table_bytes`|BIGINT|Size of the heap after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`after_compression_index_bytes`|BIGINT|Size of all the indexes after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`after_compression_toast_bytes`|BIGINT|Size the TOAST table after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`after_compression_total_bytes`|BIGINT|Size of the entire table (`after_compression_table_bytes` + `after_compression_index_bytes `+ `after_compression_toast_bytes`) after compression. Returns `NULL` if `compression_status` == `Uncompressed`. | +|`node_name`|TEXT|nodes on which the hypertable is located, applicable only to distributed hypertables. Returns `NULL` if `compression_status` == `Uncompressed`. | + +[hypertable-docs]: /use-timescale/:currentVersion:/hypertables/ + diff --git a/api/hypercore/remove_columnstore_policy.md b/api/hypercore/remove_columnstore_policy.md new file mode 100644 index 0000000000..59378e93bc --- /dev/null +++ b/api/hypercore/remove_columnstore_policy.md @@ -0,0 +1,46 @@ +--- +api_name: remove_columnstore_policy() +excerpt: Remove a columnstore policy from a hypertable +topics: [hypercore, columnstore, jobs] +keywords: [hypercore, columnstore, policies, remove] +tags: [delete, drop] +api: + license: community + type: function +--- + +# remove_columnstore_policy() + +Remove a columnstore policy from a hypertable or continuous aggregate. + +To restart automatic chunk migration to the columnstore, you need to call +[add_columnstore_policy][add_columnstore_policy] again. + +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + +## Samples + +You see the columnstore policies in the [informational views][informational-views]. + +- **Remove the columnstore policy from the `cpu` table**: + + ``` sql + SELECT remove_columnstore_policy('cpu'); + ``` + +- **Remove the columnstore policy from the `cpu_weekly` continuous aggregate**: + + ``` sql + SELECT remove_columnstore_policy('cpu_weekly'); + ``` + +## Arguments + +| Name | Type | Default | Required | Description | +|--|--|--|--|-| +|`hypertable`|REGCLASS|-|✔| Name of the hypertable or continuous aggregate to remove the policy from| +| `if_exists` | BOOLEAN | `false` |✖| Set to `true` so this job fails with a warning rather than an error if a columnstore policy does not exist on `hypertable` | + + +[informational-views]: /api/:currentVersion:/informational-views/jobs/ +[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index c301d6a965..84b99864fb 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -121,6 +121,11 @@ module.exports = [ excerpt: "Seamlessly switch between fast row-oriented storage and efficient column-oriented storage", href: "hypercore", children: [ + { + title: "ALTER TABLE", + href: "alter_table", + excerpt: "Enable the columnstore for a hypertable.", + }, { title: "add_columnstore_policy", href: "add_columnstore_policy", From bd9946cf4299a165e895ef3829f4267b2e34e2e7 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 12:57:21 +0100 Subject: [PATCH 04/31] chore: add remove_columnstore_policy to the build. --- api/page-index/page-index.js | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 84b99864fb..7960662eff 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -131,6 +131,11 @@ module.exports = [ href: "add_columnstore_policy", excerpt: "Automatically move chunks in a hypertable to the columnstore after a specific time interval.", }, + { + title: "remove_columnstore_policy", + href: "remove_columnstore_policy", + excerpt: "Remove the compression policy", + }, ], }, { From 5ad76e20d0654bf6d071d0c1b7c812fd5c1f5018 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 14:03:41 +0100 Subject: [PATCH 05/31] chore: add convert_to_columnstore to the build. --- _partials/_deprecated_2_18_0.md | 1 + _partials/_since_2_18_0.md | 1 + api/hypercore/add_columnstore_policy.md | 3 ++- api/hypercore/alter_table.md | 3 ++- api/hypercore/chunk_columnstore_settings.md | 3 ++- api/hypercore/chunk_columnstore_stats.md | 3 ++- api/hypercore/convert_to_columnstore.md | 3 ++- api/hypercore/convert_to_rowstore.md | 3 ++- api/hypercore/hypertable_columnstore_settings.md | 3 ++- api/hypercore/hypertable_columnstore_stats.md | 3 ++- api/hypercore/index.md | 3 ++- api/hypercore/remove_columnstore_policy.md | 3 ++- api/page-index/page-index.js | 5 +++++ 13 files changed, 27 insertions(+), 10 deletions(-) create mode 100644 _partials/_deprecated_2_18_0.md create mode 100644 _partials/_since_2_18_0.md diff --git a/_partials/_deprecated_2_18_0.md b/_partials/_deprecated_2_18_0.md new file mode 100644 index 0000000000..a4d4c0aec0 --- /dev/null +++ b/_partials/_deprecated_2_18_0.md @@ -0,0 +1 @@ +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** diff --git a/_partials/_since_2_18_0.md b/_partials/_since_2_18_0.md new file mode 100644 index 0000000000..a4d4c0aec0 --- /dev/null +++ b/_partials/_since_2_18_0.md @@ -0,0 +1 @@ +**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index 4ea13cd029..f6ec287705 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -9,6 +9,7 @@ api: license: community type: function --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # add_columnstore_policy() @@ -21,7 +22,7 @@ You do this by calling `ALTER TABLE` for hypertables and `ALTER MATERIALIZED VIE To view the policies that you set or the policies that already exist, see [informational views][informational-views], to remove a policy, see [remove_columnstore_policy][remove_columnstore_policy]. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index ea55c24dcb..c864f570cf 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -9,6 +9,7 @@ api: type: command products: [cloud, self_hosted] --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # ALTER TABLE (Hypercore) @@ -19,7 +20,7 @@ After you have enabled the columnstore, either: specific time interval. - [convert_to_columnstore][convert_to_columnstore]: manually add a specific chunk in a hypertable to the columnstore. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/chunk_columnstore_settings.md b/api/hypercore/chunk_columnstore_settings.md index 6fb484788f..7578d1e01b 100644 --- a/api/hypercore/chunk_columnstore_settings.md +++ b/api/hypercore/chunk_columnstore_settings.md @@ -8,12 +8,13 @@ api: license: community type: view --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # timescaledb_information.chunk_columnstore_settings Retrieve information about each chunk in the columnstore. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/chunk_columnstore_stats.md b/api/hypercore/chunk_columnstore_stats.md index a0a37ccc14..6db70748ed 100644 --- a/api/hypercore/chunk_columnstore_stats.md +++ b/api/hypercore/chunk_columnstore_stats.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # chunk_columnstore_stats() Community @@ -22,7 +23,7 @@ Get chunk-specific statistics related to hypercore. Inserting into a chunk in the columnstore does not change the chunk size. For more information about how to compute chunk sizes, see [chunks_detailed_size][chunks_detailed_size]. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index d3111b9990..78c85ecd7b 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # convert_to_columnstore() Community @@ -19,7 +20,7 @@ Although `convert_to_columnstore` gives you more more fine grained control, best To move a chunk from the columnstore back to the rowstore, use [`convert_to_rowstore`][convert_to_rowstore]. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index 99f2e760fd..2babe2bff5 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -7,6 +7,7 @@ api: license: community type: function --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # convert_to_rowstore() Community @@ -18,7 +19,7 @@ data. After the update, [convert the chunk to the columnstore][convert_to_column This workflow is especially useful if you need to backfill old data. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/hypertable_columnstore_settings.md b/api/hypercore/hypertable_columnstore_settings.md index 2dd6598f48..90951436c2 100644 --- a/api/hypercore/hypertable_columnstore_settings.md +++ b/api/hypercore/hypertable_columnstore_settings.md @@ -8,12 +8,13 @@ api: license: community type: view --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # timescaledb_information.hypertable_columnstore_settings Retrieve information about the settings for each hypertable in the columnstore. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/hypertable_columnstore_stats.md b/api/hypercore/hypertable_columnstore_stats.md index e8806e1f6e..2e30fb2d74 100644 --- a/api/hypercore/hypertable_columnstore_stats.md +++ b/api/hypercore/hypertable_columnstore_stats.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # hypertable_columnstore_stats() Community @@ -16,7 +17,7 @@ Get statistics related to compression in the columnstore. All sizes are in bytes For more information about using hypertables, including chunk size partitioning, see the [hypertable section][hypertable-docs]. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/hypercore/index.md b/api/hypercore/index.md index 61fe9a4218..07c7e61643 100644 --- a/api/hypercore/index.md +++ b/api/hypercore/index.md @@ -7,6 +7,7 @@ products: [cloud, self_hosted] api: license: community --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # Hypercore @@ -28,7 +29,7 @@ in each $CLOUD_LONG service: - **Full mutability with transactional semantics**: regardless of where data is stored, hypercore provides full ACID support. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Hypercore workflow diff --git a/api/hypercore/remove_columnstore_policy.md b/api/hypercore/remove_columnstore_policy.md index 59378e93bc..b501e1f38c 100644 --- a/api/hypercore/remove_columnstore_policy.md +++ b/api/hypercore/remove_columnstore_policy.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # remove_columnstore_policy() @@ -16,7 +17,7 @@ Remove a columnstore policy from a hypertable or continuous aggregate. To restart automatic chunk migration to the columnstore, you need to call [add_columnstore_policy][add_columnstore_policy] again. -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** + ## Samples diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 7960662eff..03edeab8cf 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -136,6 +136,11 @@ module.exports = [ href: "remove_columnstore_policy", excerpt: "Remove the compression policy", }, + { + title: "convert_to_columnstore", + href: "convert_to_columnstore", + excerpt: "Compress or recompress a specific chunk in the rowstore and add it to the columnstore", + }, ], }, { From 993cf077334682ce71bbf858e5d0a7d180284f3e Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 14:25:01 +0100 Subject: [PATCH 06/31] chore: add convert_to_rowstore to the build. --- api/page-index/page-index.js | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 03edeab8cf..c1f175c7ae 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -141,6 +141,11 @@ module.exports = [ href: "convert_to_columnstore", excerpt: "Compress or recompress a specific chunk in the rowstore and add it to the columnstore", }, + { + title: "convert_to_rowstore", + href: "convert_to_rowstore", + excerpt: "Decompress a chunk from the columnstore and add it to the rowstore", + }, ], }, { From d1fde3520a88a4224bc7c1e63b9983eee94681c6 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 14:39:31 +0100 Subject: [PATCH 07/31] chore: updates for build. --- api/hypercore/convert_to_rowstore.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index 2babe2bff5..c48b4f4bef 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -7,6 +7,7 @@ api: license: community type: function --- + import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # convert_to_rowstore() Community @@ -25,12 +26,12 @@ This workflow is especially useful if you need to backfill old data. To modify or add a lot of data to a chunk: - +1. **Stop the [jobs][alter_job] that are automatically adding chunks to the columnstore** -1. **Stop the [jobs][alter_job] that are automatically adding chunks to the columnstore** ``` sql SELECT alter_job(JOB_ID, scheduled => false); ``` + You retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view. 1. **Convert the chunks to update back to the rowstore** @@ -49,8 +50,8 @@ To modify or add a lot of data to a chunk: 1. **[Update the data][insert] in the chunk you added to the rowstore** - Best practice is to structure your INSERT statement to include appropriate - partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk: + Best practice is to structure your INSERT statement to include appropriate + partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk: ``` sql INSERT INTO metrics (time, value) @@ -58,24 +59,25 @@ To modify or add a lot of data to a chunk: ``` 1. **Convert the updated chunks back to the columnstore** + ``` sql SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); ``` 1. **Restart the [jobs][alter_job] that are automatically adding chunks to the columnstore** + ``` sql SELECT alter_job(JOB_ID, scheduled => true); ``` - - ## Arguments -| Name | Type | Default | Required | Description | -|--|--|--|--|------------------------------------------------------------------------------------------------------------| -|`chunk`|`REGCLASS`|-|✖| Name of the chunk to be moved to the rowstore. | +| Name | Type | Default | Required | Description| +|--|--|--|--|-| +|`chunk`|`REGCLASS`|-|✖| Name of the chunk to be moved to the rowstore. | |`if_compressed`|`BOOLEAN`|`true`|✔| Set to `false` so this job fails with an error rather than an warning if `chunk` is not in the columnstore | + [job]: /api/:currentVersion:/actions/ [alter_job]: /api/:currentVersion:/actions/alter_job/ [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ From f72724131915e7d2dc6758c872f70cbe166cc78f Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 14:49:35 +0100 Subject: [PATCH 08/31] chore: updates for build. --- api/hypercore/add_columnstore_policy.md | 2 +- api/hypercore/convert_to_rowstore.md | 62 ------------------------- 2 files changed, 1 insertion(+), 63 deletions(-) diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index f6ec287705..2be57a3bef 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -11,7 +11,7 @@ api: --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; -# add_columnstore_policy() +# add_columnstore_policy() Create a [job][job] that automatically moves chunks in a hypertable to the columnstore after a specific time interval. diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index c48b4f4bef..80958227eb 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -19,67 +19,5 @@ any [jobs][job] moving chunks to the columnstore, convert the chunk back to the data. After the update, [convert the chunk to the columnstore][convert_to_columnstore] and restart the jobs. This workflow is especially useful if you need to backfill old data. - -## Samples - -To modify or add a lot of data to a chunk: - -1. **Stop the [jobs][alter_job] that are automatically adding chunks to the columnstore** - - ``` sql - SELECT alter_job(JOB_ID, scheduled => false); - ``` - - You retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view. - -1. **Convert the chunks to update back to the rowstore** - - - Convert single chunk: - - ``` sql - SELECT convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); - ``` - - - Convert all chunks in a hypertable named `metrics`: - - ``` sql - SELECT convert_to_rowstore(c, true) FROM show_chunks('metrics') c; - ``` - -1. **[Update the data][insert] in the chunk you added to the rowstore** - - Best practice is to structure your INSERT statement to include appropriate - partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk: - - ``` sql - INSERT INTO metrics (time, value) - VALUES ('2025-01-01T00:00:00', 42); - ``` - -1. **Convert the updated chunks back to the columnstore** - - ``` sql - SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); - ``` - -1. **Restart the [jobs][alter_job] that are automatically adding chunks to the columnstore** - - ``` sql - SELECT alter_job(JOB_ID, scheduled => true); - ``` - -## Arguments - -| Name | Type | Default | Required | Description| -|--|--|--|--|-| -|`chunk`|`REGCLASS`|-|✖| Name of the chunk to be moved to the rowstore. | -|`if_compressed`|`BOOLEAN`|`true`|✔| Set to `false` so this job fails with an error rather than an warning if `chunk` is not in the columnstore | - - -[job]: /api/:currentVersion:/actions/ -[alter_job]: /api/:currentVersion:/actions/alter_job/ -[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ -[informational-views]: /api/:currentVersion:/informational-views/jobs/ -[insert]: /use-timescale/:currentVersion:/write-data/insert/ From 67c638f85a85933e82b9be2afc652f49d7c5f9f8 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 15:08:02 +0100 Subject: [PATCH 09/31] chore: remove links from steps in a list. --- api/hypercore/convert_to_rowstore.md | 57 ++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index 80958227eb..a1b9834123 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -21,3 +21,60 @@ This workflow is especially useful if you need to backfill old data. +## Samples + +To modify or add a lot of data to a chunk: + +1. **Stop the jobs that are automatically adding chunks to the columnstore** + + Retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view + to find the job you need to [alter_job][alter_job]. + + ``` sql + SELECT alter_job(JOB_ID, scheduled => false); + ``` + +1. **Convert the chunks to update back to the rowstore** + + - Convert single chunk: + + ``` sql + SELECT convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); + ``` + + - Convert all chunks in a hypertable named `metrics`: + + ``` sql + SELECT convert_to_rowstore(c, true) FROM show_chunks('metrics') c; + ``` + +1. **Update the data in the chunk you added to the rowstore** + + Best practice is to structure your [INSERT][insert] statement to include appropriate + partition key values, such as the timestamp. TimescaleDB adds the data to the correct chunk: + + ``` sql + INSERT INTO metrics (time, value) + VALUES ('2025-01-01T00:00:00', 42); + ``` + +1. **Convert the updated chunks back to the columnstore** + + ``` sql + SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); + ``` + +1. **Restart the jobs that are automatically converting chunks to the columnstore** + + ``` sql + SELECT alter_job(JOB_ID, scheduled => true); + ``` + + +[job]: /api/:currentVersion:/actions/ +[alter_job]: /api/:currentVersion:/actions/alter_job/ +[convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ +[informational-views]: /api/:currentVersion:/informational-views/jobs/ +[insert]: /use-timescale/:currentVersion:/write-data/insert/ + + From 39c8f8c13d8af1c12743088a7edd6f3a29688749 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 15:19:08 +0100 Subject: [PATCH 10/31] chore: remove links from steps in a list. --- api/hypercore/convert_to_rowstore.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index a1b9834123..6fcb9f3c0c 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -25,6 +25,8 @@ This workflow is especially useful if you need to backfill old data. To modify or add a lot of data to a chunk: + + 1. **Stop the jobs that are automatically adding chunks to the columnstore** Retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view @@ -70,6 +72,14 @@ To modify or add a lot of data to a chunk: SELECT alter_job(JOB_ID, scheduled => true); ``` + + +## Arguments + +| Name | Type | Default | Required | Description| +|--|----------|---------|----------|-| +|`chunk`| REGCLASS | - | ✖ | Name of the chunk to be moved to the rowstore. | +|`if_compressed`| BOOLEAN | `true` | ✔ | Set to `false` so this job fails with an error rather than an warning if `chunk` is not in the columnstore | [job]: /api/:currentVersion:/actions/ [alter_job]: /api/:currentVersion:/actions/alter_job/ From af34323e47d1590aba25f20925f61a0e3da9bf30 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 15:49:52 +0100 Subject: [PATCH 11/31] chore: hopefully this will work now. --- api/hypercore/chunk_columnstore_settings.md | 6 ++-- api/hypercore/chunk_columnstore_stats.md | 2 +- api/hypercore/convert_to_columnstore.md | 2 +- api/hypercore/convert_to_rowstore.md | 2 +- .../hypertable_columnstore_settings.md | 2 +- api/hypercore/hypertable_columnstore_stats.md | 4 +-- api/page-index/page-index.js | 28 ++++++++++++++++--- 7 files changed, 33 insertions(+), 13 deletions(-) diff --git a/api/hypercore/chunk_columnstore_settings.md b/api/hypercore/chunk_columnstore_settings.md index 7578d1e01b..474291041c 100644 --- a/api/hypercore/chunk_columnstore_settings.md +++ b/api/hypercore/chunk_columnstore_settings.md @@ -12,13 +12,13 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # timescaledb_information.chunk_columnstore_settings -Retrieve information about each chunk in the columnstore. +Retrieve the compression settings for each chunk in the columnstore. ## Samples -* Show settings for all chunks in the columnstore: +* **Show settings for all chunks in the columnstore**: ```sql SELECT * FROM timescaledb_information.chunk_columnstore_settings @@ -30,7 +30,7 @@ Retrieve information about each chunk in the columnstore. measurements | _timescaledb_internal._hyper_1_1_chunk| | "time" DESC ``` -* Find all chunk columnstore settings for a specific hypertable: +* **Find all chunk columnstore settings for a specific hypertable**: ```sql SELECT * FROM timescaledb_information.chunk_columnstore_settings WHERE hypertable::TEXT LIKE 'metrics'; diff --git a/api/hypercore/chunk_columnstore_stats.md b/api/hypercore/chunk_columnstore_stats.md index 6db70748ed..2f4468a42e 100644 --- a/api/hypercore/chunk_columnstore_stats.md +++ b/api/hypercore/chunk_columnstore_stats.md @@ -12,7 +12,7 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # chunk_columnstore_stats() Community -Get chunk-specific statistics related to hypercore. +Retrieve statistics about the chunks in the columnstore `chunk_columnstore_stats` returns the size of chunks in the columnstore, these values are computed when you call either: - [add_columnstore_policy][add_columnstore_policy]: create a [job][job] that automatically moves chunks in a hypertable to the columnstore at a diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index 78c85ecd7b..c536a31161 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -12,7 +12,7 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # convert_to_columnstore() Community -Manually control the exact time you move a specific chunk in a hypertable to the columnstore. +Manually convert a specific chunk in the hypertable rowstore to the columnstore. Although `convert_to_columnstore` gives you more more fine grained control, best practice is to use [`add_columnstore_policy`][add_columnstore_policy]. You can also add chunks to the columnstore at a specific time diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index 6fcb9f3c0c..ac63c5ebe4 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -12,7 +12,7 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # convert_to_rowstore() Community -Move a chunk of data from the columnstore to the rowstore. +Manually convert a specific chunk in the hypertable columnstore to the rowstore. If you need to modify or add a lot of data to a chunk in the columnstore, best practice is to stop any [jobs][job] moving chunks to the columnstore, convert the chunk back to the rowstore, then modify the diff --git a/api/hypercore/hypertable_columnstore_settings.md b/api/hypercore/hypertable_columnstore_settings.md index 90951436c2..4c730622b5 100644 --- a/api/hypercore/hypertable_columnstore_settings.md +++ b/api/hypercore/hypertable_columnstore_settings.md @@ -12,7 +12,7 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # timescaledb_information.hypertable_columnstore_settings -Retrieve information about the settings for each hypertable in the columnstore. +Retrieve information about the settings for all hypertables in the columnstore. diff --git a/api/hypercore/hypertable_columnstore_stats.md b/api/hypercore/hypertable_columnstore_stats.md index 2e30fb2d74..e41dd955df 100644 --- a/api/hypercore/hypertable_columnstore_stats.md +++ b/api/hypercore/hypertable_columnstore_stats.md @@ -12,10 +12,10 @@ import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # hypertable_columnstore_stats() Community -Get statistics related to compression in the columnstore. All sizes are in bytes. +Retrieve compression statistics for the columnstore. For more information about using hypertables, including chunk size partitioning, -see the [hypertable section][hypertable-docs]. +see [hypertables][hypertable-docs]. diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index c1f175c7ae..a05e75ba09 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -129,22 +129,42 @@ module.exports = [ { title: "add_columnstore_policy", href: "add_columnstore_policy", - excerpt: "Automatically move chunks in a hypertable to the columnstore after a specific time interval.", + excerpt: "Automatically convert chunks in the hypertable rowstore to the columnstore after a specific time interval", }, { title: "remove_columnstore_policy", href: "remove_columnstore_policy", - excerpt: "Remove the compression policy", + excerpt: "Remove a columnstore policy from a hypertable or continuous aggregate", }, { title: "convert_to_columnstore", href: "convert_to_columnstore", - excerpt: "Compress or recompress a specific chunk in the rowstore and add it to the columnstore", + excerpt: "Manually convert a specific chunk in the hypertable rowstore to the columnstore", }, { title: "convert_to_rowstore", href: "convert_to_rowstore", - excerpt: "Decompress a chunk from the columnstore and add it to the rowstore", + excerpt: "Manually convert a specific chunk in the hypertable columnstore to the rowstore", + }, + { + title: "hypertable_columnstore_settings", + href: "hypertable_columnstore_settings", + excerpt: "Retrieve information about the settings for all hypertables in the columnstore", + }, + { + title: "hypertable_columnstore_stats", + href: "hypertable_columnstore_stats", + excerpt: "Retrieve compression statistics for the columnstore", + }, + { + title: "chunk_columnstore_settings", + href: "chunk_columnstore_settings", + excerpt: "Retrieve the compression settings for each chunk in the columnstore", + }, + { + title: "chunk_columnstore_stats", + href: "chunk_columnstore_stats", + excerpt: "Retrieve statistics about the chunks in the columnstore", }, ], }, From a80f250840db68c077cc96f7cdf58720355aab80 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 16:27:44 +0100 Subject: [PATCH 12/31] chore: add deprecation for compression. --- _partials/_deprecated_2_18_0.md | 2 +- _partials/_since_2_18_0.md | 2 +- api/add_compression_policy.md | 3 +++ api/alter_table_compression.md | 3 +++ api/chunk_compression_stats.md | 3 +++ api/compress_chunk.md | 4 ++++ api/compression.md | 3 +++ api/decompress_chunk.md | 3 +++ api/hypertable_compression_stats.md | 3 +++ api/recompress_chunk.md | 4 ++++ api/remove_compression_policy.md | 3 +++ 11 files changed, 31 insertions(+), 2 deletions(-) diff --git a/_partials/_deprecated_2_18_0.md b/_partials/_deprecated_2_18_0.md index a4d4c0aec0..fd1ca0adc0 100644 --- a/_partials/_deprecated_2_18_0.md +++ b/_partials/_deprecated_2_18_0.md @@ -1 +1 @@ -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** +**Deprecated [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** diff --git a/_partials/_since_2_18_0.md b/_partials/_since_2_18_0.md index a4d4c0aec0..d175e38f3c 100644 --- a/_partials/_since_2_18_0.md +++ b/_partials/_since_2_18_0.md @@ -1 +1 @@ -**@since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** +**Since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** diff --git a/api/add_compression_policy.md b/api/add_compression_policy.md index 520cd4f885..8a50702643 100644 --- a/api/add_compression_policy.md +++ b/api/add_compression_policy.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # add_compression_policy() @@ -23,6 +24,8 @@ command. To enable compression on continuous aggregates, use the command. To view the policies that you set or the policies that already exist, see [informational views][informational-views]. + + ## Required arguments |Name|Type|Description| diff --git a/api/alter_table_compression.md b/api/alter_table_compression.md index c2990aa5ac..f207b5c462 100644 --- a/api/alter_table_compression.md +++ b/api/alter_table_compression.md @@ -8,6 +8,7 @@ api: license: community type: command --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # ALTER TABLE (Compression) @@ -28,6 +29,8 @@ ALTER TABLE SET (timescaledb.compress, ); ``` + + ## Required arguments |Name|Type|Description| diff --git a/api/chunk_compression_stats.md b/api/chunk_compression_stats.md index 5f7b995937..374aecbc11 100644 --- a/api/chunk_compression_stats.md +++ b/api/chunk_compression_stats.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # chunk_compression_stats() Community @@ -20,6 +21,8 @@ the chunk. An insert into a compressed chunk does not update the compressed sizes. For more information about how to compute chunk sizes, see the `chunks_detailed_size` section. + + ### Required arguments |Name|Type|Description| diff --git a/api/compress_chunk.md b/api/compress_chunk.md index d8c4962439..fbe23a3e49 100644 --- a/api/compress_chunk.md +++ b/api/compress_chunk.md @@ -9,6 +9,8 @@ api: type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; + # compress_chunk() Community The `compress_chunk` function is used to compress (or recompress, if necessary) @@ -27,6 +29,8 @@ You can get a list of chunks belonging to a hypertable using the [`show_chunks` function](/api/latest/hypertable/show_chunks/). + + ### Required arguments |Name|Type|Description| diff --git a/api/compression.md b/api/compression.md index e802dd4a8c..5f7d922437 100644 --- a/api/compression.md +++ b/api/compression.md @@ -4,6 +4,7 @@ excerpt: Compress your hypertable keywords: [compression] tags: [hypertables] --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # Compression Community @@ -30,6 +31,8 @@ Compressed chunks have the following limitations: after constraint creation. * [Timescale SkipScan][skipscan] does not currently work on compressed chunks. + + ## Restrictions In general, compressing a hypertable imposes some limitations on the types diff --git a/api/decompress_chunk.md b/api/decompress_chunk.md index 8d5bf8a33e..c9a06804a2 100644 --- a/api/decompress_chunk.md +++ b/api/decompress_chunk.md @@ -7,6 +7,7 @@ api: license: community type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # decompress_chunk() Community @@ -22,6 +23,8 @@ turn the policy back on. The database automatically recompresses your chunks in the next scheduled job. + + ### Required arguments |Name|Type|Description| diff --git a/api/hypertable_compression_stats.md b/api/hypertable_compression_stats.md index ebc37ba065..bce06b8d48 100644 --- a/api/hypertable_compression_stats.md +++ b/api/hypertable_compression_stats.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # hypertable_compression_stats() Community @@ -19,6 +20,8 @@ see the [hypertable section][hypertable-docs]. For more information about compression, see the [compression sction][compression-docs]. + + ### Required arguments |Name|Type|Description| diff --git a/api/recompress_chunk.md b/api/recompress_chunk.md index 85d6274881..d1b577524f 100644 --- a/api/recompress_chunk.md +++ b/api/recompress_chunk.md @@ -9,6 +9,8 @@ api: type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; + # recompress_chunk() Recompresses a compressed chunk that had more data inserted after compression. @@ -41,6 +43,8 @@ the procedure with `CALL`. Don't use a `SELECT` statement. chunk for the first time, use [`compress_chunk`](/api/latest/compression/compress_chunk/). + + ## Required arguments |Name|Type|Description| diff --git a/api/remove_compression_policy.md b/api/remove_compression_policy.md index 489d5ec052..36ec1a4d65 100644 --- a/api/remove_compression_policy.md +++ b/api/remove_compression_policy.md @@ -8,6 +8,7 @@ api: license: community type: function --- +import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # remove_compression_policy() @@ -15,6 +16,8 @@ If you need to remove the compression policy. To restart policy-based compression you need to add the policy again. To view the policies that already exist, see [informational views][informational-views]. + + ### Required arguments |Name|Type|Description| From c03cae9734729eabdb3e8de4a698b46d8e453088 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 16:59:48 +0100 Subject: [PATCH 13/31] chore: simplify workflow, keep code for main docs. --- _partials/_hypercore_policy_workflow.md | 57 +++++++++++++++++++++++++ api/hypercore/index.md | 31 -------------- 2 files changed, 57 insertions(+), 31 deletions(-) create mode 100644 _partials/_hypercore_policy_workflow.md diff --git a/_partials/_hypercore_policy_workflow.md b/_partials/_hypercore_policy_workflow.md new file mode 100644 index 0000000000..20e6a4aa14 --- /dev/null +++ b/_partials/_hypercore_policy_workflow.md @@ -0,0 +1,57 @@ +1. **Enable columnstore** + + * [Use `ALTER TABLE` for a hypertable][alter_table_hypercore] + ```sql + ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol'); + ``` + * [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate] + ```sql + ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ``` + +1. **Add a policy to move chunks to the columnstore at a specific time interval** + + For example, 60 days after the data was added to the table: + ``` sql + SELECT add_columnstore_policy('older_stock_prices', after => INTERVAL '60d'); + ``` + See [add_columnstore_policy][add_columnstore_policy]. + +1. **View the policies that you set or the policies that already exist** + + ``` sql + SELECT * FROM timescaledb_information.jobs + WHERE proc_name='policy_compression'; + ``` + See [timescaledb_information.jobs][informational-views]. + +1. **Pause a columnstore policy** + + ``` sql + SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'stocks_real_time' + + -- Select the JOB_ID from the results + + SELECT alter_job(JOB_ID, scheduled => false); + ``` + See [alter_job][alter_job]. + +1. **Restart a columnstore policy** + ``` sql + SELECT alter_job(JOB_ID, scheduled => true); + ``` + See [alter_job][alter_job]. + +1. **Remove a columnstore policy** + ``` sql + SELECT remove_columnstore_policy('older_stock_prices'); + ``` + See [remove_columnstore_policy][remove_columnstore_policy]. +1. **Disable columnstore** + + If your table has chunks in the columnstore, you have to + [convert the chunks back to the rowstore][convert_to_rowstore] before you disable the columnstore. + ``` sql + ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = false); + ``` + See [alter_table_hypercore][alter_table_hypercore]. diff --git a/api/hypercore/index.md b/api/hypercore/index.md index 07c7e61643..a87f3cb733 100644 --- a/api/hypercore/index.md +++ b/api/hypercore/index.md @@ -64,37 +64,6 @@ Best practice for using Hypercore is to: ``` See [timescaledb_information.jobs][informational-views]. -1. **Pause a columnstore policy** - - ``` sql - SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'stocks_real_time' - - -- Select the JOB_ID from the results - - SELECT alter_job(JOB_ID, scheduled => false); - ``` - See [alter_job][alter_job]. - -1. **Restart a columnstore policy** - ``` sql - SELECT alter_job(JOB_ID, scheduled => true); - ``` - See [alter_job][alter_job]. - -1. **Remove a columnstore policy** - ``` sql - SELECT remove_columnstore_policy('older_stock_prices'); - ``` - See [remove_columnstore_policy][remove_columnstore_policy]. -1. **Disable columnstore** - - If your table has chunks in the columnstore, you have to - [convert the chunks back to the rowstore][convert_to_rowstore] before you disable the columnstore. - ``` sql - ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = false); - ``` - See [alter_table_hypercore][alter_table_hypercore]. - You can also [convert_to_columnstore][convert_to_columnstore] and [convert_to_rowstore][convert_to_rowstore] manually From e58a7c9fa31cf5af6acf8e42b77c9fccd312db41 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 9 Jan 2025 18:00:48 +0100 Subject: [PATCH 14/31] chore: updates on review. . --- _partials/_deprecated_2_18_0.md | 2 +- _partials/_since_2_18_0.md | 2 +- api/hypercore/alter_table.md | 2 ++ api/hypercore/chunk_columnstore_settings.md | 4 ++++ api/hypercore/chunk_columnstore_stats.md | 2 ++ api/hypercore/convert_to_columnstore.md | 2 +- api/hypercore/hypertable_columnstore_settings.md | 2 ++ api/hypercore/hypertable_columnstore_stats.md | 2 ++ 8 files changed, 15 insertions(+), 3 deletions(-) diff --git a/_partials/_deprecated_2_18_0.md b/_partials/_deprecated_2_18_0.md index fd1ca0adc0..4231729519 100644 --- a/_partials/_deprecated_2_18_0.md +++ b/_partials/_deprecated_2_18_0.md @@ -1 +1 @@ -**Deprecated [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** +Deprecated [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) diff --git a/_partials/_since_2_18_0.md b/_partials/_since_2_18_0.md index d175e38f3c..19a03870be 100644 --- a/_partials/_since_2_18_0.md +++ b/_partials/_since_2_18_0.md @@ -1 +1 @@ -**Since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)** +Since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index c864f570cf..87e76a361c 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -24,6 +24,8 @@ After you have enabled the columnstore, either: ## Samples +To enable the columnstore: + - **Configure a hypertable that ingests device data to use the columnstore**: In this example, the `metrics` hypertable is often queried about a specific device or set of devices. diff --git a/api/hypercore/chunk_columnstore_settings.md b/api/hypercore/chunk_columnstore_settings.md index 474291041c..249b2cf77f 100644 --- a/api/hypercore/chunk_columnstore_settings.md +++ b/api/hypercore/chunk_columnstore_settings.md @@ -18,6 +18,10 @@ Retrieve the compression settings for each chunk in the columnstore. ## Samples +To retrieve information about settings: + +```sql + * **Show settings for all chunks in the columnstore**: ```sql diff --git a/api/hypercore/chunk_columnstore_stats.md b/api/hypercore/chunk_columnstore_stats.md index 2f4468a42e..50e60308eb 100644 --- a/api/hypercore/chunk_columnstore_stats.md +++ b/api/hypercore/chunk_columnstore_stats.md @@ -27,6 +27,8 @@ chunk sizes, see [chunks_detailed_size][chunks_detailed_size]. ## Samples +To retrieve statistics about chunks: + - **Show the status of the first two chunks in the `conditions` hypertable**: ```sql SELECT * FROM chunk_columnstore_stats('conditions') diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index c536a31161..ee80e43cb2 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -24,7 +24,7 @@ To move a chunk from the columnstore back to the rowstore, use [`convert_to_rows ## Samples -Compress a single chunk. +To compress a single chunk: ``` sql SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); diff --git a/api/hypercore/hypertable_columnstore_settings.md b/api/hypercore/hypertable_columnstore_settings.md index 4c730622b5..e8f1a60024 100644 --- a/api/hypercore/hypertable_columnstore_settings.md +++ b/api/hypercore/hypertable_columnstore_settings.md @@ -18,6 +18,8 @@ Retrieve information about the settings for all hypertables in the columnstore. ## Samples +To retrieve information about settings: + - **Show columnstore settings for all hypertables**: ```sql diff --git a/api/hypercore/hypertable_columnstore_stats.md b/api/hypercore/hypertable_columnstore_stats.md index e41dd955df..4d24da5349 100644 --- a/api/hypercore/hypertable_columnstore_stats.md +++ b/api/hypercore/hypertable_columnstore_stats.md @@ -21,6 +21,8 @@ see [hypertables][hypertable-docs]. ## Samples +To retrieve compression statistics: + - **Show the compression status of the `conditions` hypertable**: ```sql From 1409dfd81c436fb4bbaa29710f222fee83c097f2 Mon Sep 17 00:00:00 2001 From: Iain Date: Fri, 10 Jan 2025 11:14:40 +0100 Subject: [PATCH 15/31] chore: updates on review. --- api/hypercore/add_columnstore_policy.md | 1 + api/hypercore/index.md | 1 + 2 files changed, 2 insertions(+) diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index 2be57a3bef..1a1657ae45 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -9,6 +9,7 @@ api: license: community type: function --- + import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # add_columnstore_policy() diff --git a/api/hypercore/index.md b/api/hypercore/index.md index a87f3cb733..62ae1181a7 100644 --- a/api/hypercore/index.md +++ b/api/hypercore/index.md @@ -7,6 +7,7 @@ products: [cloud, self_hosted] api: license: community --- + import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; # Hypercore From de1c436e9471a334fd5f2d67b09da25bd3d9c2cb Mon Sep 17 00:00:00 2001 From: Iain Date: Mon, 6 Jan 2025 10:27:40 +0100 Subject: [PATCH 16/31] feat: update structure for hypercore --- api/hypercore/page-index/page-index.js | 55 ++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 api/hypercore/page-index/page-index.js diff --git a/api/hypercore/page-index/page-index.js b/api/hypercore/page-index/page-index.js new file mode 100644 index 0000000000..cde701128c --- /dev/null +++ b/api/hypercore/page-index/page-index.js @@ -0,0 +1,55 @@ +module.exports = [ + { + title: "Hypercore", + href: "hypercore", + excerpt: + "Seamlessly switch between row-oriented and column-oriented storage", + children: [ + { + title: "add_columnstore_policy", + href: "add_columnstore_policy", + excerpt: "Compresses a chunk automatically in the background after it reaches a given age", + }, + { + title: "chunk_columnstore_settings", + href: "chunk_columnstore_settings", + excerpt: "Show the compression settings for each chunk that has compression enabled", + }, + { + title: "chunk_columnstore_stats", + href: "chunk_columnstore_stats", + excerpt: "Get chunk-specific statistics related to hypertable compression", + }, + { + title: "columnstore_settings", + href: "columnstore_settings", + excerpt: "Get information about compression-related settings for hypertables", + }, + { + title: "convert_to_columnstore", + href: "convert_to_columnstore", + excerpt: "Compress or recompress a specific chunk in the rowstore and add it to the columnstore", + }, + { + title: "convert_to_rowstore", + href: "convert_to_rowstore", + excerpt: "Decompress a chunk from the columnstore and add it to the rowstore", + }, + { + title: "hypertable_columnstore_settings", + href: "hypertable_columnstore_settings", + excerpt: "Returns information about the compression settings for each hypertable in the columnstore.", + }, + { + title: "hypertable_columnstore_stats", + href: "hypertable_columnstore_stats", + excerpt: "Get statistics related to hypertable compression", + }, + { + title: "remove_columnstore_policy", + href: "remove_columnstore_policy", + excerpt: "Remove the compression policy", + }, + ], + }, +]; From 834a406b88a88a92e1fbc8ee016f73c151e32103 Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 15 Jan 2025 15:51:11 +0100 Subject: [PATCH 17/31] chore: add replacedby link. Has to be hardcoded so will not work until these docs go live. --- .../add_compression_policy.md | 2 +- .../alter_table_compression.md | 2 +- .../chunk_compression_stats.md | 2 +- api/{ => compression}/compress_chunk.md | 2 +- api/{ => compression}/decompress_chunk.md | 2 +- .../hypertable_compression_stats.md | 2 +- api/{compression.md => compression/index.md} | 2 +- api/{ => compression}/recompress_chunk.md | 2 +- .../remove_compression_policy.md | 2 +- api/page-index/page-index.js | 199 +++++++++--------- 10 files changed, 108 insertions(+), 109 deletions(-) rename api/{ => compression}/add_compression_policy.md (97%) rename api/{ => compression}/alter_table_compression.md (96%) rename api/{ => compression}/chunk_compression_stats.md (96%) rename api/{ => compression}/compress_chunk.md (92%) rename api/{ => compression}/decompress_chunk.md (90%) rename api/{ => compression}/hypertable_compression_stats.md (94%) rename api/{compression.md => compression/index.md} (96%) rename api/{ => compression}/recompress_chunk.md (95%) rename api/{ => compression}/remove_compression_policy.md (89%) diff --git a/api/add_compression_policy.md b/api/compression/add_compression_policy.md similarity index 97% rename from api/add_compression_policy.md rename to api/compression/add_compression_policy.md index 8a50702643..f0b7e7e52a 100644 --- a/api/add_compression_policy.md +++ b/api/compression/add_compression_policy.md @@ -24,7 +24,7 @@ command. To enable compression on continuous aggregates, use the command. To view the policies that you set or the policies that already exist, see [informational views][informational-views]. - + Replaced by add_columnstore_policy(). ## Required arguments diff --git a/api/alter_table_compression.md b/api/compression/alter_table_compression.md similarity index 96% rename from api/alter_table_compression.md rename to api/compression/alter_table_compression.md index f207b5c462..500ee3c587 100644 --- a/api/alter_table_compression.md +++ b/api/compression/alter_table_compression.md @@ -29,7 +29,7 @@ ALTER TABLE SET (timescaledb.compress, ); ``` - + Replaced by ALTER TABLE (Hypercore). ## Required arguments diff --git a/api/chunk_compression_stats.md b/api/compression/chunk_compression_stats.md similarity index 96% rename from api/chunk_compression_stats.md rename to api/compression/chunk_compression_stats.md index 374aecbc11..6abb8ddf03 100644 --- a/api/chunk_compression_stats.md +++ b/api/compression/chunk_compression_stats.md @@ -21,7 +21,7 @@ the chunk. An insert into a compressed chunk does not update the compressed sizes. For more information about how to compute chunk sizes, see the `chunks_detailed_size` section. - + Replaced by chunk_columnstore_stats(). ### Required arguments diff --git a/api/compress_chunk.md b/api/compression/compress_chunk.md similarity index 92% rename from api/compress_chunk.md rename to api/compression/compress_chunk.md index fbe23a3e49..4e858e2a44 100644 --- a/api/compress_chunk.md +++ b/api/compression/compress_chunk.md @@ -29,7 +29,7 @@ You can get a list of chunks belonging to a hypertable using the [`show_chunks` function](/api/latest/hypertable/show_chunks/). - + Replaced by convert_to_columnstore(). ### Required arguments diff --git a/api/decompress_chunk.md b/api/compression/decompress_chunk.md similarity index 90% rename from api/decompress_chunk.md rename to api/compression/decompress_chunk.md index c9a06804a2..ffab2a595f 100644 --- a/api/decompress_chunk.md +++ b/api/compression/decompress_chunk.md @@ -23,7 +23,7 @@ turn the policy back on. The database automatically recompresses your chunks in the next scheduled job. - + Replaced by convert_to_rowstore(). ### Required arguments diff --git a/api/hypertable_compression_stats.md b/api/compression/hypertable_compression_stats.md similarity index 94% rename from api/hypertable_compression_stats.md rename to api/compression/hypertable_compression_stats.md index bce06b8d48..99babe8ac2 100644 --- a/api/hypertable_compression_stats.md +++ b/api/compression/hypertable_compression_stats.md @@ -20,7 +20,7 @@ see the [hypertable section][hypertable-docs]. For more information about compression, see the [compression sction][compression-docs]. - + Replaced by hypertable_columnstore_stats(). ### Required arguments diff --git a/api/compression.md b/api/compression/index.md similarity index 96% rename from api/compression.md rename to api/compression/index.md index 5f7d922437..a665241aff 100644 --- a/api/compression.md +++ b/api/compression/index.md @@ -31,7 +31,7 @@ Compressed chunks have the following limitations: after constraint creation. * [Timescale SkipScan][skipscan] does not currently work on compressed chunks. - + Replaced by Hypercore. ## Restrictions diff --git a/api/recompress_chunk.md b/api/compression/recompress_chunk.md similarity index 95% rename from api/recompress_chunk.md rename to api/compression/recompress_chunk.md index d1b577524f..9b9c996b65 100644 --- a/api/recompress_chunk.md +++ b/api/compression/recompress_chunk.md @@ -43,7 +43,7 @@ the procedure with `CALL`. Don't use a `SELECT` statement. chunk for the first time, use [`compress_chunk`](/api/latest/compression/compress_chunk/). - + Replaced by convert_to_columnstore(). ## Required arguments diff --git a/api/remove_compression_policy.md b/api/compression/remove_compression_policy.md similarity index 89% rename from api/remove_compression_policy.md rename to api/compression/remove_compression_policy.md index 36ec1a4d65..48e48e0de1 100644 --- a/api/remove_compression_policy.md +++ b/api/compression/remove_compression_policy.md @@ -16,7 +16,7 @@ If you need to remove the compression policy. To restart policy-based compression you need to add the policy again. To view the policies that already exist, see [informational views][informational-views]. - + Replaced by remove_columnstore_policy(). ### Required arguments diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index a05e75ba09..c1f2fa0a3d 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -168,106 +168,6 @@ module.exports = [ }, ], }, - { - title: "Distributed hypertables", - type: "directory", - href: "distributed-hypertables", - children: [ - { - title: "create_distributed_hypertable", - href: "create_distributed_hypertable", - }, - { - title: "add_data_node", - href: "add_data_node", - }, - { - title: "attach_data_node", - href: "attach_data_node", - }, - { - title: "alter_data_node", - href: "alter_data_node", - }, - { - title: "detach_data_node", - href: "detach_data_node", - }, - { - title: "delete_data_node", - href: "delete_data_node", - }, - { - title: "distributed_exec", - href: "distributed_exec", - }, - { - title: "set_number_partitions", - href: "set_number_partitions", - }, - { - title: "set_replication_factor", - href: "set_replication_factor", - }, - { - title: "copy_chunk", - href: "copy_chunk_experimental", - }, - { - title: "move_chunk", - href: "move_chunk_experimental", - }, - { - title: "cleanup_copy_chunk_operation", - href: "cleanup_copy_chunk_operation_experimental", - }, - { - title: "create_distributed_restore_point", - href: "create_distributed_restore_point", - }, - ], - }, - { - title: "Compression", - type: "directory", - href: "compression", - description: - "We highly recommend reading the blog post and tutorial about compression before trying to set it up for the first time.", - children: [ - { - title: "ALTER TABLE (Compression)", - href: "alter_table_compression", - }, - { - title: "add_compression_policy", - href: "add_compression_policy", - }, - { - title: "remove_compression_policy", - href: "remove_compression_policy", - }, - { - title: "compress_chunk", - href: "compress_chunk", - }, - { - title: "decompress_chunk", - href: "decompress_chunk", - }, - { - title: "recompress_chunk", - href: "recompress_chunk", - }, - { - title: "hypertable_compression_stats", - href: "hypertable_compression_stats", - }, - { - title: "chunk_compression_stats", - href: "chunk_compression_stats", - }, - ], - }, { title: "Continuous aggregates", type: "redirect-to-child-page", @@ -659,6 +559,105 @@ module.exports = [ description: "An overview of what different tags represent in the API section of Timescale Documentation.", }, + { + title: "Compression (deprecated)", + href: "compression", + description: + "We highly recommend reading the blog post and tutorial about compression before trying to set it up for the first time.", + children: [ + { + title: "ALTER TABLE (Compression)", + href: "alter_table_compression", + }, + { + title: "add_compression_policy", + href: "add_compression_policy", + }, + { + title: "remove_compression_policy", + href: "remove_compression_policy", + }, + { + title: "compress_chunk", + href: "compress_chunk", + }, + { + title: "decompress_chunk", + href: "decompress_chunk", + }, + { + title: "recompress_chunk", + href: "recompress_chunk", + }, + { + title: "hypertable_compression_stats", + href: "hypertable_compression_stats", + }, + { + title: "chunk_compression_stats", + href: "chunk_compression_stats", + }, + ], + }, + { + title: "Distributed hypertables (deprecated)", + type: "directory", + href: "distributed-hypertables", + children: [ + { + title: "create_distributed_hypertable", + href: "create_distributed_hypertable", + }, + { + title: "add_data_node", + href: "add_data_node", + }, + { + title: "attach_data_node", + href: "attach_data_node", + }, + { + title: "alter_data_node", + href: "alter_data_node", + }, + { + title: "detach_data_node", + href: "detach_data_node", + }, + { + title: "delete_data_node", + href: "delete_data_node", + }, + { + title: "distributed_exec", + href: "distributed_exec", + }, + { + title: "set_number_partitions", + href: "set_number_partitions", + }, + { + title: "set_replication_factor", + href: "set_replication_factor", + }, + { + title: "copy_chunk", + href: "copy_chunk_experimental", + }, + { + title: "move_chunk", + href: "move_chunk_experimental", + }, + { + title: "cleanup_copy_chunk_operation", + href: "cleanup_copy_chunk_operation_experimental", + }, + { + title: "create_distributed_restore_point", + href: "create_distributed_restore_point", + }, + ], + }, ], }, ]; From fe6d956e13c7345e6d19885d0b0f2a1dfd009b46 Mon Sep 17 00:00:00 2001 From: Iain Cox Date: Wed, 15 Jan 2025 16:42:32 +0100 Subject: [PATCH 18/31] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Awesome review, thank you. Co-authored-by: Erik Nordström <819732+erimatnor@users.noreply.github.com> Signed-off-by: Iain Cox --- _partials/_hypercore_policy_workflow.md | 4 ++-- api/hypercore/add_columnstore_policy.md | 8 ++++---- api/hypercore/convert_to_columnstore.md | 4 ++-- api/hypercore/convert_to_rowstore.md | 4 ++-- api/hypercore/index.md | 2 +- api/hypercore/page-index/page-index.js | 18 +++++++++--------- api/hypercore/remove_columnstore_policy.md | 4 ++-- 7 files changed, 22 insertions(+), 22 deletions(-) diff --git a/_partials/_hypercore_policy_workflow.md b/_partials/_hypercore_policy_workflow.md index 20e6a4aa14..1d6e705884 100644 --- a/_partials/_hypercore_policy_workflow.md +++ b/_partials/_hypercore_policy_workflow.md @@ -13,7 +13,7 @@ For example, 60 days after the data was added to the table: ``` sql - SELECT add_columnstore_policy('older_stock_prices', after => INTERVAL '60d'); + CALL add_columnstore_policy('older_stock_prices', after => INTERVAL '60d'); ``` See [add_columnstore_policy][add_columnstore_policy]. @@ -44,7 +44,7 @@ 1. **Remove a columnstore policy** ``` sql - SELECT remove_columnstore_policy('older_stock_prices'); + CALL remove_columnstore_policy('older_stock_prices'); ``` See [remove_columnstore_policy][remove_columnstore_policy]. 1. **Disable columnstore** diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index 1a1657ae45..ab210780ca 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -48,22 +48,22 @@ To create a columnstore job: * 60 days after the data was added to the table: ``` sql - SELECT add_columnstore_policy('stocks_real_time', after => INTERVAL '60d'); + CALL add_columnstore_policy('stocks_real_time', after => INTERVAL '60d'); ``` * 3 months prior to the moment you run the query: ``` sql - SELECT add_columnstore_policy('stocks_real_time', created_before => INTERVAL '3 months'); + CALL add_columnstore_policy('stocks_real_time', created_before => INTERVAL '3 months'); ``` * With an integer-based time column: ``` sql - SELECT add_columnstore_policy('table_with_bigint_time', BIGINT '600000'); + CALL add_columnstore_policy('table_with_bigint_time', BIGINT '600000'); ``` * Older than eight weeks: ``` sql - SELECT add_columnstore_policy('cpu_weekly', INTERVAL '8 weeks'); + CALL add_columnstore_policy('cpu_weekly', INTERVAL '8 weeks'); ``` 1. **View the policies that you set or the policies that already exist** diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index ee80e43cb2..2b7f6c8c2d 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -24,10 +24,10 @@ To move a chunk from the columnstore back to the rowstore, use [`convert_to_rows ## Samples -To compress a single chunk: +To convert a single chunk to columnstore: ``` sql -SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); +CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); ``` To retrieve the chunks belonging to a hypertable, call [`show_chunks`](/api/latest/hypertable/show_chunks/). diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index ac63c5ebe4..7877922f80 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -41,7 +41,7 @@ To modify or add a lot of data to a chunk: - Convert single chunk: ``` sql - SELECT convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); + CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); ``` - Convert all chunks in a hypertable named `metrics`: @@ -63,7 +63,7 @@ To modify or add a lot of data to a chunk: 1. **Convert the updated chunks back to the columnstore** ``` sql - SELECT convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); + CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); ``` 1. **Restart the jobs that are automatically converting chunks to the columnstore** diff --git a/api/hypercore/index.md b/api/hypercore/index.md index 62ae1181a7..cec81e7a1f 100644 --- a/api/hypercore/index.md +++ b/api/hypercore/index.md @@ -53,7 +53,7 @@ Best practice for using Hypercore is to: For example, 60 days after the data was added to the table: ``` sql - SELECT add_columnstore_policy('older_stock_prices', after => INTERVAL '60d'); + CALL add_columnstore_policy('older_stock_prices', after => INTERVAL '60d'); ``` See [add_columnstore_policy][add_columnstore_policy]. diff --git a/api/hypercore/page-index/page-index.js b/api/hypercore/page-index/page-index.js index cde701128c..4a79ed6439 100644 --- a/api/hypercore/page-index/page-index.js +++ b/api/hypercore/page-index/page-index.js @@ -8,47 +8,47 @@ module.exports = [ { title: "add_columnstore_policy", href: "add_columnstore_policy", - excerpt: "Compresses a chunk automatically in the background after it reaches a given age", + excerpt: Convert a chunk to columnstore automatically in the background after it reaches a given age", }, { title: "chunk_columnstore_settings", href: "chunk_columnstore_settings", - excerpt: "Show the compression settings for each chunk that has compression enabled", + excerpt: "Show the columnstore settings for each chunk that is a columnstore", }, { title: "chunk_columnstore_stats", href: "chunk_columnstore_stats", - excerpt: "Get chunk-specific statistics related to hypertable compression", + excerpt: "Get statistics for columnstore chunks", }, { title: "columnstore_settings", href: "columnstore_settings", - excerpt: "Get information about compression-related settings for hypertables", + excerpt: "Get information about columnstore-related settings for hypertables", }, { title: "convert_to_columnstore", href: "convert_to_columnstore", - excerpt: "Compress or recompress a specific chunk in the rowstore and add it to the columnstore", + excerpt: "Convert a specific chunk from rowstore to columnstore", }, { title: "convert_to_rowstore", href: "convert_to_rowstore", - excerpt: "Decompress a chunk from the columnstore and add it to the rowstore", + excerpt: "Convert a specific chunk from columnstore to rowstore", }, { title: "hypertable_columnstore_settings", href: "hypertable_columnstore_settings", - excerpt: "Returns information about the compression settings for each hypertable in the columnstore.", + excerpt: "Returns information about the columnstore settings for each hypertable", }, { title: "hypertable_columnstore_stats", href: "hypertable_columnstore_stats", - excerpt: "Get statistics related to hypertable compression", + excerpt: "Get columnstore statistics for hypertables", }, { title: "remove_columnstore_policy", href: "remove_columnstore_policy", - excerpt: "Remove the compression policy", + excerpt: "Remove the columnstore policy", }, ], }, diff --git a/api/hypercore/remove_columnstore_policy.md b/api/hypercore/remove_columnstore_policy.md index b501e1f38c..0b409c32aa 100644 --- a/api/hypercore/remove_columnstore_policy.md +++ b/api/hypercore/remove_columnstore_policy.md @@ -26,13 +26,13 @@ You see the columnstore policies in the [informational views][informational-view - **Remove the columnstore policy from the `cpu` table**: ``` sql - SELECT remove_columnstore_policy('cpu'); + CALL remove_columnstore_policy('cpu'); ``` - **Remove the columnstore policy from the `cpu_weekly` continuous aggregate**: ``` sql - SELECT remove_columnstore_policy('cpu_weekly'); + CALL remove_columnstore_policy('cpu_weekly'); ``` ## Arguments From 8c87b3c1137381cd562bc8e4fd08730798f4c093 Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 15 Jan 2025 16:49:01 +0100 Subject: [PATCH 19/31] chore: update on review. --- api/hypercore/add_columnstore_policy.md | 2 +- api/hypercore/chunk_columnstore_stats.md | 2 +- api/hypercore/convert_to_columnstore.md | 2 +- api/hypercore/convert_to_rowstore.md | 16 ++++------------ api/hypercore/hypertable_columnstore_stats.md | 2 +- api/hypercore/remove_columnstore_policy.md | 2 +- 6 files changed, 9 insertions(+), 17 deletions(-) diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index ab210780ca..cef5df24e1 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -7,7 +7,7 @@ tags: [scheduled jobs, background jobs, automation framework] products: [cloud, self_hosted] api: license: community - type: function + type: procedure --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; diff --git a/api/hypercore/chunk_columnstore_stats.md b/api/hypercore/chunk_columnstore_stats.md index 50e60308eb..cda563a2b7 100644 --- a/api/hypercore/chunk_columnstore_stats.md +++ b/api/hypercore/chunk_columnstore_stats.md @@ -6,7 +6,7 @@ keywords: [columnstore, hypercore, statistics, chunks, information] tags: [disk space, schemas, size] api: license: community - type: function + type: procedure --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index 2b7f6c8c2d..cece6e37f1 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -6,7 +6,7 @@ keywords: [columnstore, rowstore, hypercore] tags: [chunks, hypercore] api: license: community - type: function + type: procedure --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index 7877922f80..41cb3586c0 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -5,7 +5,7 @@ topics: [columnstore, hypercore] keywords: [columnstore, hypercore, rowstore, chunks, backfilling] api: license: community - type: function + type: procedure --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; @@ -38,17 +38,9 @@ To modify or add a lot of data to a chunk: 1. **Convert the chunks to update back to the rowstore** - - Convert single chunk: - - ``` sql - CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); - ``` - - - Convert all chunks in a hypertable named `metrics`: - - ``` sql - SELECT convert_to_rowstore(c, true) FROM show_chunks('metrics') c; - ``` + ``` sql + CALL convert_to_rowstore('_timescaledb_internal._hyper_2_2_chunk'); + ``` 1. **Update the data in the chunk you added to the rowstore** diff --git a/api/hypercore/hypertable_columnstore_stats.md b/api/hypercore/hypertable_columnstore_stats.md index 4d24da5349..446deb74a6 100644 --- a/api/hypercore/hypertable_columnstore_stats.md +++ b/api/hypercore/hypertable_columnstore_stats.md @@ -6,7 +6,7 @@ keywords: [columnstore, hypercore, hypertables, information] tags: [statistics, size] api: license: community - type: function + type: procedure --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; diff --git a/api/hypercore/remove_columnstore_policy.md b/api/hypercore/remove_columnstore_policy.md index 0b409c32aa..08bb519745 100644 --- a/api/hypercore/remove_columnstore_policy.md +++ b/api/hypercore/remove_columnstore_policy.md @@ -6,7 +6,7 @@ keywords: [hypercore, columnstore, policies, remove] tags: [delete, drop] api: license: community - type: function + type: procedure --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; From 25a2bcc79db447bebf2e40220dfff90edd52c0b0 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 16 Jan 2025 09:46:17 +0100 Subject: [PATCH 20/31] chore: add enable_segmentwise_recompression. --- api/compression/alter_table_compression.md | 13 ++++++++----- api/hypercore/alter_table.md | 19 ++++++++++--------- 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/api/compression/alter_table_compression.md b/api/compression/alter_table_compression.md index 500ee3c587..6d44be5ea1 100644 --- a/api/compression/alter_table_compression.md +++ b/api/compression/alter_table_compression.md @@ -39,11 +39,14 @@ ALTER TABLE SET (timescaledb.compress, ## Optional arguments -|Name|Type|Description| -|-|-|-| -|`timescaledb.compress_orderby`|TEXT|Order used by compression, specified in the same way as the ORDER BY clause in a SELECT query. The default is the descending order of the hypertable's time column.| -|`timescaledb.compress_segmentby`|TEXT|Column list on which to key the compressed segments. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. The default is no `segment by` columns.| -|`timescaledb.compress_chunk_time_interval`|TEXT|EXPERIMENTAL: Set compressed chunk time interval used to roll chunks into. This parameter compresses every chunk, and then irreversibly merges it into a previous adjacent chunk if possible, to reduce the total number of chunks in the hypertable. Note that chunks will not be split up during decompression. It should be set to a multiple of the current chunk interval. This option can be changed independently of other compression settings and does not require the `timescaledb.compress` argument.| +|Name|Type| Description | +|-|-|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|`timescaledb.compress_orderby`|TEXT| Order used by compression, specified in the same way as the ORDER BY clause in a SELECT query. The default is the descending order of the hypertable's time column. | +|`timescaledb.compress_segmentby`|TEXT| Column list on which to key the compressed segments. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. The default is no `segment by` columns. | +|`timescaledb.enable_segmentwise_recompression`|TEXT| Set to `OFF` to disable segmentwise recompression on compressed chunks. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. The default is `ON`. | +|`timescaledb.compress_chunk_time_interval`|TEXT| EXPERIMENTAL: Set compressed chunk time interval used to roll chunks into. This parameter compresses every chunk, and then irreversibly merges it into a previous adjacent chunk if possible, to reduce the total number of chunks in the hypertable. Note that chunks will not be split up during decompression. It should be set to a multiple of the current chunk interval. This option can be changed independently of other compression settings and does not require the `timescaledb.compress` argument. | + +|`timescaledb.enable_segmentwise_recompression`|TEXT| ON| ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. | ## Parameters diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index 87e76a361c..89e21f09fe 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -61,15 +61,16 @@ ALTER TABLE SET (timescaledb.enable_columnstore, ); ``` -| Name | Type | Default | Required | Description | -|--|--|------------------------------------------------------|--|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -|`table_name`|TEXT| - | ✖ | The hypertable to enable columstore for. | -|`timescaledb.enable_columnstore`|BOOLEAN| true | ✖ | Enable columnstore. | -|`timescaledb.orderby`|TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | -|`timescaledb.segmentby`|TEXT| No segementation by column.| ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | -|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | -|`timescaledb.compress_chunk_time_interval`|TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | -|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | +| Name | Type | Default | Required | Description | +|--|--|------------------------------------------------------|--|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|`table_name`|TEXT| - | ✖ | The hypertable to enable columstore for. | +|`timescaledb.enable_columnstore`|BOOLEAN| `true` | ✖ | Enable columnstore. | +|`timescaledb.orderby`|TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | +|`timescaledb.segmentby`|TEXT| No segementation by column. | ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | +|`timescaledb.enable_segmentwise_recompression`|TEXT| ON| ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. | +|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | +|`timescaledb.compress_chunk_time_interval`|TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | +|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | From c8495175c65d540545c5055625bca286f2665ee1 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 16 Jan 2025 10:17:50 +0100 Subject: [PATCH 21/31] chore: add enable_segmentwise_recompression. --- api/compression/alter_table_compression.md | 5 ++--- api/hypercore/alter_table.md | 10 +++++----- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/api/compression/alter_table_compression.md b/api/compression/alter_table_compression.md index 6d44be5ea1..1eb420a43f 100644 --- a/api/compression/alter_table_compression.md +++ b/api/compression/alter_table_compression.md @@ -40,13 +40,12 @@ ALTER TABLE SET (timescaledb.compress, ## Optional arguments |Name|Type| Description | -|-|-|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|-|-|--| |`timescaledb.compress_orderby`|TEXT| Order used by compression, specified in the same way as the ORDER BY clause in a SELECT query. The default is the descending order of the hypertable's time column. | |`timescaledb.compress_segmentby`|TEXT| Column list on which to key the compressed segments. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. The default is no `segment by` columns. | -|`timescaledb.enable_segmentwise_recompression`|TEXT| Set to `OFF` to disable segmentwise recompression on compressed chunks. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. The default is `ON`. | +|`timescaledb.enable_segmentwise_recompression`|TEXT| Set to `OFF` to disable segmentwise recompression on compressed chunks. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. The default is `ON`. | |`timescaledb.compress_chunk_time_interval`|TEXT| EXPERIMENTAL: Set compressed chunk time interval used to roll chunks into. This parameter compresses every chunk, and then irreversibly merges it into a previous adjacent chunk if possible, to reduce the total number of chunks in the hypertable. Note that chunks will not be split up during decompression. It should be set to a multiple of the current chunk interval. This option can be changed independently of other compression settings and does not require the `timescaledb.compress` argument. | -|`timescaledb.enable_segmentwise_recompression`|TEXT| ON| ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. | ## Parameters diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index 89e21f09fe..ddb94a4a24 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -58,20 +58,20 @@ ALTER TABLE SET (timescaledb.enable_columnstore, timescaledb.orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', timescaledb.segmentby = ' [, ...]', timescaledb.compress_chunk_time_interval='interval' + timescaledb.enable_segmentwise_recompression = 'ON' | 'OFF' ); ``` | Name | Type | Default | Required | Description | -|--|--|------------------------------------------------------|--|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|--|--|------------------------------------------------------|--|--| |`table_name`|TEXT| - | ✖ | The hypertable to enable columstore for. | |`timescaledb.enable_columnstore`|BOOLEAN| `true` | ✖ | Enable columnstore. | |`timescaledb.orderby`|TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | |`timescaledb.segmentby`|TEXT| No segementation by column. | ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | -|`timescaledb.enable_segmentwise_recompression`|TEXT| ON| ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. | -|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | +|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | |`timescaledb.compress_chunk_time_interval`|TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | -|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | - +|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`.| +|`timescaledb.enable_segmentwise_recompression`|TEXT| ON | ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. | [chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/ From e4a4a571d9f64400ee772c986e3c03e959655de1 Mon Sep 17 00:00:00 2001 From: Iain Date: Fri, 17 Jan 2025 12:41:36 +0100 Subject: [PATCH 22/31] chore: update deprecation notices. --- _partials/_multi-node-deprecation.md | 3 ++- api/compression/index.md | 3 ++- api/distributed-hypertables.md | 4 ++-- api/hypercore/add_columnstore_policy.md | 2 +- api/hypercore/alter_table.md | 2 +- api/hypercore/chunk_columnstore_settings.md | 2 +- api/hypercore/chunk_columnstore_stats.md | 2 +- api/hypercore/convert_to_columnstore.md | 2 +- api/hypercore/convert_to_rowstore.md | 2 +- api/hypercore/hypertable_columnstore_settings.md | 2 +- api/hypercore/hypertable_columnstore_stats.md | 4 ++-- api/page-index/page-index.js | 4 ++-- 12 files changed, 17 insertions(+), 15 deletions(-) diff --git a/_partials/_multi-node-deprecation.md b/_partials/_multi-node-deprecation.md index a5e1eda837..31f68e1bd0 100644 --- a/_partials/_multi-node-deprecation.md +++ b/_partials/_multi-node-deprecation.md @@ -1,9 +1,10 @@ -[Multi-node support is deprecated][multi-node-deprecation]. +[Multi-node support is sunsetted][multi-node-deprecation]. TimescaleDB v2.13 is the last release that includes multi-node support for PostgreSQL versions 13, 14, and 15. + [multi-node-deprecation]: https://github.com/timescale/timescaledb/blob/main/docs/MultiNodeDeprecation.md diff --git a/api/compression/index.md b/api/compression/index.md index a665241aff..ce5493af13 100644 --- a/api/compression/index.md +++ b/api/compression/index.md @@ -6,7 +6,7 @@ tags: [hypertables] --- import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; -# Compression Community +# Compression: deprecated, use [Hypercore][hypercore] Community Before you set up compression, you need to [configure the hypertable for compression][configure-compression] and then @@ -62,3 +62,4 @@ You can also use advanced insert statements like `ON CONFLICT` and `RETURNING`. [compress_chunk]: /api/:currentVersion:/compression/compress_chunk/ [configure-compression]: /api/:currentVersion:/compression/alter_table_compression/ [skipscan]: /use-timescale/:currentVersion:/query-data/skipscan/ +[hypercore]: /api/:currentVersion:/hypercore/ diff --git a/api/distributed-hypertables.md b/api/distributed-hypertables.md index 9919824880..f9a42cfdc9 100644 --- a/api/distributed-hypertables.md +++ b/api/distributed-hypertables.md @@ -1,5 +1,5 @@ --- -title: Distributed hypertables +title: Distributed hypertables: sunsetted excerpt: Create and manage distributed hypertables keywords: [distributed hypertables] --- @@ -8,7 +8,7 @@ import MultiNodeDeprecation from "versionContent/_partials/_multi-node-deprecati -# Distributed Hypertables Community +# Distributed hypertables: sunsetted Community Distributed hypertables are an extension of regular hypertables, available when using a [multi-node installation][getting-started-multi-node] of TimescaleDB. diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index cef5df24e1..4f63d817cf 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -1,7 +1,7 @@ --- api_name: add_columnstore_policy() excerpt: Set a policy to automatically move chunks in a hypertable to the columnstore when they reach a given age. -topics: [columnstore, hypercore, jobs] +topics: [hypercore, columnstore, jobs] keywords: [columnstore, hypercore, policies] tags: [scheduled jobs, background jobs, automation framework] products: [cloud, self_hosted] diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index ddb94a4a24..dd42cd4006 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -1,7 +1,7 @@ --- api_name: ALTER TABLE (Hypercore) excerpt: Enable the columnstore for a hypertable. -topics: [columnstore, hypercore] +topics: [hypercore, columnstore] keywords: [columnstore, hypercore] tags: [settings, hypertables, alter, change] api: diff --git a/api/hypercore/chunk_columnstore_settings.md b/api/hypercore/chunk_columnstore_settings.md index 249b2cf77f..4940ca2599 100644 --- a/api/hypercore/chunk_columnstore_settings.md +++ b/api/hypercore/chunk_columnstore_settings.md @@ -1,7 +1,7 @@ --- api_name: timescaledb_information.chunk_columnstore_settings excerpt: Get information about settings on each chunk in the columnstore -topics: [information, columnstore, hypercore, chunk] +topics: [hypercore, information, columnstore, chunk] keywords: [columnstore, hypercore, chunk, information] tags: [chunk, columnstore settings] api: diff --git a/api/hypercore/chunk_columnstore_stats.md b/api/hypercore/chunk_columnstore_stats.md index cda563a2b7..c903a95760 100644 --- a/api/hypercore/chunk_columnstore_stats.md +++ b/api/hypercore/chunk_columnstore_stats.md @@ -1,7 +1,7 @@ --- api_name: chunk_columnstore_stats() excerpt: Get statistics about chunks in the columnstore -topics: [columnstore, hypercore] +topics: [hypercore, columnstore] keywords: [columnstore, hypercore, statistics, chunks, information] tags: [disk space, schemas, size] api: diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index cece6e37f1..5a82673dcb 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -1,7 +1,7 @@ --- api_name: convert_to_columnstore() excerpt: Manually add a chunk to thee columnstore -topics: [columnstore, rowstore, hypercore] +topics: [hypercore, columnstore, rowstore] keywords: [columnstore, rowstore, hypercore] tags: [chunks, hypercore] api: diff --git a/api/hypercore/convert_to_rowstore.md b/api/hypercore/convert_to_rowstore.md index 41cb3586c0..f388470442 100644 --- a/api/hypercore/convert_to_rowstore.md +++ b/api/hypercore/convert_to_rowstore.md @@ -1,7 +1,7 @@ --- api_name: convert_to_rowstore() excerpt: Move a chunk from the columnstore to the rowstore -topics: [columnstore, hypercore] +topics: [hypercore, columnstore] keywords: [columnstore, hypercore, rowstore, chunks, backfilling] api: license: community diff --git a/api/hypercore/hypertable_columnstore_settings.md b/api/hypercore/hypertable_columnstore_settings.md index e8f1a60024..3752e7b8ae 100644 --- a/api/hypercore/hypertable_columnstore_settings.md +++ b/api/hypercore/hypertable_columnstore_settings.md @@ -1,7 +1,7 @@ --- api_name: timescaledb_information.hypertable_columnstore_settings excerpt: Get information about columnstore settings for all hypertables -topics: [information, hypercore, columnstore, hypertable] +topics: [hypercore, information, columnstore, hypertable] keywords: [columnstore, hypercore, hypertable, information] tags: [hypertable columnstore, columnstore settings] api: diff --git a/api/hypercore/hypertable_columnstore_stats.md b/api/hypercore/hypertable_columnstore_stats.md index 446deb74a6..a6558e3a59 100644 --- a/api/hypercore/hypertable_columnstore_stats.md +++ b/api/hypercore/hypertable_columnstore_stats.md @@ -1,8 +1,8 @@ --- api_name: hypertable_columnstore_stats() excerpt: Get columnstore statistics related to the columnstore -topics: [columnstore, hypercore] -keywords: [columnstore, hypercore, hypertables, information] +topics: [hypercore, columnstore] +keywords: [hypercore, columnstore, hypertables, information] tags: [statistics, size] api: license: community diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index c1f2fa0a3d..f7fa149fb2 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -560,7 +560,7 @@ module.exports = [ "An overview of what different tags represent in the API section of Timescale Documentation.", }, { - title: "Compression (deprecated)", + title: "Compression: deprecated, use Hypercore", href: "compression", description: "We highly recommend reading the blog post and tutorial about compression before trying to set it up for the first time.", @@ -600,7 +600,7 @@ module.exports = [ ], }, { - title: "Distributed hypertables (deprecated)", + title: "Distributed hypertables: sunsetted v2.14.x", type: "directory", href: "distributed-hypertables", children: [ From 8025d119983a98afe5cfe369b51217ad044e5e6a Mon Sep 17 00:00:00 2001 From: Iain Date: Fri, 17 Jan 2025 13:24:45 +0100 Subject: [PATCH 23/31] chore: update deprecation notices. --- api/compression/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/compression/index.md b/api/compression/index.md index ce5493af13..e12c51713f 100644 --- a/api/compression/index.md +++ b/api/compression/index.md @@ -6,7 +6,7 @@ tags: [hypertables] --- import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; -# Compression: deprecated, use [Hypercore][hypercore] Community +# Compression: deprecated use Hypercore Community Before you set up compression, you need to [configure the hypertable for compression][configure-compression] and then From ce662f7279bb2cfdb63dd20cb8982780506dc069 Mon Sep 17 00:00:00 2001 From: Iain Date: Fri, 17 Jan 2025 14:50:52 +0100 Subject: [PATCH 24/31] chore: remove stuff breaking the build. --- api/compression/index.md | 2 +- api/distributed-hypertables.md | 4 ++-- api/page-index/page-index.js | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/api/compression/index.md b/api/compression/index.md index e12c51713f..98120d0d8b 100644 --- a/api/compression/index.md +++ b/api/compression/index.md @@ -6,7 +6,7 @@ tags: [hypertables] --- import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; -# Compression: deprecated use Hypercore Community +# Compression (deprecated use Hypercore) Community Before you set up compression, you need to [configure the hypertable for compression][configure-compression] and then diff --git a/api/distributed-hypertables.md b/api/distributed-hypertables.md index f9a42cfdc9..f050a1e21e 100644 --- a/api/distributed-hypertables.md +++ b/api/distributed-hypertables.md @@ -1,5 +1,5 @@ --- -title: Distributed hypertables: sunsetted +title: Distributed hypertables ( sunsetted v2.14.x ) excerpt: Create and manage distributed hypertables keywords: [distributed hypertables] --- @@ -8,7 +8,7 @@ import MultiNodeDeprecation from "versionContent/_partials/_multi-node-deprecati -# Distributed hypertables: sunsetted Community +# Distributed hypertables ( sunsetted v2.14.x) Community Distributed hypertables are an extension of regular hypertables, available when using a [multi-node installation][getting-started-multi-node] of TimescaleDB. diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index f7fa149fb2..9d4f5a6c18 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -560,7 +560,7 @@ module.exports = [ "An overview of what different tags represent in the API section of Timescale Documentation.", }, { - title: "Compression: deprecated, use Hypercore", + title: "Compression (deprecated, use Hypercore)", href: "compression", description: "We highly recommend reading the blog post and tutorial about compression before trying to set it up for the first time.", @@ -600,7 +600,7 @@ module.exports = [ ], }, { - title: "Distributed hypertables: sunsetted v2.14.x", + title: "Distributed hypertables (sunsetted v2.14.x)", type: "directory", href: "distributed-hypertables", children: [ From 7ca04c92b99e342cd8305abc73e753fd190760ce Mon Sep 17 00:00:00 2001 From: Iain Date: Sat, 18 Jan 2025 17:57:22 +0100 Subject: [PATCH 25/31] chore: remove stuff breaking the build. --- api/hypercore/add_columnstore_policy.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index 4f63d817cf..2832822d47 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -85,13 +85,13 @@ Calls to `add_columnstore_policy` require either `after` or `created_before`, bu | Name | Type | Default | Required | Description | |--|--|--|--|--| -| `hypertable` |REGCLASS| - | ✔ | Name of the hypertable or continuous aggregate to run this [job][job] on.| -| `after` |INTERVAL or INTEGER|- | ✖ | Add chunks containing data older than `now - {after}::interval` to the columnstore.
Use an object type that matchs the time column type in `hypertable`:
  • TIMESTAMP, TIMESTAMPTZ, or DATE: use an INTERVAL type.
  • Integer-based timestamps : set an integer type using the [integer_now_func][set_integer_now_func].
`after` is mutually exclusive with `created_before`. | +| `hypertable` |REGCLASS| - | ✔ | Name of the hypertable or continuous aggregate to run this [job][job] on.| +| `after` |INTERVAL or INTEGER|- | ✖ | Add chunks containing data older than `now - {after}::interval` to the columnstore.
Use an object type that matchs the time column type in `hypertable`:
  • TIMESTAMP, TIMESTAMPTZ, or DATE: use an INTERVAL type.
  • Integer-based timestamps : set an integer type using the [integer_now_func][set_integer_now_func].
`after` is mutually exclusive with `created_before`. | | `created_before` |INTERVAL| NULL | ✖ | Add chunks with a creation time of `now() - created_before` to the columnstore.
`created_before` is
  • Not supported for continuous aggregates.
  • Mutually exclusive with `after`.
| -| `schedule_interval` |INTERVAL| 12 hours when [chunk_time_interval][chunk_time_interval] >= `1 day` for `hypertable`. Otherwise `chunk_time_interval` / `2`. | ✖ | Set the interval between the finish time of the last execution of this policy and the next start.| -| `initial_start` |TIMESTAMPTZ| The interval from the finish time of the last execution to the [next_start][next-start].| ✖| Set the time this job is first run. This is also the time that `next_start` is calculated from.| -| `timezone` |TEXT| UTC. However, daylight savings time(DST) changes may shift this alignment. | ✖ | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`.| -| `if_not_exists` |BOOLEAN| `false` | ✖ | Set to `true` so this job fails with a warning rather than an error if a columnstore policy already exists on `hypertable` | +| `schedule_interval` |INTERVAL| 12 hours when [chunk_time_interval][chunk_time_interval] >= `1 day` for `hypertable`. Otherwise `chunk_time_interval` / `2`. | ✖ | Set the interval between the finish time of the last execution of this policy and the next start.| +| `initial_start` |TIMESTAMPTZ| The interval from the finish time of the last execution to the [next_start][next-start].| ✖| Set the time this job is first run. This is also the time that `next_start` is calculated from.| +| `timezone` |TEXT| UTC. However, daylight savings time(DST) changes may shift this alignment. | ✖ | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`.| +| `if_not_exists` |BOOLEAN| `false` | ✖ | Set to `true` so this job fails with a warning rather than an error if a columnstore policy already exists on `hypertable` | From 629d4a89e072bac7030e044673657590a022ccde Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 23 Jan 2025 06:10:47 -0700 Subject: [PATCH 26/31] chore: updates on review. --- api/compression/add_compression_policy.md | 4 ++-- api/compression/alter_table_compression.md | 4 ++-- api/compression/chunk_compression_stats.md | 4 ++-- api/compression/compress_chunk.md | 3 ++- api/compression/decompress_chunk.md | 4 ++-- api/compression/hypertable_compression_stats.md | 4 ++-- api/compression/index.md | 5 ++++- api/compression/recompress_chunk.md | 3 ++- api/compression/remove_compression_policy.md | 4 ++-- api/hypercore/convert_to_columnstore.md | 9 ++++++--- 10 files changed, 26 insertions(+), 18 deletions(-) diff --git a/api/compression/add_compression_policy.md b/api/compression/add_compression_policy.md index f0b7e7e52a..66637f260f 100644 --- a/api/compression/add_compression_policy.md +++ b/api/compression/add_compression_policy.md @@ -12,6 +12,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # add_compression_policy() + Replaced by add_columnstore_policy(). + Allows you to set a policy by which the system compresses a chunk automatically in the background after it reaches a given age. @@ -24,8 +26,6 @@ command. To enable compression on continuous aggregates, use the command. To view the policies that you set or the policies that already exist, see [informational views][informational-views]. - Replaced by add_columnstore_policy(). - ## Required arguments |Name|Type|Description| diff --git a/api/compression/alter_table_compression.md b/api/compression/alter_table_compression.md index 1eb420a43f..1b6827e308 100644 --- a/api/compression/alter_table_compression.md +++ b/api/compression/alter_table_compression.md @@ -12,6 +12,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # ALTER TABLE (Compression) + Replaced by ALTER TABLE (Hypercore). + 'ALTER TABLE' statement is used to turn on compression and set compression options. @@ -29,8 +31,6 @@ ALTER TABLE SET (timescaledb.compress, ); ``` - Replaced by ALTER TABLE (Hypercore). - ## Required arguments |Name|Type|Description| diff --git a/api/compression/chunk_compression_stats.md b/api/compression/chunk_compression_stats.md index 6abb8ddf03..989a015053 100644 --- a/api/compression/chunk_compression_stats.md +++ b/api/compression/chunk_compression_stats.md @@ -12,6 +12,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # chunk_compression_stats() Community + Replaced by chunk_columnstore_stats(). + Get chunk-specific statistics related to hypertable compression. All sizes are in bytes. @@ -21,8 +23,6 @@ the chunk. An insert into a compressed chunk does not update the compressed sizes. For more information about how to compute chunk sizes, see the `chunks_detailed_size` section. - Replaced by chunk_columnstore_stats(). - ### Required arguments |Name|Type|Description| diff --git a/api/compression/compress_chunk.md b/api/compression/compress_chunk.md index 4e858e2a44..d0779260de 100644 --- a/api/compression/compress_chunk.md +++ b/api/compression/compress_chunk.md @@ -13,6 +13,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # compress_chunk() Community + Replaced by convert_to_columnstore(). + The `compress_chunk` function is used to compress (or recompress, if necessary) a specific chunk. This is most often used instead of the [`add_compression_policy`][add_compression_policy] function, when a user @@ -29,7 +31,6 @@ You can get a list of chunks belonging to a hypertable using the [`show_chunks` function](/api/latest/hypertable/show_chunks/). - Replaced by convert_to_columnstore(). ### Required arguments diff --git a/api/compression/decompress_chunk.md b/api/compression/decompress_chunk.md index ffab2a595f..5c8c0c9b4d 100644 --- a/api/compression/decompress_chunk.md +++ b/api/compression/decompress_chunk.md @@ -11,6 +11,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # decompress_chunk() Community + Replaced by convert_to_rowstore(). + If you need to modify or add a lot of data to a chunk that has already been compressed, you should decompress the chunk first. This is especially useful for backfilling old data. @@ -23,8 +25,6 @@ turn the policy back on. The database automatically recompresses your chunks in the next scheduled job. - Replaced by convert_to_rowstore(). - ### Required arguments |Name|Type|Description| diff --git a/api/compression/hypertable_compression_stats.md b/api/compression/hypertable_compression_stats.md index 99babe8ac2..8e244e968f 100644 --- a/api/compression/hypertable_compression_stats.md +++ b/api/compression/hypertable_compression_stats.md @@ -12,6 +12,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # hypertable_compression_stats() Community + Replaced by hypertable_columnstore_stats(). + Get statistics related to hypertable compression. All sizes are in bytes. For more information about using hypertables, including chunk size partitioning, @@ -20,8 +22,6 @@ see the [hypertable section][hypertable-docs]. For more information about compression, see the [compression sction][compression-docs]. - Replaced by hypertable_columnstore_stats(). - ### Required arguments |Name|Type|Description| diff --git a/api/compression/index.md b/api/compression/index.md index 98120d0d8b..a5b5722f2e 100644 --- a/api/compression/index.md +++ b/api/compression/index.md @@ -8,6 +8,10 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # Compression (deprecated use Hypercore) Community + Replaced by Hypercore. + +Compression functionality is included in Hypercore. + Before you set up compression, you need to [configure the hypertable for compression][configure-compression] and then [set up a compression policy][add_compression_policy]. @@ -31,7 +35,6 @@ Compressed chunks have the following limitations: after constraint creation. * [Timescale SkipScan][skipscan] does not currently work on compressed chunks. - Replaced by Hypercore. ## Restrictions diff --git a/api/compression/recompress_chunk.md b/api/compression/recompress_chunk.md index 9b9c996b65..367c7ee9b2 100644 --- a/api/compression/recompress_chunk.md +++ b/api/compression/recompress_chunk.md @@ -13,6 +13,8 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # recompress_chunk() + Replaced by convert_to_columnstore(). + Recompresses a compressed chunk that had more data inserted after compression. ```sql @@ -43,7 +45,6 @@ the procedure with `CALL`. Don't use a `SELECT` statement. chunk for the first time, use [`compress_chunk`](/api/latest/compression/compress_chunk/). - Replaced by convert_to_columnstore(). ## Required arguments diff --git a/api/compression/remove_compression_policy.md b/api/compression/remove_compression_policy.md index 48e48e0de1..de0174c56c 100644 --- a/api/compression/remove_compression_policy.md +++ b/api/compression/remove_compression_policy.md @@ -12,12 +12,12 @@ import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; # remove_compression_policy() + Replaced by remove_columnstore_policy(). + If you need to remove the compression policy. To restart policy-based compression you need to add the policy again. To view the policies that already exist, see [informational views][informational-views]. - Replaced by remove_columnstore_policy(). - ### Required arguments |Name|Type|Description| diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index 5a82673dcb..b85f7444cb 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -43,9 +43,12 @@ To retrieve the chunks belonging to a hypertable, call [`show_chunks`](/api/late ## Returns -|Column| Type | Description | -|-|-------------------------------------------------|-------------------------------------------------| -| `convert_to_columnstore` | REGCLASS | The name of the chunk added to the columnstore. | +Calls to `convert_to_columnstore` return one of the following: + +| Column | Type | Description | +|-----------------------------|----------|---------------------------------------------------------------| +| `name` \| descriptive words | Table | A table-like result set with zero or more rows containing ... | +| chunk name | REGCLASS | The name of the chunk added to the columnstore. | [add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ From da173a727f4cde2f60ef4d137c86b3dfffa11cb4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Erik=20Nordstr=C3=B6m?= <819732+erimatnor@users.noreply.github.com> Date: Thu, 23 Jan 2025 15:06:59 +0100 Subject: [PATCH 27/31] Add API description for merge_chunks() (#3738) TimescaleDB 2.18 includes a new procedure to merge chunks. This change adds the API documentation. --- api/merge_chunks.md | 56 ++++++++++++++++++++++++++++++++++++ api/page-index/page-index.js | 4 +++ 2 files changed, 60 insertions(+) create mode 100644 api/merge_chunks.md diff --git a/api/merge_chunks.md b/api/merge_chunks.md new file mode 100644 index 0000000000..73924abae3 --- /dev/null +++ b/api/merge_chunks.md @@ -0,0 +1,56 @@ +--- +api_name: merge_chunks() +excerpt: Merge two or more chunks into one chunk +topics: [hypertables] +keywords: [hypertables, chunk, merge] +api: + license: community + type: procedure +--- + +# merge_chunks() + +Merge two or more chunks into one. + +The partition boundaries for the new chunk is the union of all partitions of the merged chunks. +The new chunk retains the name, constraints, and triggers of the _first_ chunk in the partition order. + +You can only merge chunks that have directly adjacent partitions. It is not possible to merge +chunks that have another chunk, or an empty range between them in any of the partitioning +dimensions. + +In this first release, chunk merging has the following limitations. You cannot: + +* Merge compressed chunks +* Merge chunks using table access methods other than heap +* Merge chunks with tiered data +* Read or write from the chunks while they are being merged + + + +## Samples + +- Merge two chunks: + + ```sql + CALL merge_chunks('_timescaledb_internal._hyper_1_1_chunk', '_timescaledb_internal._hyper_1_2_chunk'); + ``` + +- Merge more than two chunks: + + ```sql + CALL merge_chunks('{_timescaledb_internal._hyper_1_1_chunk, _timescaledb_internal._hyper_1_2_chunk, _timescaledb_internal._hyper_1_3_chunk}'); + ``` + + +## Arguments + +You can merge either two chunks, or an arbitrary number of chunks specified as an array of chunk identifiers. +When you call `merge_chunks`, you must specify either `chunk1` and `chunk2`, or `chunks`. You cannot use both +arguments. + + +| Name | Type | Default | Required | Description | +|--------------------|-------------|--|--|------------------------------------------------| +| `chunk1`, `chunk2` | REGCLASS | - | ✖ | The two chunk to merge in partition order | +| `chunks` | REGCLASS[] |- | ✖ | The array of chunks to merge in partition order | diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 9d4f5a6c18..e6e70dfdce 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -34,6 +34,10 @@ module.exports = [ title: "reorder_chunk", href: "reorder_chunk", }, + { + title: "merge_chunks", + href: "merge_chunks", + }, { title: "move_chunk", href: "move_chunk", From 41564b4bbd43846b10769f4716ec6ba5193b9b80 Mon Sep 17 00:00:00 2001 From: Iain Date: Tue, 28 Jan 2025 15:28:05 +0100 Subject: [PATCH 28/31] chore: update return table. --- _partials/_hypercore_policy_workflow.md | 8 ++++++++ api/hypercore/convert_to_columnstore.md | 11 +++++------ api/hypercore/index.md | 1 - 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/_partials/_hypercore_policy_workflow.md b/_partials/_hypercore_policy_workflow.md index 1d6e705884..503bf63971 100644 --- a/_partials/_hypercore_policy_workflow.md +++ b/_partials/_hypercore_policy_workflow.md @@ -55,3 +55,11 @@ ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = false); ``` See [alter_table_hypercore][alter_table_hypercore]. + + +[alter_table_hypercore]: /api/:currentVersion:/hypercore/alter_table/ +[alter_job]: /api/:currentVersion:/actions/alter_job/ +[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ +[informational-views]: /api/:currentVersion:/informational-views/jobs/ +[remove_columnstore_policy]: /api/:currentVersion:/hypercore/remove_columnstore_policy/ +[convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index b85f7444cb..08b7325ee2 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -39,16 +39,15 @@ To retrieve the chunks belonging to a hypertable, call [`show_chunks`](/api/late |----------------------|--|---------|--|----------------------------------------------------------------------------------------------------------------| | `chunk` | REGCLASS | - |✔| Name of the chunk to add to the columnstore. | | `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the columnstore. | -| `recompress` | BOOLEAN | `false` |✖| Set to `true` to add a chunk that had more data inserted after being added to the columnstore. | +| `recompress` | BOOLEAN | `false` |✖| Set to `true` to recompress a chunk that had more data inserted after it was added to the columnstore. | ## Returns -Calls to `convert_to_columnstore` return one of the following: +Calls to `convert_to_columnstore` return: -| Column | Type | Description | -|-----------------------------|----------|---------------------------------------------------------------| -| `name` \| descriptive words | Table | A table-like result set with zero or more rows containing ... | -| chunk name | REGCLASS | The name of the chunk added to the columnstore. | +| Column | Type | Description | +|-------------------|--------------------|----------------------------------------------------------------------------------------------------| +| `chunk name` or `table` | REGCLASS or String | The name of the chunk added to the columnstore, or a table-like result set with zero or more rows. | [add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ diff --git a/api/hypercore/index.md b/api/hypercore/index.md index cec81e7a1f..599ef5810f 100644 --- a/api/hypercore/index.md +++ b/api/hypercore/index.md @@ -84,7 +84,6 @@ Chunks in the columnstore have the following limitations: [compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ -[convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ [informational-views]: /api/:currentVersion:/informational-views/jobs/ [skipscan]: /use-timescale/:currentVersion:/query-data/skipscan/ [add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ From 80b0e3de605be3c19d5d60f759472246b6ca9f89 Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 29 Jan 2025 13:04:41 +0100 Subject: [PATCH 29/31] chore: change from deprecated to Old API. --- _partials/_deprecated_2_18_0.md | 2 +- api/compression/index.md | 2 +- api/page-index/page-index.js | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/_partials/_deprecated_2_18_0.md b/_partials/_deprecated_2_18_0.md index 4231729519..7b09a2ef7a 100644 --- a/_partials/_deprecated_2_18_0.md +++ b/_partials/_deprecated_2_18_0.md @@ -1 +1 @@ -Deprecated [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) +Old API from [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0) diff --git a/api/compression/index.md b/api/compression/index.md index a5b5722f2e..fa9f8e8c83 100644 --- a/api/compression/index.md +++ b/api/compression/index.md @@ -6,7 +6,7 @@ tags: [hypertables] --- import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx"; -# Compression (deprecated use Hypercore) Community +# Compression (Old API, use Hypercore) Community Replaced by Hypercore. diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index e6e70dfdce..495a4f43a1 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -564,7 +564,7 @@ module.exports = [ "An overview of what different tags represent in the API section of Timescale Documentation.", }, { - title: "Compression (deprecated, use Hypercore)", + title: "Compression (Old API, use Hypercore)", href: "compression", description: "We highly recommend reading the blog post and tutorial about compression before trying to set it up for the first time.", From e5135b4cf29f51debaa3b2f62d82d74d3788cd08 Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 29 Jan 2025 13:07:33 +0100 Subject: [PATCH 30/31] chore: change from deprecated to Old API. --- api/distributed-hypertables.md | 4 ++-- api/page-index/page-index.js | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/api/distributed-hypertables.md b/api/distributed-hypertables.md index f050a1e21e..78f727e849 100644 --- a/api/distributed-hypertables.md +++ b/api/distributed-hypertables.md @@ -1,5 +1,5 @@ --- -title: Distributed hypertables ( sunsetted v2.14.x ) +title: Distributed hypertables ( Sunsetted v2.14.x ) excerpt: Create and manage distributed hypertables keywords: [distributed hypertables] --- @@ -8,7 +8,7 @@ import MultiNodeDeprecation from "versionContent/_partials/_multi-node-deprecati -# Distributed hypertables ( sunsetted v2.14.x) Community +# Distributed hypertables ( Sunsetted v2.14.x) Community Distributed hypertables are an extension of regular hypertables, available when using a [multi-node installation][getting-started-multi-node] of TimescaleDB. diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 495a4f43a1..34dcf872bd 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -604,7 +604,7 @@ module.exports = [ ], }, { - title: "Distributed hypertables (sunsetted v2.14.x)", + title: "Distributed hypertables (Sunsetted v2.14.x)", type: "directory", href: "distributed-hypertables", children: [ From 5deb4f89efb1ba80b21f4a1a853a2920862444fb Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 29 Jan 2025 14:35:25 +0100 Subject: [PATCH 31/31] chore: update on review. --- api/hypercore/convert_to_columnstore.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/api/hypercore/convert_to_columnstore.md b/api/hypercore/convert_to_columnstore.md index 08b7325ee2..08b79dc5f2 100644 --- a/api/hypercore/convert_to_columnstore.md +++ b/api/hypercore/convert_to_columnstore.md @@ -35,11 +35,11 @@ To retrieve the chunks belonging to a hypertable, call [`show_chunks`](/api/late ## Arguments -| Name | Type | Default | Required | Description | -|----------------------|--|---------|--|----------------------------------------------------------------------------------------------------------------| -| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the columnstore. | -| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the columnstore. | -| `recompress` | BOOLEAN | `false` |✖| Set to `true` to recompress a chunk that had more data inserted after it was added to the columnstore. | +| Name | Type | Default | Required | Description | +|----------------------|--|---------|--|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `chunk` | REGCLASS | - |✔| Name of the chunk to add to the columnstore. | +| `if_not_columnstore` | BOOLEAN | `true` |✖| Set to `false` so this job fails with an error rather than a warning if `chunk` is already in the columnstore. | +| `recompress` | BOOLEAN | `false` |✖| Set to `true` to recompress data that was partially compressed as a result of modifications to `chunk`. This is usually more efficient, but in some cases it can result is a more expensive operation.
Set to `false` to completely decompress and recompress the data in `chunk`. | ## Returns