From ffc456e1be0932b5315870732678550293c6b363 Mon Sep 17 00:00:00 2001 From: Iain Date: Tue, 28 Jan 2025 08:02:12 +0100 Subject: [PATCH 1/5] chore: start integrate indexing in the doc. --- _partials/_cloud_self_configuration.md | 87 +++++++++++++++++ _partials/_early_access.md | 7 +- _partials/_hypercore_policy_workflow.md | 17 +++- api/hypercore/add_columnstore_policy.md | 2 +- api/hypercore/alter_materialized_view.md | 96 +++++++++++++++++++ api/hypercore/alter_table.md | 28 ++++-- api/hypercore/index.md | 2 +- api/page-index/page-index.js | 5 + .../configuration/timescaledb-config.md | 20 ++-- .../configuration/advanced-parameters.md | 13 ++- .../automate-hypercore-with-policies.md | 76 --------------- .../optimize-data-for-real-time-analytics.md | 6 +- 12 files changed, 252 insertions(+), 107 deletions(-) create mode 100644 _partials/_cloud_self_configuration.md create mode 100644 api/hypercore/alter_materialized_view.md delete mode 100644 use-timescale/hypercore/automate-hypercore-with-policies.md diff --git a/_partials/_cloud_self_configuration.md b/_partials/_cloud_self_configuration.md new file mode 100644 index 0000000000..5204daa9c6 --- /dev/null +++ b/_partials/_cloud_self_configuration.md @@ -0,0 +1,87 @@ +import EarlyAccess from "versionContent/_partials/_early_access.mdx"; + +## Policies + +### `timescaledb.max_background_workers (int)` + +Max background worker processes allocated to TimescaleDB. Set to at least 1 + +the number of databases loaded with a TimescaleDB extension in a PostgreSQL +instance. Default value is 16. + +### `timescaledb.enable_tiered_reads (bool)` + +Enable [tiered reads][enabling-data-tiering] to that you query your data normally when it's distributed across different storage tiers. +Your hypertable is spread across the tiers, so queries and `JOIN`s work and fetch the same data as usual. + +By default, tiered data is not accessed by queries. Querying tiered data may slow down query performance +as the data is not stored locally on Timescale's high-performance storage tier. + +## Hypercore features + +### `timescaledb.default_hypercore_use_access_method (bool)` + +The default value for `hypercore_use_access_method` for functions that have this parameter. This function is in `user` context, meaning that any user can set it for the session. The default value is `false`. + + + +## $SERVICE_LONG tuning + +### `timescaledb.disable_load (bool)` + +Disable the loading of the actual extension + +### `timescaledb.enable_cagg_reorder_groupby (bool)` +Enable group by reordering + +### `timescaledb.enable_chunk_append (bool)` +Enable chunk append node + +### `timescaledb.enable_constraint_aware_append (bool)` +Enable constraint-aware append scans + +### `timescaledb.enable_constraint_exclusion (bool)` +Enable constraint exclusion + +### `timescaledb.enable_job_execution_logging (bool)` +Enable job execution logging + +### `timescaledb.enable_optimizations (bool)` +Enable TimescaleDB query optimizations + +### `timescaledb.enable_ordered_append (bool)` +Enable ordered append scans + +### `timescaledb.enable_parallel_chunk_append (bool)` +Enable parallel chunk append node + +### `timescaledb.enable_runtime_exclusion (bool)` +Enable runtime chunk exclusion + +### `timescaledb.enable_tiered_reads (bool)` + +Enable [tiered reads][enabling-data-tiering] to that you query your data normally when it's distributed across different storage tiers. +Your hypertable is spread across the tiers, so queries and `JOIN`s work and fetch the same data as usual. + +By default, tiered data is not accessed by queries. Querying tiered data may slow down query performance +as the data is not stored locally on Timescale's high-performance storage tier. + + +### `timescaledb.enable_transparent_decompression (bool)` +Enable transparent decompression + + +### `timescaledb.restoring (bool)` +Install timescale in restoring mode + + +### `timescaledb.max_cached_chunks_per_hypertable (int)` +Maximum cached chunks + +### `timescaledb.max_open_chunks_per_insert (int)` +Maximum open chunks per insert + +### `timescaledb.max_tuples_decompressed_per_dml_transaction (int)` + +The max number of tuples that can be decompressed during an INSERT, UPDATE, or DELETE. + +[enabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/ diff --git a/_partials/_early_access.md b/_partials/_early_access.md index b27534d7f6..2062b00a2d 100644 --- a/_partials/_early_access.md +++ b/_partials/_early_access.md @@ -1,5 +1,2 @@ - -This feature is early access. Early access features might be subject to billing -changes in the future. If you have feedback, reach out to your customer success -manager, or [contact us](https://www.timescale.com/contact/). - +Early access: TimescaleDB v2.18.0 + diff --git a/_partials/_hypercore_policy_workflow.md b/_partials/_hypercore_policy_workflow.md index ac88c8fe1f..a57f3ebfb3 100644 --- a/_partials/_hypercore_policy_workflow.md +++ b/_partials/_hypercore_policy_workflow.md @@ -1,3 +1,5 @@ +import EarlyAccess from "versionContent/_partials/_early_access.mdx"; + 1. **Connect to your $SERVICE_LONG** In [$CONSOLE][services-portal] open an [SQL editor][in-console-editors]. You can also connect to your service using [psql][connect-using-psql]. @@ -16,8 +18,17 @@ ```sql ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); ``` - Before you say `huh`, a continuous aggregate is a specialized hypertable. - + Before you say `huh`, a continuous aggregate is a specialized hypertable. + + * Enable indexing over all data in the rowstore and columnstore: + + ```sql + alter table stocks_real_time, + set access method hypercore, + set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol'); + ``` + This is also early access for continuous aggregates. + 1. **Add a policy to move chunks to the columnstore at a specific time interval** For example, 60 days after the data was added to the table: @@ -74,7 +85,7 @@ [job]: /api/:currentVersion:/actions/add_job/ [alter_table_hypercore]: /api/:currentVersion:/hypercore/alter_table/ -[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ +[compression_continuous-aggregate]: /api/:currentVersion:/hypercore/alter_materialized_view/ [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md index 842f49d7f6..5fe61e2e45 100644 --- a/api/hypercore/add_columnstore_policy.md +++ b/api/hypercore/add_columnstore_policy.md @@ -104,7 +104,7 @@ Calls to `add_columnstore_policy` require either `after` or `created_before`, bu [compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/ -[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ +[compression_continuous-aggregate]: /api/:currentVersion:/hypercore/alter_materialized_view/ [set_integer_now_func]: /api/:currentVersion:/hypertable/set_integer_now_func [informational-views]: /api/:currentVersion:/informational-views/jobs/ [chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/ diff --git a/api/hypercore/alter_materialized_view.md b/api/hypercore/alter_materialized_view.md new file mode 100644 index 0000000000..27b518ec44 --- /dev/null +++ b/api/hypercore/alter_materialized_view.md @@ -0,0 +1,96 @@ +--- +api_name: ALTER MATERIALIZED VIEW (Hypercore) +excerpt: Change an existing continuous aggregate +topics: [hypercore, continuous aggregates, columnstore,] +keywords: [hypercore, continuous aggregates, columnstore, ] +tags: [materialized views, hypertables, alter, change] +api: + license: community + type: command +--- + +import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; +import EarlyAccess from "versionContent/_partials/_early_access.mdx"; + +# ALTER MATERIALIZED VIEW (Hypercore) Community + +`ALTER MATERIALIZED VIEW` statement can be used to modify some of the `WITH` +clause [options][create_materialized_view] for the continuous aggregate view. +`ALTER MATERIALIZED VIEW` statement also supports the following +[PostgreSQL clauses][postgres-alterview] on the +continuous aggregate view: + +* `RENAME TO` clause to rename the continuous aggregate view +* `RENAME [COLUMN]` clause to rename the continuous aggregate column +* `SET SCHEMA` clause to set the new schema for the continuous aggregate view +* `SET TABLESPACE` clause to move the materialization of the continuous + aggregate view to the new tablespace +* `OWNER TO` clause to set new owner for the continuous aggregate view + + + +## Samples + +- Disable real-time aggregates for a continuous aggregate: + + ```sql + ALTER MATERIALIZED VIEW contagg_view SET (timescaledb.materialized_only); + ``` + +- Enable hypercore for a continuous aggregate: + + ```sql + ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ``` + +- Rename a column for a continuous aggregate: + + ```sql + ALTER MATERIALIZED VIEW contagg_view RENAME COLUMN old_name TO new_name; + ``` + +- **Enable indexing on data in the columnstore** + + ```sql + ALTER MATERIALIZED VIEW stock_candlestick_daily + set access method hypercore, + set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ``` + + +The only options that currently can be modified with `ALTER +MATERIALIZED VIEW` are `materialized_only` and `compress`. The other options +`continuous` and `create_group_indexes` can only be set when creating +the continuous aggregate. + + +## Arguments + +The syntax is: + + +``` sql +ALTER MATERIALIZED VIEW SET (timescaledb.enable_columnstore, + timescaledb.materialized_only = 'true' | 'false', + timescaledb.orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', + timescaledb.segmentby = ' [, ...]', + timescaledb.compress_chunk_time_interval='interval', + SET ACCESS METHOD { new_access_method | DEFAULT } +); +``` + +| Name | Type | Default | Required | Description | +|------------------------------------------------|--|------------------------------------------------------|--|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `view_name` |TEXT| - | ✖ | The materialized view to enable columstore for. | +| `timescaledb.enable_columnstore` |BOOLEAN| `true` | ✖ | Enable columnstore. | +|`timescaledb.materialized_only`|BOOLEAN| `true` | ✖ | Enable and disable real time aggregation | +| `timescaledb.orderby` |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | +| `timescaledb.segmentby` |TEXT| No segementation by column. | ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | +| `column_name` |TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | +| `timescaledb.compress_chunk_time_interval` |TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | +|`SET ACCESS METHOD`|TEXT| DEFAULT ([heap][default_table_access_method])| ✖| To enable indexing on the columstore, set to `hypercore` after you [create a continuous aggregate][create-cagg]. | + +[create_materialized_view]: /api/:currentVersion:/continuous-aggregates/create_materialized_view/#parameters +[postgres-alterview]: https://www.postgresql.org/docs/current/sql-alterview.html +[create-cagg]: /use-timescale/:currentVersion:/continuous-aggregates/create-a-continuous-aggregate/ +[default_table_access_method]: https://www.postgresql.org/docs/17/runtime-config-client.html#GUC-DEFAULT-TABLE-ACCESS-METHOD diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index dd42cd4006..cab9035d35 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -10,6 +10,7 @@ api: products: [cloud, self_hosted] --- import Since2180 from "versionContent/_partials/_since_2_18_0.mdx"; +import EarlyAccess from "versionContent/_partials/_early_access.mdx"; # ALTER TABLE (Hypercore) @@ -49,6 +50,16 @@ To enable the columnstore: ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '0'); ``` +- **Enable indexing on data in the columnstore** + + ```sql + alter table metrics + set access method hypercore, + set (timescaledb.compress_orderby = 'created_at', + timescaledb.compress_segmentby = 'location_id'); + ``` + + ## Arguments The syntax is: @@ -57,25 +68,28 @@ The syntax is: ALTER TABLE SET (timescaledb.enable_columnstore, timescaledb.orderby = ' [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]', timescaledb.segmentby = ' [, ...]', - timescaledb.compress_chunk_time_interval='interval' - timescaledb.enable_segmentwise_recompression = 'ON' | 'OFF' + timescaledb.compress_chunk_time_interval='interval', + timescaledb.enable_segmentwise_recompression = 'ON' | 'OFF', + SET ACCESS METHOD { new_access_method | DEFAULT } ); ``` | Name | Type | Default | Required | Description | -|--|--|------------------------------------------------------|--|--| +|--|--|------------------------------------------------------|--|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`table_name`|TEXT| - | ✖ | The hypertable to enable columstore for. | |`timescaledb.enable_columnstore`|BOOLEAN| `true` | ✖ | Enable columnstore. | |`timescaledb.orderby`|TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. | |`timescaledb.segmentby`|TEXT| No segementation by column. | ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate. | -|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | +|`column_name`|TEXT| - | ✖ | The name of the column to `orderby` or `segmentby`. | |`timescaledb.compress_chunk_time_interval`|TEXT| - | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. | -|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`.| -|`timescaledb.enable_segmentwise_recompression`|TEXT| ON | ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. | - +|`interval`|TEXT| - | ✖ | Set to a multiple of the [chunk_time_interval][chunk_time_interval] for `table`. | +|`timescaledb.enable_segmentwise_recompression`|TEXT| ON | ✖| Set to `OFF` to disable segmentwise recompression on chunks in the columnstore. This can be beneficial for some user workloads where segmentwise recompression is slow, and full recompression is more performant. | +|`SET ACCESS METHOD`|TEXT| DEFAULT ([heap][default_table_access_method])| ✖| To enable indexing on the columstore, set to `hypercore` after you [create a hypertable][create-hypertable]. | [chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/ [add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ [job]: /api/:currentVersion:/actions/add_job/ +[default_table_access_method]: https://www.postgresql.org/docs/17/runtime-config-client.html#GUC-DEFAULT-TABLE-ACCESS-METHOD +[create-hypertable]: /api/:currentVersion:/hypertable/create_hypertable diff --git a/api/hypercore/index.md b/api/hypercore/index.md index cec81e7a1f..c1062b4772 100644 --- a/api/hypercore/index.md +++ b/api/hypercore/index.md @@ -81,7 +81,7 @@ Chunks in the columnstore have the following limitations: [alter_table_hypercore]: /api/:currentVersion:/hypercore/alter_table/ -[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/ +[compression_continuous-aggregate]: /api/:currentVersion:/hypercore/alter_materialized_view/ [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/ diff --git a/api/page-index/page-index.js b/api/page-index/page-index.js index 9d4f5a6c18..babc8912d7 100644 --- a/api/page-index/page-index.js +++ b/api/page-index/page-index.js @@ -121,6 +121,11 @@ module.exports = [ excerpt: "Seamlessly switch between fast row-oriented storage and efficient column-oriented storage", href: "hypercore", children: [ + { + title: "ALTER MATERIALIZED VIEW", + href: "alter_materialized_view", + excerpt: "Enable the columnstore for a materialized view.", + }, { title: "ALTER TABLE", href: "alter_table", diff --git a/self-hosted/configuration/timescaledb-config.md b/self-hosted/configuration/timescaledb-config.md index 9fbc6d68c0..5ea1e71e26 100644 --- a/self-hosted/configuration/timescaledb-config.md +++ b/self-hosted/configuration/timescaledb-config.md @@ -7,6 +7,7 @@ tags: [tune] --- import MultiNodeDeprecation from "versionContent/_partials/_multi-node-deprecation.mdx"; +import ConfigCloudSelf from "versionContent/_partials/_cloud_self_configuration.mdx"; # TimescaleDB configuration and tuning @@ -15,14 +16,6 @@ settings that may be useful to your specific installation and performance needs. also be set within the `postgresql.conf` file or as command-line parameters when starting PostgreSQL. -## Policies - -### `timescaledb.max_background_workers (int)` - -Max background worker processes allocated to TimescaleDB. Set to at least 1 + -the number of databases loaded with a TimescaleDB extension in a PostgreSQL -instance. Default value is 16. - ## Query Planning and Execution ### `timescaledb.enable_chunkwise_aggregation (bool)` @@ -31,7 +24,6 @@ planning. The first part of the aggregation is executed on a per-chunk basis. Then, these partial results are combined and finalized. Splitting aggregations decreases the size of the created hash tables and increases data locality, which speeds up queries. - ### `timescaledb.vectorized_aggregation (bool)` Enables or disables the vectorized optimizations in the query executor. For @@ -44,6 +36,16 @@ Set to `ON` to dramatically decrease the amount of data written on a continuous in the presence of a small number of changes, reduce the i/o cost of refreshing a [continuous aggregate][continuous-aggregates], and generate fewer Write-Ahead Logs (WAL). Only works for continuous aggregates that don't have compression enabled. +## Policies + +### `timescaledb.max_background_workers (int)` + +Max background worker processes allocated to TimescaleDB. Set to at least 1 + +the number of databases loaded with a TimescaleDB extension in a PostgreSQL +instance. Default value is 16. + + + ## Hypercore features ### `timescaledb.default_hypercore_use_access_method (bool)` diff --git a/use-timescale/configuration/advanced-parameters.md b/use-timescale/configuration/advanced-parameters.md index 143436ce81..baf970519a 100644 --- a/use-timescale/configuration/advanced-parameters.md +++ b/use-timescale/configuration/advanced-parameters.md @@ -9,6 +9,8 @@ cloud_ui: - [services, :serviceId, operations, database_parameters] --- +import ConfigCloudSelf from "versionContent/_partials/_cloud_self_configuration.mdx"; + # Advanced parameters It is possible to configure a wide variety of service database parameters by @@ -23,9 +25,9 @@ prompting you to confirm changes before the service is modified. ## Multiple databases -To create more than one Timescale database, you need to create a new -service for each database. Timescale does not support multiple -databases within the same service. Having a separate service for each database +To create more than one database, you need to create a new +$SERVICE_SHORT for each database. $CLOUD_LONG does not support multiple +databases within the same service. Having a separate $SERVICE_SHORT for each database affords each database its own isolated resources. You can also use [schemas][schemas] to organize tables into logical groups. A @@ -33,6 +35,9 @@ single database can contain multiple schemas, which in turn contain tables. The main difference between isolating with databases versus schemas is that a user can access objects in any of the schemas in the database they are connected to, so long as they have the corresponding privileges. Schemas can help isolate -smaller use cases that do not warrant their own service. +smaller use cases that do not warrant their own $SERVICE_SHORT. + + + [schemas]: /use-timescale/:currentVersion:/schema-management/ diff --git a/use-timescale/hypercore/automate-hypercore-with-policies.md b/use-timescale/hypercore/automate-hypercore-with-policies.md deleted file mode 100644 index 9396cc09f1..0000000000 --- a/use-timescale/hypercore/automate-hypercore-with-policies.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -title: Manage data with Hypercore -excerpt: Reduce your chunk size by more than 90% and speed up your queries by automatically converting data between the rowstore and columnstore. -products: [cloud,] -keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore, hyperstore] ---- - -import PrereqCloud from "versionContent/_partials/_prereqs-cloud-only.mdx"; -import HCConversionOverview from "versionContent/_partials/_hypercore-conversion-overview.mdx"; -import HCPolicyWorkflow from "versionContent/_partials/_hypercore_policy_workflow.mdx"; -import UsageBasedStorage from "versionContent/_partials/_usage-based-storage-intro.mdx"; - - -# Manage data with Hypercore - -Hypercore is the Timescale hybrid row-columnar storage engine. The rowstore contains row-oriented tables optimized - for high-speed inserts and updates. The columnstore is a column-oriented storage format optimized for analytics. -You ingest `hot` data into the rowstore. As data cools and becomes more suited for analytics, -$CLOUD_LONG automatically converts these chunks of data to the columnstore. You define the moment when data is converted -using a columnstore policy. - - - -This page shows you how get the best results when you set a policy to automatically move chunks in a hypertable to the -columnstore. - -## Prerequisites - - - -This page uses the [real-time-stock-data][ingest-data] sample data in the samples. - -## Manage your data with columnstore policies - -The compression ratio and query performance of data in the columnstore is dependent on the order and structure of your -data. Rows that change over a dimension should be close to each other. With time-series data, you `orderby` the time -dimension. For example, `Timestamp`: - -| Timestamp | Device ID | Device Type | CPU |Disk IO| -|---|---|---|---|---| -|12:00:01|A|SSD|70.11|13.4| - -This ensures that records are compressed and accessed in the same order. However, you would always have to -access the data using the time dimension, then filter all the rows using other criteria. To make your queries more -efficient, you segment your data based on the way you want to access it. For example, to rapidly access data about a -single device, you `segmentby` the `Device ID` column. This enables you to run much faster analytical queries on -data in the columnstore. - -When $CLOUD_LONG converts a chunk to the columnstore, TimescaleDB automatically creates a different schema for your -data. $TIMESCALE_DB creates and uses custom indexes to incorporate the `segmentby` and `orderby` parameters when -you write to and read from the columstore. - -To setup your Hypercore automation: - - - - - -## Reference - -For integers, timestamps, and other integer-like types, data is compressed using [delta encoding][delta], -[delta-of-delta][delta-delta], [simple-8b][simple-8b], and [run-length encoding][run-length]. For columns with few -repeated values,[XOR-based][xor] and [dictionary compression][dictionary] is used. For all other types, -[dictionary compression][dictionary] is used. - - - -[create-hypertable]: /use-timescale/:currentVersion:/compression/ -[add_columnstore_policy]: /api/:currentVersion:/hypercore/add_columnstore_policy/ -[delta]: /use-timescale/:currentVersion:/compression/compression-methods/#delta-encoding -[delta-delta]: /use-timescale/:currentVersion:/compression/compression-methods/#delta-of-delta-encoding -[simple-8b]: /use-timescale/:currentVersion:/compression/compression-methods/#simple-8b -[run-length]: /use-timescale/:currentVersion:/compression/compression-methods/#run-length-encoding -[xor]: /use-timescale/:currentVersion:/compression/compression-methods/#xor-based-encoding -[dictionary]: /use-timescale/:currentVersion:/compression/compression-methods/#dictionary-compression -[ingest-data]: /getting-started/:currentVersion:/time-series-data/#ingest-the-dataset diff --git a/use-timescale/hypercore/optimize-data-for-real-time-analytics.md b/use-timescale/hypercore/optimize-data-for-real-time-analytics.md index b2c558c0ea..573ec3f936 100644 --- a/use-timescale/hypercore/optimize-data-for-real-time-analytics.md +++ b/use-timescale/hypercore/optimize-data-for-real-time-analytics.md @@ -9,7 +9,7 @@ import PrereqCloud from "versionContent/_partials/_prereqs-cloud-only.mdx"; import HCConversionOverview from "versionContent/_partials/_hypercore-conversion-overview.mdx"; import HCPolicyWorkflow from "versionContent/_partials/_hypercore_policy_workflow.mdx"; import UsageBasedStorage from "versionContent/_partials/_usage-based-storage-intro.mdx"; - +import EarlyAccess from "versionContent/_partials/_early_access.mdx"; # Optimize data for real-time analytics with Hypercore @@ -23,6 +23,10 @@ using a columnstore policy. + Indexes are a central component to designing efficient and performant databases: they allow +unstructured data to be searched in efficiently executed queries. TimescaleDB supplies the hypercore table access +method that enables chunks in a hypertable to maintain indexes over data in the rowstore and the columnstore. + This page shows you how get the best results when you set a policy to automatically move chunks in a hypertable to the columnstore. From df9f20bb71a0a3ebd5fc228262000569c5f06345 Mon Sep 17 00:00:00 2001 From: Iain Date: Tue, 28 Jan 2025 13:49:39 +0100 Subject: [PATCH 2/5] chore: include indexing in optimize and modify docs. --- _partials/_early_access.md | 1 - _partials/_hypercore_manual_workflow.md | 13 ++++++++++++ _partials/_hypercore_policy_workflow.md | 21 ++++++++++++++++--- api/hypercore/alter_materialized_view.md | 10 +++++++-- api/hypercore/alter_table.md | 13 +++++++++--- ...est-practice-for-large-amounts-of-data.md} | 3 ++- use-timescale/hypercore/index.md | 13 +++++++----- .../modify-data-in-the-columnstore.md | 2 +- use-timescale/page-index/page-index.js | 4 ++-- 9 files changed, 62 insertions(+), 18 deletions(-) rename use-timescale/hypercore/{indexing-data-in-hypercore.md => best-practice-for-large-amounts-of-data.md} (99%) diff --git a/_partials/_early_access.md b/_partials/_early_access.md index 2062b00a2d..89ad944fba 100644 --- a/_partials/_early_access.md +++ b/_partials/_early_access.md @@ -1,2 +1 @@ Early access: TimescaleDB v2.18.0 - diff --git a/_partials/_hypercore_manual_workflow.md b/_partials/_hypercore_manual_workflow.md index 2210f435aa..4f6fc85d0a 100644 --- a/_partials/_hypercore_manual_workflow.md +++ b/_partials/_hypercore_manual_workflow.md @@ -30,6 +30,17 @@ CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); ``` + To enable indexing over the specific chunk you are adding to the + columnstore, use Hypercore TAM to compress chunks using `hypercore_use_access_method`. + + ``` sql + + call convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk', + hypercore_use_access_method => true); + ``` + You must [enable columnstore on a hypertable][setup-hypercore] before you apply `hypercore_use_access_method` + to a chunk. You can also do this using [ALTER TABLE][compression_alter-table]. + 1. **Restart the jobs that are automatically converting chunks to the columnstore** ``` sql @@ -39,3 +50,5 @@ [alter_job]: /api/:currentVersion:/actions/alter_job/ [informational-views]: /api/:currentVersion:/informational-views/jobs/ [insert]: /use-timescale/:currentVersion:/write-data/insert/ +[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/optimize-data-for-real-time-analytics/ +[compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/ diff --git a/_partials/_hypercore_policy_workflow.md b/_partials/_hypercore_policy_workflow.md index a57f3ebfb3..fab292f5f8 100644 --- a/_partials/_hypercore_policy_workflow.md +++ b/_partials/_hypercore_policy_workflow.md @@ -12,11 +12,15 @@ import EarlyAccess from "versionContent/_partials/_early_access.mdx"; * [Use `ALTER TABLE` for a hypertable][alter_table_hypercore] ```sql - ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol'); + ALTER TABLE stocks_real_time SET ( + timescaledb.enable_columnstore = true, + timescaledb.segmentby = 'symbol'); ``` * [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate] ```sql - ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ALTER MATERIALIZED VIEW stock_candlestick_daily set ( + timescaledb.enable_columnstore = true, + timescaledb.segmentby = 'symbol' ); ``` Before you say `huh`, a continuous aggregate is a specialized hypertable. @@ -37,6 +41,16 @@ import EarlyAccess from "versionContent/_partials/_early_access.mdx"; ``` See [add_columnstore_policy][add_columnstore_policy]. + * To enable indexing over data in the rowstore and the columnstore, tell the policy + to use the Hypercore table access method. + + ``` sql + CALL add_columnstore_policy( + 'older_stock_prices', + after => INTERVAL '60d', + hypercore_use_access_method => true); + ``` + 1. **View the policies that you set or the policies that already exist** ``` sql @@ -52,7 +66,8 @@ import EarlyAccess from "versionContent/_partials/_early_access.mdx"; After the update, [convert the chunk to the columnstore][convert_to_columnstore] and restart the jobs. ``` sql - SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'stocks_real_time' + SELECT * FROM timescaledb_information.jobs where + proc_name = 'policy_compression' AND relname = 'stocks_real_time' -- Select the JOB_ID from the results diff --git a/api/hypercore/alter_materialized_view.md b/api/hypercore/alter_materialized_view.md index 27b518ec44..ad9adca642 100644 --- a/api/hypercore/alter_materialized_view.md +++ b/api/hypercore/alter_materialized_view.md @@ -49,14 +49,20 @@ continuous aggregate view: ALTER MATERIALIZED VIEW contagg_view RENAME COLUMN old_name TO new_name; ``` -- **Enable indexing on data in the columnstore** +- **Enable indexing on data in the columnstore** ```sql ALTER MATERIALIZED VIEW stock_candlestick_daily set access method hypercore, set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' ); + ``` + +- **Enable indexing on a chunk you are adding to the columnstore** + + ```sql + ALTER MATERIALIZED VIEW _timescaledb_internal._hyper_1_21_chunk + set access method hypercore; ``` - The only options that currently can be modified with `ALTER MATERIALIZED VIEW` are `materialized_only` and `compress`. The other options diff --git a/api/hypercore/alter_table.md b/api/hypercore/alter_table.md index cab9035d35..1129aef56c 100644 --- a/api/hypercore/alter_table.md +++ b/api/hypercore/alter_table.md @@ -50,15 +50,22 @@ To enable the columnstore: ALTER TABLE metrics SET (timescaledb.compress_chunk_time_interval = '0'); ``` -- **Enable indexing on data in the columnstore** - +- **Enable indexing on all data you add to the the columnstore** + ```sql alter table metrics set access method hypercore, set (timescaledb.compress_orderby = 'created_at', timescaledb.compress_segmentby = 'location_id'); ``` - + +- **Enable indexing on a chunk you are adding to the columnstore** + + ```sql + alter table _timescaledb_internal._hyper_1_21_chunk + set access method hypercore; + ``` + ## Arguments diff --git a/use-timescale/hypercore/indexing-data-in-hypercore.md b/use-timescale/hypercore/best-practice-for-large-amounts-of-data.md similarity index 99% rename from use-timescale/hypercore/indexing-data-in-hypercore.md rename to use-timescale/hypercore/best-practice-for-large-amounts-of-data.md index 0ec0d8d84d..cdc388b972 100644 --- a/use-timescale/hypercore/indexing-data-in-hypercore.md +++ b/use-timescale/hypercore/best-practice-for-large-amounts-of-data.md @@ -8,7 +8,8 @@ keywords: [hypertable, compression, row-columnar storage, hypercore, hyperstore] import Prereq from "versionContent/_partials/_prereqs-cloud-and-self.mdx"; import HypercoreManualWorkflow from "versionContent/_partials/_hypercore_manual_workflow.mdx"; -# Indexing data in Hypercore +# SUGGESTION: Best practice for large amounts of data + Indexes are a central component to designing efficient and performant databases: they allow unstructured data to be efficiently searched to diff --git a/use-timescale/hypercore/index.md b/use-timescale/hypercore/index.md index 48b3c348e6..c484137662 100644 --- a/use-timescale/hypercore/index.md +++ b/use-timescale/hypercore/index.md @@ -9,10 +9,10 @@ import UsageBasedStorage from "versionContent/_partials/_usage-based-storage-int # Hypercore -Hypercore is the Timescale hybrid row-columnar storage engine, designed specifically for +Hypercore is the $COMPANY hybrid row-columnar storage engine, designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables -Timescale Cloud to deliver the best of both worlds, solving the key challenges in real-time analytics: +$CLOUD_LONG to deliver the best of both worlds, solving the key challenges in real-time analytics: - High ingest throughput - Low-latency ingestion @@ -21,7 +21,7 @@ Timescale Cloud to deliver the best of both worlds, solving the key challenges i - Streamlined data management Hypercore’s hybrid approach combines the benefits of row-oriented and column-oriented formats -in each Timescale Cloud service: +in each $SERVICE_LONG: - **Fast ingest with rowstore**: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle @@ -38,6 +38,9 @@ in each Timescale Cloud service: to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed. +- **Indexing row and columnar data**: the hypercore table access method enables chunks in a + hypertable to maintain indexes over data in the rowstore and the columnstore. Early access: TimescaleDB v2.18.0 + ![Hypercore workflow](https://assets.timescale.com/docs/images/hypercore-overview.png) @@ -46,9 +49,9 @@ This section shows you how to: * [Optimize data for real-time analytics with Hypercore][setup-hypercore] * [Modify data in the columnstore][modify-data-in-the-columnstore] -* [Indexing data in Hypercore][indexing-data-in-hypercore] +* [Best practice for large amounts of data][indexing-data-in-hypercore] [setup-hypercore]: /use-timescale/:currentVersion:/hypercore/optimize-data-for-real-time-analytics/ [modify-data-in-the-columnstore]: /use-timescale/:currentVersion:/hypercore/modify-data-in-the-columnstore/ -[indexing-data-in-hypercore]: /use-timescale/:currentVersion:/hypercore/indexing-data-in-hypercore/ +[indexing-data-in-hypercore]: /use-timescale/:currentVersion:/hypercore/best-practice-for-large-amounts-of-data/ [compression]: /use-timescale/:currentVersion:/compression/ diff --git a/use-timescale/hypercore/modify-data-in-the-columnstore.md b/use-timescale/hypercore/modify-data-in-the-columnstore.md index e994ca4f6c..9dfdb01491 100644 --- a/use-timescale/hypercore/modify-data-in-the-columnstore.md +++ b/use-timescale/hypercore/modify-data-in-the-columnstore.md @@ -21,7 +21,7 @@ This page shows you how to update small and large amounts of new data, and updat -- [Optimize cooler data in hypercore][setup-hypercore] +- [Optimize data for real-time analytics with Hypercore][setup-hypercore] ## Modify small amounts of data diff --git a/use-timescale/page-index/page-index.js b/use-timescale/page-index/page-index.js index f05d75556e..ded75063f5 100644 --- a/use-timescale/page-index/page-index.js +++ b/use-timescale/page-index/page-index.js @@ -209,8 +209,8 @@ module.exports = [ excerpt: "Add and remove data in the columnstore", }, { - title: "Indexing data in columnstore", - href: "indexing-data-in-hypercore", + title: "Best practice for large amounts of data", + href: "best-practice-for-large-amounts-of-data", excerpt: "Indexing data in columnstore", }, ], From 7fa19bfbf16fc85a9dbce23b066472f9051a9ec6 Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 29 Jan 2025 12:43:46 +0100 Subject: [PATCH 3/5] chore: update doc names. --- _partials/_hypercore_manual_workflow.md | 20 ++--- _partials/_usage-based-storage-intro.md | 2 +- ...best-practice-for-large-amounts-of-data.md | 2 +- .../efficient-data-access-in-hypercore.md | 75 +++++++++++++++++++ use-timescale/hypercore/index.md | 18 ++--- ...mnstore.md => modify-data-in-hypercore.md} | 18 +++-- ...md => real-time-analytics-in-hypercore.md} | 8 +- use-timescale/page-index/page-index.js | 16 ++-- 8 files changed, 118 insertions(+), 41 deletions(-) create mode 100644 use-timescale/hypercore/efficient-data-access-in-hypercore.md rename use-timescale/hypercore/{modify-data-in-the-columnstore.md => modify-data-in-hypercore.md} (87%) rename use-timescale/hypercore/{optimize-data-for-real-time-analytics.md => real-time-analytics-in-hypercore.md} (94%) diff --git a/_partials/_hypercore_manual_workflow.md b/_partials/_hypercore_manual_workflow.md index 4f6fc85d0a..0a7fa9c7d4 100644 --- a/_partials/_hypercore_manual_workflow.md +++ b/_partials/_hypercore_manual_workflow.md @@ -1,3 +1,4 @@ +import EarlyAccess from "versionContent/_partials/_early_access.mdx"; 1. **Stop the jobs that are automatically adding chunks to the columnstore** @@ -30,16 +31,15 @@ CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk'); ``` - To enable indexing over the specific chunk you are adding to the - columnstore, use Hypercore TAM to compress chunks using `hypercore_use_access_method`. + * To enable indexing over the specific chunk you are adding to the + columnstore, enable the Hypercore table access method: - ``` sql - - call convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk', - hypercore_use_access_method => true); - ``` - You must [enable columnstore on a hypertable][setup-hypercore] before you apply `hypercore_use_access_method` - to a chunk. You can also do this using [ALTER TABLE][compression_alter-table]. + ``` sql + call convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk', + hypercore_use_access_method => true); + ``` + You must [enable columnstore on a hypertable][setup-hypercore] before you apply `hypercore_use_access_method` + to a chunk. You can also do this using [ALTER TABLE][compression_alter-table]. 1. **Restart the jobs that are automatically converting chunks to the columnstore** @@ -50,5 +50,5 @@ [alter_job]: /api/:currentVersion:/actions/alter_job/ [informational-views]: /api/:currentVersion:/informational-views/jobs/ [insert]: /use-timescale/:currentVersion:/write-data/insert/ -[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/optimize-data-for-real-time-analytics/ +[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/real-time-analytics-in-hypercore/ [compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/ diff --git a/_partials/_usage-based-storage-intro.md b/_partials/_usage-based-storage-intro.md index 9dda0efd83..0054c6fcc0 100644 --- a/_partials/_usage-based-storage-intro.md +++ b/_partials/_usage-based-storage-intro.md @@ -1,7 +1,7 @@ $CLOUD_LONG charges are based on the amount of storage you use. You don't pay for fixed storage size, and you don't need to worry about scaling disk size as your data grows; We handle it all for you. To reduce your data costs further, -use [hypercore][hypercore], a [data retention policy][data-retention], and +use [Hypercore][hypercore], a [data retention policy][data-retention], and [tiered storage][data-tiering]. [hypercore]: /api/:currentVersion:/hypercore/ diff --git a/use-timescale/hypercore/best-practice-for-large-amounts-of-data.md b/use-timescale/hypercore/best-practice-for-large-amounts-of-data.md index cdc388b972..006cbff536 100644 --- a/use-timescale/hypercore/best-practice-for-large-amounts-of-data.md +++ b/use-timescale/hypercore/best-practice-for-large-amounts-of-data.md @@ -2,7 +2,7 @@ title: Indexing data in Hypercore excerpt: Use indexes on hybrid row-columnar data products: [cloud,] -keywords: [hypertable, compression, row-columnar storage, hypercore, hyperstore] +keywords: [hypertable, compression, row-columnar storage, hypercore] --- import Prereq from "versionContent/_partials/_prereqs-cloud-and-self.mdx"; diff --git a/use-timescale/hypercore/efficient-data-access-in-hypercore.md b/use-timescale/hypercore/efficient-data-access-in-hypercore.md new file mode 100644 index 0000000000..7c9d67db1a --- /dev/null +++ b/use-timescale/hypercore/efficient-data-access-in-hypercore.md @@ -0,0 +1,75 @@ +--- +title: Efficient data access in Hypercore +excerpt: Read and write data efficiently in Hypercore. +products: [cloud,self_hosted] +keywords: [hypertable, compression, row-columnar storage, hypercore] +--- +import Prereq from "versionContent/_partials/_prereqs-cloud-and-self.mdx"; + +# Access your data efficiently in Hypercore + + +Intro + + + +This page shows you how to ... . + +## Prerequisites + + + +- [Prepare your data for real-time analytics in Hypercore][setup-hypercore] + + +## Read data efficiently + +Explain why and how to read data efficiently. + + + +1. **Do this** + + Why and how to do this. + + ``` sql + Code example + ``` + +1. **Do that** + + Why and how to do that. + + ``` sql + Code example + ``` + + + +## Write data efficiently + + +Explain why and how to writer data efficiently. + + + +1. **Do this** + + Why and how to do this. + + ``` sql + Code example + ``` + +1. **Do that** + + Why and how to do that. + + ``` sql + Code example + ``` + + + + +[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/real-time-analytics-in-hypercore/ diff --git a/use-timescale/hypercore/index.md b/use-timescale/hypercore/index.md index c484137662..930534c969 100644 --- a/use-timescale/hypercore/index.md +++ b/use-timescale/hypercore/index.md @@ -2,7 +2,7 @@ title: Hypercore excerpt: The Timescale hybrid row-columnar storage engine for real-time analytics, powered by time-series data products: [cloud,] -keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore, hyperstore] +keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore] --- import UsageBasedStorage from "versionContent/_partials/_usage-based-storage-intro.mdx"; @@ -10,7 +10,7 @@ import UsageBasedStorage from "versionContent/_partials/_usage-based-storage-int # Hypercore Hypercore is the $COMPANY hybrid row-columnar storage engine, designed specifically for -real-time analytics and powered by time-series data. The advantage of hypercore is its ability +real-time analytics and powered by time-series data. The advantage of Hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage. This flexibility enables $CLOUD_LONG to deliver the best of both worlds, solving the key challenges in real-time analytics: @@ -34,11 +34,11 @@ in each $SERVICE_LONG: saving significant storage space. - **Full mutability with transactional semantics**: regardless of where data is stored, - hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates + Hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed. -- **Indexing row and columnar data**: the hypercore table access method enables chunks in a +- **Indexing row and columnar data**: the `hypercore` table access method enables chunks in a hypertable to maintain indexes over data in the rowstore and the columnstore. Early access: TimescaleDB v2.18.0 ![Hypercore workflow](https://assets.timescale.com/docs/images/hypercore-overview.png) @@ -47,11 +47,11 @@ in each $SERVICE_LONG: This section shows you how to: -* [Optimize data for real-time analytics with Hypercore][setup-hypercore] -* [Modify data in the columnstore][modify-data-in-the-columnstore] -* [Best practice for large amounts of data][indexing-data-in-hypercore] +* [Prepare your data for real-time analytics in Hypercore][setup-hypercore] +* [Efficient data access in Hypercore][indexing-data-in-hypercore] +* [Modify data in the columnstore][modify-data-in-hypercore] -[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/optimize-data-for-real-time-analytics/ -[modify-data-in-the-columnstore]: /use-timescale/:currentVersion:/hypercore/modify-data-in-the-columnstore/ +[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/real-time-analytics-in-hypercore/ +[modify-data-in-hypercore]: /use-timescale/:currentVersion:/hypercore/modify-data-in-hypercore/ [indexing-data-in-hypercore]: /use-timescale/:currentVersion:/hypercore/best-practice-for-large-amounts-of-data/ [compression]: /use-timescale/:currentVersion:/compression/ diff --git a/use-timescale/hypercore/modify-data-in-the-columnstore.md b/use-timescale/hypercore/modify-data-in-hypercore.md similarity index 87% rename from use-timescale/hypercore/modify-data-in-the-columnstore.md rename to use-timescale/hypercore/modify-data-in-hypercore.md index 9dfdb01491..bda80b97e3 100644 --- a/use-timescale/hypercore/modify-data-in-the-columnstore.md +++ b/use-timescale/hypercore/modify-data-in-hypercore.md @@ -1,16 +1,16 @@ --- -title: Modify data in the columnstore +title: Modify data in Hypercore excerpt: Update data and the table schema in the columnstore products: [cloud,] -keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore, hyperstore] +keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore] --- import Prereq from "versionContent/_partials/_prereqs-cloud-and-self.mdx"; import HypercoreManualWorkflow from "versionContent/_partials/_hypercore_manual_workflow.mdx"; -# Modify data in the columnstore +# Modify your data in Hypercore -You [setup Hypercore][setup-hypercore] to automatically convert data between the rowstore and columnstore +You [setup Hypercore][setup-h[modify-data-in-hypercore.md](modify-data-in-hypercore.md)ypercore] to automatically convert data between the rowstore and columnstore when it reaches a certain age. After you have optimized data in the columnstore, you may need to modify it. For example, to make small changes, or backfill large amounts of data. You may even have to update the schema to accommodate this changes to the data. @@ -21,7 +21,7 @@ This page shows you how to update small and large amounts of new data, and updat -- [Optimize data for real-time analytics with Hypercore][setup-hypercore] +- [Prepare your data for real-time analytics in Hypercore][setup-hypercore] ## Modify small amounts of data @@ -50,6 +50,8 @@ This workflow is especially useful if you need to backfill old data. You can add modify the schema of a table in the columnstore. To do this, you need to: + + 1. **Stop the jobs that are automatically adding chunks to the columnstore** Retrieve the list of jobs from the [timescaledb_information.jobs][informational-views] view @@ -96,13 +98,13 @@ You can add modify the schema of a table in the columnstore. To do this, you nee SELECT alter_job(JOB_ID, scheduled => true); ``` - + [write]: /use-timescale/:currentVersion:/write-data/ -[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/optimize-data-for-real-time-analytics/ +[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/real-time-analytics-in-hypercore/ [job]: /api/:currentVersion:/actions/ [alter_job]: /api/:currentVersion:/actions/alter_job/ [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/ [informational-views]: /api/:currentVersion:/informational-views/jobs/ [insert]: /use-timescale/:currentVersion:/write-data/insert/ -[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/optimize-data-for-real-time-analytics/ +[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/real-time-analytics-in-hypercore/ diff --git a/use-timescale/hypercore/optimize-data-for-real-time-analytics.md b/use-timescale/hypercore/real-time-analytics-in-hypercore.md similarity index 94% rename from use-timescale/hypercore/optimize-data-for-real-time-analytics.md rename to use-timescale/hypercore/real-time-analytics-in-hypercore.md index 573ec3f936..e6950225e3 100644 --- a/use-timescale/hypercore/optimize-data-for-real-time-analytics.md +++ b/use-timescale/hypercore/real-time-analytics-in-hypercore.md @@ -2,7 +2,7 @@ title: Optimize data for real-time analytics with Hypercore excerpt: Reduce your chunk size by more than 90% and speed up your queries by automatically converting data between the rowstore and columnstore. products: [cloud,] -keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore, hyperstore] +keywords: [hyperscore, hypertable, compression, row-columnar storage, hypercore] --- import PrereqCloud from "versionContent/_partials/_prereqs-cloud-only.mdx"; @@ -11,7 +11,7 @@ import HCPolicyWorkflow from "versionContent/_partials/_hypercore_policy_workflo import UsageBasedStorage from "versionContent/_partials/_usage-based-storage-intro.mdx"; import EarlyAccess from "versionContent/_partials/_early_access.mdx"; -# Optimize data for real-time analytics with Hypercore +# Prepare your data for real-time analytics in Hypercore Hypercore is the Timescale hybrid row-columnar storage engine. The rowstore contains row-oriented tables optimized for high-speed inserts and updates. The columnstore is a column-oriented storage format optimized for analytics. @@ -24,7 +24,7 @@ using a columnstore policy. Indexes are a central component to designing efficient and performant databases: they allow -unstructured data to be searched in efficiently executed queries. TimescaleDB supplies the hypercore table access +unstructured data to be searched in efficiently executed queries. TimescaleDB supplies the `hypercore` table access method that enables chunks in a hypertable to maintain indexes over data in the rowstore and the columnstore. This page shows you how get the best results when you set a policy to automatically move chunks in a hypertable to the @@ -84,4 +84,4 @@ repeated values,[XOR-based][xor] and [dictionary compression][dictionary] is use [alter_job]: /api/:currentVersion:/actions/alter_job/ [informational-views]: /api/:currentVersion:/informational-views/jobs/ [insert]: /use-timescale/:currentVersion:/write-data/insert/ -[modify-data-in-the-columnstore]: /use-timescale/:currentVersion:/hypercore/modify-data-in-the-columnstore/ +[modify-data-in-hypercore]: /use-timescale/:currentVersion:/hypercore/modify-data-in-hypercore/ diff --git a/use-timescale/page-index/page-index.js b/use-timescale/page-index/page-index.js index ded75063f5..69751fc241 100644 --- a/use-timescale/page-index/page-index.js +++ b/use-timescale/page-index/page-index.js @@ -199,19 +199,19 @@ module.exports = [ "Seamlessly switch between row-oriented and column-oriented storage", children: [ { - title: "Optimize data for real-time analytics", - href: "optimize-data-for-real-time-analytics", + title: "Prepare your data for real-time analytics", + href: "real-time-analytics-in-hypercore", excerpt: "Automate", }, { - title: "Modify data in the columnstore", - href: "modify-data-in-the-columnstore", - excerpt: "Add and remove data in the columnstore", + title: "Efficient data access in Hypercore", + href: "efficient-data-access-in-hypercore", + excerpt: "Read and write data efficiently in hypercore", }, { - title: "Best practice for large amounts of data", - href: "best-practice-for-large-amounts-of-data", - excerpt: "Indexing data in columnstore", + title: "Modify data in Hypercore", + href: "modify-data-in-hypercore", + excerpt: "Update data stored in the columnstore", }, ], }, From 73bd43faac43e8d6da7d326a32444dea9f260c71 Mon Sep 17 00:00:00 2001 From: Iain Date: Wed, 29 Jan 2025 12:47:46 +0100 Subject: [PATCH 4/5] chore: typo. --- use-timescale/hypercore/efficient-data-access-in-hypercore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-timescale/hypercore/efficient-data-access-in-hypercore.md b/use-timescale/hypercore/efficient-data-access-in-hypercore.md index 7c9d67db1a..308742d27a 100644 --- a/use-timescale/hypercore/efficient-data-access-in-hypercore.md +++ b/use-timescale/hypercore/efficient-data-access-in-hypercore.md @@ -49,7 +49,7 @@ Explain why and how to read data efficiently. ## Write data efficiently -Explain why and how to writer data efficiently. +Explain why and how to write data efficiently. From 16f354b8acf5c6b5e9533c27fd407283366860e4 Mon Sep 17 00:00:00 2001 From: Iain Date: Thu, 30 Jan 2025 15:02:14 +0100 Subject: [PATCH 5/5] chore: update on review. --- _partials/_cloud_self_configuration.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/_partials/_cloud_self_configuration.md b/_partials/_cloud_self_configuration.md index 5204daa9c6..baba9dce7d 100644 --- a/_partials/_cloud_self_configuration.md +++ b/_partials/_cloud_self_configuration.md @@ -71,8 +71,8 @@ Enable transparent decompression ### `timescaledb.restoring (bool)` -Install timescale in restoring mode - +Stop any background workers which could have been performing tasks. This is especially useful you +migrate data to your [$SERVICE_LONG][pg-dump-and-restore] or [self-hosted database][migrate-entire]. ### `timescaledb.max_cached_chunks_per_hypertable (int)` Maximum cached chunks @@ -85,3 +85,5 @@ Maximum open chunks per insert The max number of tuples that can be decompressed during an INSERT, UPDATE, or DELETE. [enabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/ +[pg-dump-and-restore]: /migrate/:currentVersion:/pg-dump-and-restore/ +[migrate-entire]: /self-hosted/:currentVersion:/migration/entire-database/