Hypercore add indexing to optimize and modify pages (#3773)

* chore: include indexing in optimize and modify docs.
timescale · Feb 3, 2025 · 15219b9 · 15219b9
1 parent f8241c1
commit 15219b9
Show file tree

Hide file tree

Showing 19 changed files with 412 additions and 144 deletions.
diff --git a/_partials/_cloud_self_configuration.md b/_partials/_cloud_self_configuration.md
@@ -0,0 +1,89 @@
+import EarlyAccess from "versionContent/_partials/_early_access.mdx";
+
+## Policies
+
+### `timescaledb.max_background_workers (int)`
+
+Max background worker processes allocated to TimescaleDB. Set to at least 1 +
+the number of databases loaded with a TimescaleDB extension in a PostgreSQL
+instance. Default value is 16.
+
+### `timescaledb.enable_tiered_reads (bool)`
+
+Enable [tiered reads][enabling-data-tiering] to that you query your data normally when it's distributed across different storage tiers.
+Your hypertable is spread across the tiers, so queries and `JOIN`s work and fetch the same data as usual.
+
+By default, tiered data is not accessed by queries. Querying tiered data may slow down query performance
+as the data is not stored locally on Timescale's high-performance storage tier.
+
+## Hypercore features
+
+### `timescaledb.default_hypercore_use_access_method (bool)`
+
+The default value for `hypercore_use_access_method` for functions that have this parameter. This function is in `user` context, meaning that any user can set it for the session. The default value is `false`.
+
+<EarlyAccess />
+
+## $SERVICE_LONG tuning
+
+### `timescaledb.disable_load (bool)`
+
+Disable the loading of the actual extension
+
+### `timescaledb.enable_cagg_reorder_groupby (bool)`
+Enable group by reordering
+
+### `timescaledb.enable_chunk_append (bool)`
+Enable chunk append node
+
+### `timescaledb.enable_constraint_aware_append (bool)`
+Enable constraint-aware append scans
+
+### `timescaledb.enable_constraint_exclusion (bool)`
+Enable constraint exclusion
+
+### `timescaledb.enable_job_execution_logging (bool)`
+Enable job execution logging
+
+### `timescaledb.enable_optimizations (bool)`
+Enable TimescaleDB query optimizations
+
+### `timescaledb.enable_ordered_append (bool)`
+Enable ordered append scans
+
+### `timescaledb.enable_parallel_chunk_append (bool)`
+Enable parallel chunk append node
+
+### `timescaledb.enable_runtime_exclusion (bool)`
+Enable runtime chunk exclusion
+
+### `timescaledb.enable_tiered_reads (bool)`
+
+Enable [tiered reads][enabling-data-tiering] to that you query your data normally when it's distributed across different storage tiers.
+Your hypertable is spread across the tiers, so queries and `JOIN`s work and fetch the same data as usual.
+
+By default, tiered data is not accessed by queries. Querying tiered data may slow down query performance
+as the data is not stored locally on Timescale's high-performance storage tier.
+
+
+### `timescaledb.enable_transparent_decompression (bool)`
+Enable transparent decompression
+
+
+### `timescaledb.restoring (bool)`
+Stop any background workers which could have been performing tasks. This is especially useful you 
+migrate data to your [$SERVICE_LONG][pg-dump-and-restore] or [self-hosted database][migrate-entire].
+
+### `timescaledb.max_cached_chunks_per_hypertable (int)`
+Maximum cached chunks
+
+### `timescaledb.max_open_chunks_per_insert (int)`
+Maximum open chunks per insert
+
+### `timescaledb.max_tuples_decompressed_per_dml_transaction (int)`
+
+The max number of tuples that can be decompressed during an INSERT, UPDATE, or DELETE.
+
+[enabling-data-tiering]: /use-timescale/:currentVersion:/data-tiering/enabling-data-tiering/
+[pg-dump-and-restore]: /migrate/:currentVersion:/pg-dump-and-restore/
+[migrate-entire]: /self-hosted/:currentVersion:/migration/entire-database/
diff --git a/_partials/_early_access.md b/_partials/_early_access.md
@@ -1,5 +1 @@
-<Highlight type="important">
-This feature is early access. Early access features might be subject to billing
-changes in the future. If you have feedback, reach out to your customer success
-manager, or [contact us](https://www.timescale.com/contact/).
-</Highlight>
+<Tag variant="hollow">Early access: TimescaleDB v2.18.0</Tag>
diff --git a/_partials/_hypercore_manual_workflow.md b/_partials/_hypercore_manual_workflow.md
@@ -1,3 +1,4 @@
+import EarlyAccess from "versionContent/_partials/_early_access.mdx";
 
 1. **Stop the jobs that are automatically adding chunks to the columnstore**
 
@@ -30,6 +31,16 @@
    CALL convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk');
    ```
 
+   * <EarlyAccess /> To enable indexing over the specific chunk you are adding to the
+   columnstore, enable the Hypercore table access method:
+
+      ``` sql
+      call convert_to_columnstore('_timescaledb_internal._hyper_1_2_chunk',
+         hypercore_use_access_method => true);
+      ```
+      You must [enable columnstore on a hypertable][setup-hypercore] before you apply `hypercore_use_access_method`
+      to a chunk. You can also do this using [ALTER TABLE][compression_alter-table].
+
 1. **Restart the jobs that are automatically converting chunks to the columnstore**
 
    ``` sql
@@ -39,3 +50,5 @@
 [alter_job]: /api/:currentVersion:/actions/alter_job/
 [informational-views]: /api/:currentVersion:/informational-views/jobs/
 [insert]: /use-timescale/:currentVersion:/write-data/insert/
+[setup-hypercore]: /use-timescale/:currentVersion:/hypercore/real-time-analytics-in-hypercore/
+[compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/
diff --git a/_partials/_hypercore_policy_workflow.md b/_partials/_hypercore_policy_workflow.md
@@ -1,3 +1,5 @@
+import EarlyAccess from "versionContent/_partials/_early_access.mdx";
+
 1. **Connect to your $SERVICE_LONG**
 
    In [$CONSOLE][services-portal] open an [SQL editor][in-console-editors]. You can also connect to your service using [psql][connect-using-psql].
@@ -10,14 +12,27 @@
 
    * [Use `ALTER TABLE` for a hypertable][alter_table_hypercore]
      ```sql
-     ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol');
+     ALTER TABLE stocks_real_time SET (
+        timescaledb.enable_columnstore = true, 
+        timescaledb.segmentby = 'symbol');
      ```
    * [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate]
      ```sql
-     ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' );
+     ALTER MATERIALIZED VIEW stock_candlestick_daily set (
+        timescaledb.enable_columnstore = true, 
+        timescaledb.segmentby = 'symbol' );
      ``` 
-     Before you say `huh`, a continuous aggregate is a specialized hypertable. 
-
+     Before you say `huh`, a continuous aggregate is a specialized hypertable.
+
+   * <EarlyAccess /> Enable indexing over all data in the rowstore and columnstore:
+
+     ```sql
+     alter table stocks_real_time,
+        set access method hypercore,
+        set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol');
+     ```
+     This is also early access for continuous aggregates.
+
 1. **Add a policy to move chunks to the columnstore at a specific time interval**
 
    For example, 60 days after the data was added to the table:
@@ -26,6 +41,16 @@
    ```
    See [add_columnstore_policy][add_columnstore_policy].
 
+   * <EarlyAccess /> To enable indexing over data in the rowstore and the columnstore, tell the policy 
+     to use the Hypercore table access method.
+
+      ``` sql
+      CALL add_columnstore_policy(
+         'older_stock_prices', 
+         after => INTERVAL '60d',  
+         hypercore_use_access_method => true);
+      ```
+
 1. **View the policies that you set or the policies that already exist**
 
    ``` sql
@@ -41,7 +66,8 @@
   After the update, [convert the chunk to the columnstore][convert_to_columnstore] and restart the jobs. 
 
    ``` sql
-   SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'stocks_real_time'
+   SELECT * FROM timescaledb_information.jobs where 
+      proc_name = 'policy_compression' AND relname = 'stocks_real_time'
    
    -- Select the JOB_ID from the results
      
@@ -74,7 +100,7 @@
 
 [job]: /api/:currentVersion:/actions/add_job/
 [alter_table_hypercore]: /api/:currentVersion:/hypercore/alter_table/
-[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/
+[compression_continuous-aggregate]: /api/:currentVersion:/hypercore/alter_materialized_view/
 [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/
 [convert_to_columnstore]: /api/:currentVersion:/hypercore/convert_to_columnstore/
 [convert_to_rowstore]: /api/:currentVersion:/hypercore/convert_to_rowstore/

diff --git a/_partials/_usage-based-storage-intro.md b/_partials/_usage-based-storage-intro.md
@@ -1,7 +1,7 @@
 $CLOUD_LONG charges are based on the amount of storage you use. You don't pay for 
 fixed storage size, and you don't need to worry about scaling disk size as your 
 data grows; We handle it all for you. To reduce your data costs further, 
-use [hypercore][hypercore], a [data retention policy][data-retention], and
+use [Hypercore][hypercore], a [data retention policy][data-retention], and
 [tiered storage][data-tiering].
 
 [hypercore]: /api/:currentVersion:/hypercore/

diff --git a/api/hypercore/add_columnstore_policy.md b/api/hypercore/add_columnstore_policy.md
@@ -104,7 +104,7 @@ Calls to `add_columnstore_policy` require either `after` or `created_before`, bu
 
 
 [compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/
-[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/
+[compression_continuous-aggregate]: /api/:currentVersion:/hypercore/alter_materialized_view/
 [set_integer_now_func]: /api/:currentVersion:/hypertable/set_integer_now_func
 [informational-views]: /api/:currentVersion:/informational-views/jobs/
 [chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/

diff --git a/api/hypercore/alter_materialized_view.md b/api/hypercore/alter_materialized_view.md
@@ -0,0 +1,102 @@
+---
+api_name: ALTER MATERIALIZED VIEW (Hypercore)
+excerpt: Change an existing continuous aggregate
+topics: [hypercore, continuous aggregates, columnstore,]
+keywords: [hypercore, continuous aggregates, columnstore, ]
+tags: [materialized views, hypertables, alter, change]
+api:
+  license: community
+  type: command
+---
+
+import Since2180 from "versionContent/_partials/_since_2_18_0.mdx";
+import EarlyAccess from "versionContent/_partials/_early_access.mdx";
+
+# ALTER MATERIALIZED VIEW (Hypercore) <Tag type="community">Community</Tag>
+
+`ALTER MATERIALIZED VIEW` statement can be used to modify some of the `WITH`
+clause [options][create_materialized_view] for the continuous aggregate view.
+`ALTER MATERIALIZED VIEW` statement also supports the following
+[PostgreSQL clauses][postgres-alterview] on the
+continuous aggregate view:
+
+*   `RENAME TO` clause to rename the continuous aggregate view
+*   `RENAME [COLUMN]` clause to rename the continuous aggregate column
+*   `SET SCHEMA` clause to set the new schema for the continuous aggregate view
+*   `SET TABLESPACE` clause to move the materialization of the continuous
+  aggregate view to the new tablespace
+*   `OWNER TO` clause to set new owner for the continuous aggregate view
+
+<Since2180 />
+
+## Samples
+
+- Disable real-time aggregates for a continuous aggregate:
+
+   ```sql
+   ALTER MATERIALIZED VIEW contagg_view SET (timescaledb.materialized_only);
+   ```
+
+- Enable hypercore for a continuous aggregate:
+
+   ```sql
+   ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' );
+   ```
+
+- Rename a column for a continuous aggregate:
+
+   ```sql
+   ALTER MATERIALIZED VIEW contagg_view RENAME COLUMN old_name TO new_name;
+   ```
+
+- **Enable indexing on data in the columnstore** <EarlyAccess />
+
+   ```sql
+   ALTER MATERIALIZED VIEW stock_candlestick_daily 
+      set access method hypercore,
+      set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' );
+   ``` 
+
+- **Enable indexing on a chunk you are adding to the columnstore** <EarlyAccess />
+
+   ```sql
+   ALTER MATERIALIZED VIEW  _timescaledb_internal._hyper_1_21_chunk
+      set access method hypercore;
+   ```
+
+The only options that currently can be modified with `ALTER
+MATERIALIZED VIEW` are `materialized_only` and `compress`. The other options
+`continuous` and `create_group_indexes` can only be set when creating
+the continuous aggregate.
+
+
+## Arguments
+
+The syntax is:
+
+
+``` sql
+ALTER MATERIALIZED VIEW <view_name> SET (timescaledb.enable_columnstore,
+   timescaledb.materialized_only = 'true' | 'false',
+   timescaledb.orderby = '<column_name> [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]',
+   timescaledb.segmentby = '<column_name> [, ...]',
+   timescaledb.compress_chunk_time_interval='interval',
+   SET ACCESS METHOD { new_access_method | DEFAULT }
+);
+```
+
+| Name                                           | Type | Default                                              | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+|------------------------------------------------|--|------------------------------------------------------|--|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `view_name`                                    |TEXT| -                                                    | ✖ | The materialized view to enable columstore for.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `timescaledb.enable_columnstore`               |BOOLEAN| `true`                                               | ✖ | Enable columnstore.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+|`timescaledb.materialized_only`|BOOLEAN| `true` | ✖ | Enable and disable real time aggregation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `timescaledb.orderby`                          |TEXT| Descending order on the time column in `table_name`. | ✖| The order in which items are used in the columnstore. Specified in the same way as an `ORDER BY` clause in a `SELECT` query.                                                                                                                                                                                                                                                                                                                                                                                                    |
+| `timescaledb.segmentby`                        |TEXT| No segementation by column.                          | ✖| Set the list of columns used to segment data in the columnstore for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate.                                                                                                                                                                                                                                                                                                                                    |
+| `column_name`                                  |TEXT| -                                                    | ✖ | The name of the column to `orderby` or `segmentby`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `timescaledb.compress_chunk_time_interval`     |TEXT| -                                                    | ✖ | EXPERIMENTAL: reduce the total number of chunks in the columnstore for `table`. If you set `compress_chunk_time_interval`, chunks added to the columnstore are merged with the previous adjacent chunk within `chunk_time_interval` whenever possible. These chunks are irreversibly merged. If you call [convert_to_rowstore][convert_to_rowstore], merged chunks are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required. |
+|`SET ACCESS METHOD`|TEXT| DEFAULT ([heap][default_table_access_method])| ✖| To enable indexing on the columstore, set to `hypercore` after you [create a continuous aggregate][create-cagg].    <EarlyAccess />                                                                                                                                                                                                                                                                                                                                                                                       |
+
+[create_materialized_view]: /api/:currentVersion:/continuous-aggregates/create_materialized_view/#parameters
+[postgres-alterview]: https://www.postgresql.org/docs/current/sql-alterview.html
+[create-cagg]: /use-timescale/:currentVersion:/continuous-aggregates/create-a-continuous-aggregate/
+[default_table_access_method]: https://www.postgresql.org/docs/17/runtime-config-client.html#GUC-DEFAULT-TABLE-ACCESS-METHOD