Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hypercore API ref #3699

Draft
wants to merge 41 commits into
base: release-2.18.0-main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
7bc034a
chore: space to validate initial build.
billy-the-fish Jan 9, 2025
b7e966e
chore: add first hypercore files.
billy-the-fish Jan 9, 2025
3899a6e
chore: add api ref files and alter_table to the build.
billy-the-fish Jan 9, 2025
bd9946c
chore: add remove_columnstore_policy to the build.
billy-the-fish Jan 9, 2025
5ad76e2
chore: add convert_to_columnstore to the build.
billy-the-fish Jan 9, 2025
993cf07
chore: add convert_to_rowstore to the build.
billy-the-fish Jan 9, 2025
d1fde35
chore: updates for build.
billy-the-fish Jan 9, 2025
f727241
chore: updates for build.
billy-the-fish Jan 9, 2025
67c638f
chore: remove links from steps in a list.
billy-the-fish Jan 9, 2025
39c8f8c
chore: remove links from steps in a list.
billy-the-fish Jan 9, 2025
193f0d6
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 9, 2025
af34323
chore: hopefully this will work now.
billy-the-fish Jan 9, 2025
a80f250
chore: add deprecation for compression.
billy-the-fish Jan 9, 2025
c03cae9
chore: simplify workflow, keep code for main docs.
billy-the-fish Jan 9, 2025
e58a7c9
chore: updates on review. .
billy-the-fish Jan 9, 2025
338bac1
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 10, 2025
1409dfd
chore: updates on review.
billy-the-fish Jan 10, 2025
d8008e0
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 13, 2025
a4bf7f1
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 14, 2025
de1c436
feat: update structure for hypercore
billy-the-fish Jan 6, 2025
834a406
chore: add replacedby link. Has to be hardcoded so will not work unti…
billy-the-fish Jan 15, 2025
e969ec0
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 15, 2025
fe6d956
Apply suggestions from code review
billy-the-fish Jan 15, 2025
8c87b3c
chore: update on review.
billy-the-fish Jan 15, 2025
25a2bcc
chore: add enable_segmentwise_recompression.
billy-the-fish Jan 16, 2025
c849517
chore: add enable_segmentwise_recompression.
billy-the-fish Jan 16, 2025
be89feb
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 16, 2025
93ebdde
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 17, 2025
e4a4a57
chore: update deprecation notices.
billy-the-fish Jan 17, 2025
8025d11
chore: update deprecation notices.
billy-the-fish Jan 17, 2025
ce662f7
chore: remove stuff breaking the build.
billy-the-fish Jan 17, 2025
cc5030d
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 20, 2025
7ca04c9
chore: remove stuff breaking the build.
billy-the-fish Jan 18, 2025
629d4a8
chore: updates on review.
billy-the-fish Jan 23, 2025
7e0eae4
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 23, 2025
da173a7
Add API description for merge_chunks() (#3738)
erimatnor Jan 23, 2025
277031e
Merge branch 'latest' into 235-docs-rfc-update-the-api-ref-for-hypers…
billy-the-fish Jan 27, 2025
41564b4
chore: update return table.
billy-the-fish Jan 28, 2025
80b0e3d
chore: change from deprecated to Old API.
billy-the-fish Jan 29, 2025
e5135b4
chore: change from deprecated to Old API.
billy-the-fish Jan 29, 2025
5deb4f8
chore: update on review.
billy-the-fish Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _partials/_deprecated_2_18_0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<Tag variant="hollow">Deprecated [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)</Tag>
57 changes: 57 additions & 0 deletions _partials/_hypercore_policy_workflow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
1. **Enable columnstore**

* [Use `ALTER TABLE` for a hypertable][alter_table_hypercore]
```sql
ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol');
```
* [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate]
```sql
ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' );
```

1. **Add a policy to move chunks to the columnstore at a specific time interval**

For example, 60 days after the data was added to the table:
``` sql
SELECT add_columnstore_policy('older_stock_prices', after => INTERVAL '60d');
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
```
See [add_columnstore_policy][add_columnstore_policy].

1. **View the policies that you set or the policies that already exist**

``` sql
SELECT * FROM timescaledb_information.jobs
WHERE proc_name='policy_compression';
```
See [timescaledb_information.jobs][informational-views].

1. **Pause a columnstore policy**

``` sql
SELECT * FROM timescaledb_information.jobs where proc_name = 'policy_compression' AND relname = 'stocks_real_time'

-- Select the JOB_ID from the results

SELECT alter_job(JOB_ID, scheduled => false);
```
See [alter_job][alter_job].

1. **Restart a columnstore policy**
``` sql
SELECT alter_job(JOB_ID, scheduled => true);
```
See [alter_job][alter_job].

1. **Remove a columnstore policy**
``` sql
SELECT remove_columnstore_policy('older_stock_prices');
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
```
See [remove_columnstore_policy][remove_columnstore_policy].
1. **Disable columnstore**

If your table has chunks in the columnstore, you have to
[convert the chunks back to the rowstore][convert_to_rowstore] before you disable the columnstore.
``` sql
ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = false);
```
See [alter_table_hypercore][alter_table_hypercore].
1 change: 1 addition & 0 deletions _partials/_since_2_18_0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<Tag variant="hollow">Since [TimescaleDB v2.18.0](https://github.com/timescale/timescaledb/releases/tag/2.18.0)</Tag>
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ api:
license: community
type: function
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# add_compression_policy() <Tag type="community" content="community" />

Expand All @@ -23,6 +24,8 @@ command. To enable compression on continuous aggregates, use the
command. To view the policies that you set or the policies that already exist,
see [informational views][informational-views].

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/add_columnstore_policy/">add_columnstore_policy()</a>.

## Required arguments

|Name|Type|Description|
Expand Down Expand Up @@ -89,3 +92,4 @@ SELECT add_compression_policy('cpu_weekly', INTERVAL '8 weeks');
[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/
[set_integer_now_func]: /api/:currentVersion:/hypertable/set_integer_now_func
[informational-views]: /api/:currentVersion:/informational-views/jobs/

Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ api:
license: community
type: command
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# ALTER TABLE (Compression) <Tag type="community" content="community" />

Expand All @@ -28,6 +29,8 @@ ALTER TABLE <table_name> SET (timescaledb.compress,
);
```

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/alter_table/">ALTER TABLE (Hypercore)</a>.

## Required arguments

|Name|Type|Description|
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ api:
license: community
type: function
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# chunk_compression_stats() <Tag type="community">Community</Tag>

Expand All @@ -20,6 +21,8 @@ the chunk. An insert into a compressed chunk does not update the compressed
sizes. For more information about how to compute chunk sizes, see the
`chunks_detailed_size` section.

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/chunk_columnstore_stats/">chunk_columnstore_stats()</a>.

### Required arguments

|Name|Type|Description|
Expand Down
4 changes: 4 additions & 0 deletions api/compress_chunk.md → api/compression/compress_chunk.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ api:
type: function
---

import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# compress_chunk() <Tag type="community">Community</Tag>

The `compress_chunk` function is used to compress (or recompress, if necessary)
Expand All @@ -27,6 +29,8 @@ You can get a list of chunks belonging to a hypertable using the
[`show_chunks` function](/api/latest/hypertable/show_chunks/).
</Highlight>

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/convert_to_columnstore/">convert_to_columnstore()</a>.

### Required arguments

|Name|Type|Description|
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ api:
license: community
type: function
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# decompress_chunk() <Tag type="community">Community</Tag>

Expand All @@ -22,6 +23,8 @@ turn the policy back on. The database automatically recompresses your chunks in
the next scheduled job.
</Highlight>

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/convert_to_cconvert_to_rowstore/">convert_to_rowstore()</a>.

### Required arguments

|Name|Type|Description|
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ api:
license: community
type: function
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# hypertable_compression_stats() <Tag type="community">Community</Tag>

Expand All @@ -19,6 +20,8 @@ see the [hypertable section][hypertable-docs].
For more information about compression, see the
[compression sction][compression-docs].

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/hypertable_columnstore_stats/">hypertable_columnstore_stats()</a>.

### Required arguments

|Name|Type|Description|
Expand Down
3 changes: 3 additions & 0 deletions api/compression.md → api/compression/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ excerpt: Compress your hypertable
keywords: [compression]
tags: [hypertables]
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# Compression <Tag type="community">Community</Tag>

Expand All @@ -30,6 +31,8 @@ Compressed chunks have the following limitations:
after constraint creation.
* [Timescale SkipScan][skipscan] does not currently work on compressed chunks.

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/">Hypercore</a>.
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved

## Restrictions

In general, compressing a hypertable imposes some limitations on the types
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ api:
type: function
---

import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# recompress_chunk() <Tag type="community" content="Community" />

Recompresses a compressed chunk that had more data inserted after compression.
Expand Down Expand Up @@ -41,6 +43,8 @@ the procedure with `CALL`. Don't use a `SELECT` statement.
chunk for the first time, use [`compress_chunk`](/api/latest/compression/compress_chunk/).
</Highlight>

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/convert_to_columnstore/">convert_to_columnstore()</a>.

## Required arguments

|Name|Type|Description|
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,16 @@ api:
license: community
type: function
---
import Deprecated2180 from "versionContent/_partials/_deprecated_2_18_0.mdx";

# remove_compression_policy() <Tag type="community" content="community" />

If you need to remove the compression policy. To restart policy-based
compression you need to add the policy again. To view the policies that
already exist, see [informational views][informational-views].

<Deprecated2180 /> Replaced by <a href="https://docs.timescale.com/api/latest/hypercore/remove_columnstore_policy/">remove_columnstore_policy()</a>.

### Required arguments

|Name|Type|Description|
Expand Down
108 changes: 108 additions & 0 deletions api/hypercore/add_columnstore_policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
api_name: add_columnstore_policy()
excerpt: Set a policy to automatically move chunks in a hypertable to the columnstore when they reach a given age.
topics: [columnstore, hypercore, jobs]
keywords: [columnstore, hypercore, policies]
tags: [scheduled jobs, background jobs, automation framework]
products: [cloud, self_hosted]
api:
license: community
type: function
---

import Since2180 from "versionContent/_partials/_since_2_18_0.mdx";

# add_columnstore_policy()

Create a [job][job] that automatically moves chunks in a hypertable to the columnstore after a
specific time interval.

You enable the columnstore a hypertable or continuous aggregate before you create a columnstore policy.
You do this by calling `ALTER TABLE` for hypertables and `ALTER MATERIALIZED VIEW` for continuous aggregates.

To view the policies that you set or the policies that already exist,
see [informational views][informational-views], to remove a policy, see [remove_columnstore_policy][remove_columnstore_policy].

<Since2180 />

## Samples

To create a columnstore job:

<Procedure>

1. **Enable columnstore**

* [Use `ALTER TABLE` for a hypertable][compression_alter-table]
```sql
ALTER TABLE stocks_real_time SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol');
```
* [Use ALTER MATERIALIZED VIEW for a continuous aggregate][compression_continuous-aggregate]
```sql
ALTER MATERIALIZED VIEW stock_candlestick_daily set (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'symbol' );
```

1. **Add a policy to move chunks to the columnstore at a specific time interval**

For example:

* 60 days after the data was added to the table:
``` sql
SELECT add_columnstore_policy('stocks_real_time', after => INTERVAL '60d');
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
```
* 3 months prior to the moment you run the query:

``` sql
SELECT add_columnstore_policy('stocks_real_time', created_before => INTERVAL '3 months');
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
```
* With an integer-based time column:

``` sql
SELECT add_columnstore_policy('table_with_bigint_time', BIGINT '600000');
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
```
* Older than eight weeks:

``` sql
SELECT add_columnstore_policy('cpu_weekly', INTERVAL '8 weeks');
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
```

1. **View the policies that you set or the policies that already exist**

``` sql
SELECT * FROM timescaledb_information.jobs
WHERE proc_name='policy_compression';
```
See [timescaledb_information.jobs][informational-views].

</Procedure>

## Arguments

Calls to `add_columnstore_policy` require either `after` or `created_before`, but cannot have both.

<!-- vale Google.Acronyms = NO -->
<!-- vale Vale.Spelling = NO -->

| Name | Type | Default | Required | Description |
|--|--|--|--|--|
| `hypertable` |REGCLASS| - | ✔ | Name of the hypertable or continuous aggregate to run this [job][job] on.|
| `after` |INTERVAL or INTEGER|- | ✖ | Add chunks containing data older than `now - {after}::interval` to the columnstore. <br/> Use an object type that matchs the time column type in `hypertable`: <ul><li><b><code>TIMESTAMP</code>, <code>TIMESTAMPTZ</code>, or <code>DATE</code></b>: use an <code>INTERVAL</code> type.</li><li><b> Integer-based timestamps </b>: set an integer type using the [integer_now_func][set_integer_now_func].</li></ul> `after` is mutually exclusive with `created_before`. |
| `created_before` |INTERVAL| NULL | ✖ | Add chunks with a creation time of `now() - created_before` to the columnstore. <br/> `created_before` is <ul><li>Not supported for continuous aggregates.</li><li>Mutually exclusive with `after`.</li></ul> |
| `schedule_interval` |INTERVAL| 12 hours when [chunk_time_interval][chunk_time_interval] >= `1 day` for `hypertable`. Otherwise `chunk_time_interval` / `2`. | ✖ | Set the interval between the finish time of the last execution of this policy and the next start.|
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
| `initial_start` |TIMESTAMPTZ| The interval from the finish time of the last execution to the [next_start][next-start].| ✖| Set the time this job is first run. This is also the time that `next_start` is calculated from.|
| `timezone` |TEXT| UTC. However, daylight savings time(DST) changes may shift this alignment. | ✖ | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`.|
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved
| `if_not_exists` |BOOLEAN| `false` | ✖ | Set to `true` so this job fails with a warning rather than an error if a columnstore policy already exists on `hypertable` |
billy-the-fish marked this conversation as resolved.
Show resolved Hide resolved


<!-- vale Google.Acronyms = YES -->
<!-- vale Vale.Spelling = YES -->


[compression_alter-table]: /api/:currentVersion:/hypercore/alter_table/
[compression_continuous-aggregate]: /api/:currentVersion:/continuous-aggregates/alter_materialized_view/
[set_integer_now_func]: /api/:currentVersion:/hypertable/set_integer_now_func
[informational-views]: /api/:currentVersion:/informational-views/jobs/
[chunk_time_interval]: /api/:currentVersion:/hypertable/set_chunk_time_interval/
[next-start]: /api/:currentVersion:/informational-views/jobs/#arguments
[job]: /api/:currentVersion:/actions/add_job/
[remove_columnstore_policy]: /api/:currentVersion:/hypercore/remove_columnstore_policy/
Loading
Loading