Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrations: Prometheus #3706

Open
wants to merge 16 commits into
base: latest
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions _partials/_prometheus-integrate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
import IntegrationPrereqs from "versionContent/_partials/_integration-prereqs.mdx";

[Prometheus][prometheus] is an open-source monitoring system with a dimensional data model, flexible query language, and a modern alerting approach.

This page shows you how to export your $SERVICE_SHORT telemetry to Prometheus using [PostgreSQL Exporter][postgresql-exporter].

## Prerequisites

<IntegrationPrereqs />

- [Download and run Prometheus][install-prometheus].
- [Install PostgreSQL Exporter][install-exporter].

To reduce latency and potential data transfer costs, install Prometheus and PostgreSQL Exporter on a machine in the
same AWS region as your $SERVICE_LONG.

## Export $SERVICE_SHORT telemetry to Prometheus

To export your data, do the following:

<Procedure>

1. **Create a user to access telemetry data about your $SERVICE_LONG**

1. Connect to your $SERVICE_LONG:

For $CLOUD_LONG, open an [SQL editor][run-queries] in [$CONSOLE][open-console]. For self-hosted, use [`psql`][psql].

1. Create a user named `monitoring` with a secure password:

```sql
CREATE USER monitoring WITH PASSWORD '<password>';
```

1. Grant the `pg_read_all_stats` permission to the `monitoring` user:

```sql
GRANT pg_read_all_stats to monitoring;
```

1. **Import telemetry data about your $SERVICE_LONG to PostgreSQL Exporter**

1. Connect PostgreSQL Exporter to your $SERVICE_LONG:

Use your [connection details][connection-info] to import telemetry data about your $SERVICE_LONG. You connect as
the `monitoring` user:
- Local installation:
```shell
export DATA_SOURCE_NAME="postgres://monitoring:<password>@<host>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require"
./postgres_exporter
```
- Docker:
```shell
docker run -d \
-e DATA_SOURCE_NAME="postgres://monitoring:<password>@<host>.tsdb.cloud.timescale.com:<port>/tsdb?sslmode=require" \
-p 9187:9187 \
prometheuscommunity/postgres-exporter
```

1. Check the metrics for your $SERVICE_LONG in the Prometheus format:
- Browser:

Navigate to `http://<exporter-host>:9187/metrics`.
- Command line:
```shell
curl http://<exporter-host>:9187/metrics
```

1. **Configure Prometheus to scrape metrics**

1. In your Prometheus installation, update `prometheus.yml` to point to your PostgreSQL Exporter instance as a scrape
target. In the following example, you replace `<exporter-host>` with the hostname or IP address of the PostgreSQL
Exporter.

```yaml
global:
scrape_interval: 15s

scrape_configs:
- job_name: 'postgresql'
static_configs:
- targets: ['<exporter-host>:9187']
```

If `prometheus.yml` has not been created during installation, create it manually. If you are using Docker, you can
find the IPAddress in `Inspect` > `Networks` for the container running PostgreSQL Exporter.

1. Restart Prometheus.

1. Check the Prometheus UI at `http://<prometheus-host>:9090/targets` and `http://<prometheus-host>:9090/tsdb-status`.

You see the PostgreSQL Exporter target and the metrics scraped from it.

</Procedure>

You can further [visualize your data][grafana-prometheus] with Grafana. Use the
[Grafana PostgreSQL dashboard][postgresql-exporter-dashboard] or [create a custom dashboard][grafana] that suits your needs.

[install-exporter]: https://grafana.com/oss/prometheus/exporters/postgres-exporter/?tab=installation
[postgresql-exporter-dashboard]: https://grafana.com/oss/prometheus/exporters/postgres-exporter/?tab=dashboards
[install-prometheus]: https://prometheus.io/docs/prometheus/latest/installation/
[grafana]: /use-timescale/:currentVersion:/integrations/grafana/
[grafana-prometheus]: https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-prometheus/
[prometheus]: https://prometheus.io/docs/introduction/overview/
[run-queries]: /getting-started/:currentVersion:/run-queries-from-console/
[psql]: /use-timescale/:currentVersion:/integrations/psql/
[connection-info]: /use-timescale/:currentVersion:/integrations/find-connection-details/
[postgresql-exporter]: https://grafana.com/oss/prometheus/exporters/postgres-exporter/
[open-console]: https://console.cloud.timescale.com/dashboard/services
8 changes: 5 additions & 3 deletions use-timescale/integrations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,11 @@ Some of the most in-demand integrations for $CLOUD_LONG are listed below, with l

## Observability and alerting

| Name | Description |
|:---------------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Name | Description |
|:---------------------------:|----------------------------------------------------------------------------------------------------------------------------------|
| [Grafana][grafana] | An open-source analytics and monitoring solution that enables you to query, visualize, alert on, and explore your metrics, logs. |
| [Tableau][tableau] | A popular analytics platform that helps you gain greater intelligence about your business. |
| [Tableau][tableau] | A popular analytics platform that helps you gain greater intelligence about your business. |
| [Prometheus][prometheus]| An open-source monitoring system with a dimensional data model, flexible query language, and a modern alerting approach. |


## Configuration and deployment
Expand Down Expand Up @@ -60,3 +61,4 @@ Some of the most in-demand integrations for $CLOUD_LONG are listed below, with l
[terraform]: /use-timescale/:currentVersion:/integrations/terraform
[apache-airflow]: /use-timescale/:currentVersion:/integrations/apache-airflow
[postgresql-integrations]: https://slashdot.org/software/p/PostgreSQL/integrations/
[prometheus]: /use-timescale/:currentVersion:/integrations/prometheus
12 changes: 12 additions & 0 deletions use-timescale/integrations/prometheus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: Integrate Prometheus with Timescale Cloud
excerpt: Export telemetry metrics from your Timescale Cloud service to Prometheus
products: [cloud, mst, self_hosted]
keywords: [integrate]
---

import PrometheusIntegrate from "versionContent/_partials/_prometheus-integrate.mdx";

# Integrate Prometheus with $CLOUD_LONG

<PrometheusIntegrate />
61 changes: 2 additions & 59 deletions use-timescale/metrics-logging/metrics-to-prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,65 +10,8 @@ cloud_ui:
- [services, :serviceId, operations, integrations]
---

import ExporterRegionNote from 'versionContent/_partials/_cloud-integrations-exporter-region.mdx';

import PrometheusIntegrate from "versionContent/_partials/_prometheus-integrate.mdx";

# Export metrics to Prometheus

You can export your Timescale service telemetry to Prometheus.

## Create a monitoring user

1. Connect to the Timescale service as tsdbadmin (using psql or a similar tool).

1. Create a new user to scrape the metrics from the database using the required permissions:

a. Create a user named `monitoring` with the desired password by replacing `<password>`:

```sql
CREATE USER monitoring WITH PASSWORD '<password>';
```

b. Grant `pg_read_all_stats` to the `monitoring` user:

```sql
GRANT pg_read_all_stats to monitoring;
```

## Install PostgreSQL exporter

1. Install the postgresql_exporter on a host that you manage to connect to the Timescale service and collect telemetry. Below is an example using Docker to run postgresql_exporter:

make sure to replace `<PASSWORD>` with the created password and `<TIMESCALE-HOST:PORT>` with the timescale service host and port.

```bash
docker run \
-p 9187:9187 \
-e DATA_SOURCE_NAME="postgres://monitoring:<PASSWORD>@<TIMESCALE-HOST:PORT>/tsdb?sslmode=require" \
quay.io/prometheuscommunity/postgres-exporter
```
To reduce latency and potential data transfer costs, best practice is to run `postgresql_exporter` in the same AWS region as your Timescale Cloud service.

1. Once the postgresql_exporter is up and running, and successfully connected to the Timescale service, you can configure your Prometheus server to scrape the postgresql_exporter metrics endpoint. This endpoint exposes all the metrics provided by the exporter.

To view all the exposed metrics, execute curl on the same host:
```bash
curl localhost:9187/metrics
```

## Grafana dashboard for PostgreSQL metrics

Use the [PostgreSQL dashboard][postgresql-exporter-dashboard] to visualize the following metrics:
* Current QPS
* Fetched, Returned, Inserted, Updated, Deleted Rows
* Database Deadlocks and Conflicts
* Cache Hit Ratio
* Number of Active Connections
* Buffers

<Highlight type="note">
Furthermore, you can create custom dashboards tailored to your specific needs using the metrics obtained from the Timescale service.
</Highlight>


[postgresql-exporter-dashboard]: https://grafana.com/oss/prometheus/exporters/postgres-exporter/?tab=dashboards
<PrometheusIntegrate />
5 changes: 5 additions & 0 deletions use-timescale/page-index/page-index.js
Original file line number Diff line number Diff line change
Expand Up @@ -798,6 +798,11 @@ module.exports = [
href: "pgadmin",
excerpt: "Install pgAdmin to connect to Timescale",
},
{
title: "Prometheus",
href: "prometheus",
excerpt: "Integrate Prometheus with Timescale Cloud",
},
{
title: "psql",
href: "psql",
Expand Down