diff --git a/docs/evaluate/index.mdx b/docs/evaluate/index.mdx index c1d9fe0238..8345330c91 100644 --- a/docs/evaluate/index.mdx +++ b/docs/evaluate/index.mdx @@ -22,3 +22,5 @@ Thousands of companies of all sizes are leveraging Temporal's capabilities for b - [Why Temporal](/evaluate/why-temporal) - [Security](/security) - [Release stages](/evaluate/release-stages) +- [Development features](/evaluate/development-features) +- [Production deployment features](/evaluate/production-features) diff --git a/docs/evaluate/production-features/index.mdx b/docs/evaluate/production-features/index.mdx new file mode 100644 index 0000000000..6641612d75 --- /dev/null +++ b/docs/evaluate/production-features/index.mdx @@ -0,0 +1,36 @@ +--- +id: index +title: Temporal's production deployment features +description: Explore your deployment options for production traffic with Temporal, offering both Self-hosted and Temporal Cloud solutions. +sidebar_label: Production features +tags: + - production-features +keywords: + - temporal service deployment + - self-host temporal service + - temporal cloud benefits + - production traffic management + - high availability workflows + - multi-tenant temporal service + - workflow state retention + - temporal community support + - temporal cloud vs self-hosted + - workflow history export +--- + +Transform your Temporal applications into production-ready systems by deploying your application code, Workflows, Activities, and Workers for operational use. +When your application is ready to start serving production traffic, we offer two Temporal Service options: + +- **[Choose Temporal Cloud for your Temporal Service](/cloud)** + Let us handle the Temporal Service operations so you can focus on your applications. +- **[Self-host a Temporal Service](/self-hosted-guide)** + Deploy your own production level Temporal Service to orchestrate your durable applications. + +| Feature | Temporal Cloud | Self-hosted | +| ---------------------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | +| **Multi-tenant** | ✅ Up to 100 Namespaces | ✅ Unlimited Namespaces | +| **High availability and failover** | ✅ [Multi-region Namespaces](/evaluate/production-features/temporal-cloud/high-availability) | ✅ [Global Namespaces & Multi-Cluster Replication](/evaluate/production-features/self-hosted/high-availability) | +| **Application state persistence** | ✅ 30-90 day Retention | ✅ Unlimited | +| **Long term state retention** | ✅ Workflow History Export | ✅ Archival | +| **Community support** | ✅ Slack, Forum | ✅ Slack, Forum | +| **Paid support** | ✅ Prioritized responses | ✖️ | \ No newline at end of file diff --git a/docs/evaluate/production-features/self-hosted/high-availability.mdx b/docs/evaluate/production-features/self-hosted/high-availability.mdx new file mode 100644 index 0000000000..01f89d7b95 --- /dev/null +++ b/docs/evaluate/production-features/self-hosted/high-availability.mdx @@ -0,0 +1,17 @@ +--- +id: high-availability +title: High availability - Self-hosted Temporal production feature +description: Explore your deployment options for production traffic with Temporal, offering both Self-hosted and Temporal Cloud solutions. +sidebar_label: High availability +tags: + - production-deployment + - production-features + - self-hosted-temporal +--- + +Explore Temporal's self-hosted high-availability features. + +* [Global Namespace](/namespaces#global-namespace): + A Namespace that exists across Temporal Services to enhance your Namespace availability to higher levels. + A global Namespace automatically replicates metadata and Workflow Execution state from an active Temporal Service to a standby Temporal Service. + In the case of operational issues, a [failover](/glossary#failover) allows responsibility to pass from the active to the standby Temporal Service. diff --git a/docs/evaluate/production-features/self-hosted/index.mdx b/docs/evaluate/production-features/self-hosted/index.mdx new file mode 100644 index 0000000000..2e0d399d9d --- /dev/null +++ b/docs/evaluate/production-features/self-hosted/index.mdx @@ -0,0 +1,28 @@ +--- +id: index +title: Self-hosted production features +description: Roll your own solutions with Temporal Server +sidebar_label: Self-host +tags: + - production-deployment + - production-features + - temporal-self-hosted +keywords: + - temporal service deployment + - self-host temporal service + - production traffic management + - high availability workflows + - multi-tenant temporal service + - workflow state retention + - temporal community support + - workflow history export +--- + +Explore self-hosting when you need a production-ready Temporal Service to coordinate the execution of your Workflows and Activities. +See the [Self-hosted guide](/self-hosted-guide) to get started. + +Make the most of your self-hosted deployment with these features: + +* [High Availability](/evaluate/production-features/self-hosted/high-availability): Maximize availability and provide disaster-tolerant deployment for your self-hosted workloads. + + diff --git a/docs/evaluate/production-features/temporal-cloud/high-availability.mdx b/docs/evaluate/production-features/temporal-cloud/high-availability.mdx new file mode 100644 index 0000000000..580bf98437 --- /dev/null +++ b/docs/evaluate/production-features/temporal-cloud/high-availability.mdx @@ -0,0 +1,119 @@ +--- +id: high-availability +title: High availability - Temporal Cloud production feature +description: Add push-button high availability for your Temporal Service to ensure continuous reliable operation. +sidebar_label: High availability +tags: + - production-deployment + - production-features + - temporal-cloud + - high-availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- + +:::tip Support, stability, and dependency info +- Multi-region Namespaces are in [Public Preview](/evaluate/release-stages#public-preview) for Temporal Cloud. + +::: + +Temporal Cloud offers disaster-tolerant deployment for workloads with stringent availability requirements. +With the multi-region feature enabled, Temporal Cloud automates [failover](/glossary#failover] and synchronizes data between Namespace regions. + +This page introduces Temporal Cloud patterns that support your workload's availability requirements. + +## Multi-region Namespaces + +Multi-region Namespaces provide failover capabilities to mitigate service outages due to regional failures. +They reduce risk and minimize operational disruption. +This feature seamlessly shif​​ts Workflow execution between regions to maintain service availability. + +Your Clients use a single logical Namespace with a single endpoint that operates in two physical regions: one active and one standby. +As Workflows progress in the active region, history events asynchronously replicate to the standby region. +Data replication ensures both regions are in sync so the standby is ready to take over when needed. + +Should an incident or outage occur in the active region, Temporal Cloud initiates a "failover" to the standby region. +During a failover, the roles of the active and standby regions reverse. +The standby takes over as the primary region. + +### Advantages of multi-region Namespaces {#multi-region-advantages} + +**Why choose a multi-region Namespace (MRN)?** + +* **Reduce Risk**: + MRN's protects your operations from unexpected regional outages. + Its automated disaster recovery features ensure that workloads remain available and continue execution. +* **Minimize Operational Disruption**: + Seamless failovers shift Workflow Executions to a secondary region during outages. + MRNs maintain service availability without needing manual synchronization between Namespaces. + Real-time alerts during failover events keep you informed. +* **No manual deployment or configuration needed:** + Temporal Cloud simplifies deployment with push-button operation. + This eliminates the need for manual deployment or configuration. +* **Fault tolerance**. + Your open Workflows continue their progress in the standby region. + This minimizes interruption and data loss during regional failures. +* **No code changes**. + Your Workers and Workflow starter code don't need updates to take advantage of multi-region setup or to respond to failover conditions. + This allows for a smooth transition and continued operation. + +### Service Level Objectives (SLO) and guarantees {#multi-region-SLO} + +**What reliability promises does this feature offer?** + +* Temporal provides a 99.99% Contractual SLA that provides redress in the event of downtime ([SLA](https://docs.temporal.io/cloud/sla)). +* [RTO](https://csrc.nist.gov/glossary/term/recovery_time_objective): 20 minutes or less. +* [RPO](https://csrc.nist.gov/glossary/term/recovery_point_objective): Near zero. + +### Target workloads {#target-workloads} + +**Who benefits from this feature?** + +Multi-region Namespaces are a great solution for Workloads where a regional cloud outage would cause: + +* Revenue loss +* Poor customer experience +* Problems stemming from policy/legal requirements that demand high availability + +Some examples: financial services, e-commerce, gaming, global SaaS platforms, bookings & reservations, delivery & shipping, order management. + +### Explore {#explore-multi-region} + +**Read more about our multi-region features** + +* [Multi-region Namespaces](/cloud/multi-region) offer High Availability service for Temporal Cloud customers who need the highest level of availability at all times. +* [Multi-region Pricing](/cloud/pricing) scales to use. +* Multi-region high availability supports [PrivateLink routing](/cloud/multi-region#routing). + +## Single-region Namespaces + +A typical Temporal Cloud Namespace is deployed into one [AWS region](https://docs.temporal.io/cloud/service-availability). +Temporal Cloud provides [99.99% availability](https://docs.temporal.io/cloud/sla) and a contractual [service level agreement](https://docs.temporal.io/cloud/sla) (SLA) of 99.9% guarantee against service errors. +It provides a great all-around solution that's suitable for most organizations. + +### Advantages of single-region Namespaces {#single-region-advantages} + +**Why choose single-region Namespaces?** + +* This option offers sufficient availability for most use cases and workloads. +* Temporal Cloud provides 99.99% availability and a contractual service level agreement of 99.9% guarantee against service errors. + Read more on our [SLA page](https://docs.temporal.io/cloud/sla). + +Some downsides of single-region Namespaces compared to multi-region Namespaces: + +* Stalled work during failures: Open Workflow Executions pause until the region/Namespace recovers. +* Blocked work initiation: No new Workflow Executions will start until the region/Namespace recovers. + +### Explore {#explore-temporal-namespaces} + +**Read more about Namespaces** + +* [Temporal Cloud Namespaces](/cloud/namespaces) offer outstanding reliable service for Temporal Cloud customers. \ No newline at end of file diff --git a/docs/evaluate/production-features/temporal-cloud/index.mdx b/docs/evaluate/production-features/temporal-cloud/index.mdx new file mode 100644 index 0000000000..db9d77faf2 --- /dev/null +++ b/docs/evaluate/production-features/temporal-cloud/index.mdx @@ -0,0 +1,34 @@ +--- +id: index +title: Temporal Cloud production features +sidebar_label: Temporal Cloud +description: Learn about Temporal Cloud, Temporal's software-as-a-service infrastructure platform designed to manage the durability of your Temporal Applications. +sidebar_position: 1 +tags: + - production-deployment + - production-features + - temporal-cloud +keywords: + - temporal service deployment + - temporal cloud benefits + - production traffic management + - high availability workflows + - multi-tenant temporal service + - workflow state retention + - temporal community support + - workflow history export +--- + + +[Temporal Cloud](https://temporal.io/cloud) a software-as-a-service (SaaS) infrastructure platform designed to manage the durability of your Temporal Applications. +Temporal Cloud oversees and manages the execution process for your production evironments for your distributed applications and services. +It provides durable execution, allowing failable processes to be retried, and capturing progress so that your Workflows execute reliably, recoverably, and performantly. + +Temporal Cloud features include: + +- [High Availability](/evaluate/production-features/temporal-cloud/high-availability): Add push-button high availability to your Temporal Cloud Service with multi-region Namespaces (MRN). +MRNs provide seamless failover capabilities to your production work, ensuring continuous reliable operation. + +We handle the operations of running the Temporal Service. +You focus on your application. +See the [Temporal Cloud guide](/cloud) to get started. diff --git a/docs/glossary.md b/docs/glossary.md index f26764dcd8..4469a91c44 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -183,6 +183,13 @@ An append-only log of Events that represents the full state a Workflow Execution _Tags: [term](/tags/term), [explanation](/tags/explanation)_ +#### [Failover](/cloud/multi-region) + +A Failover shifts Workflow Execution processing from an active Temporal Namespace to a standby Temporal Namespace during outages or other incidents. +Standby Namespaces use replication to duplicate data and prevent data loss during Failover. + +_Tags: [term](/tags/term), [explanation](/tags/explanation)_ + #### [Failure](/temporal#failure) Temporal Failures are representations of various types of errors that occur in the system. @@ -215,7 +222,9 @@ _Tags: [product-release-stages](/tags/product-release-stages), [term](/tags/term #### [Global Namespace](/namespaces#global-namespace) -A Global Namespace is a Namespace that exists across Clusters when Multi-Cluster Replication is set up. +A Global Namespace is a Namespace that duplicates data from an active [Temporal Service](#temporal-cluster) to a standby Service using the replication to keep both Namespaces in sync. +Global Namespaces are designed to respond to service issues like network congestion. +When service to the primary Cluster is compromised, a [Failover](#failover) transfers control from the active to the standby cluster. _Tags: [term](/tags/term), [explanation](/tags/explanation)_ @@ -267,6 +276,16 @@ Multi-Cluster Replication is a feature which asynchronously replicates Workflow _Tags: [term](/tags/term), [explanation](/tags/explanation)_ +#### [Multi-region Namespace](/cloud/multi-region) + +A multi-region Namespace (MRN) is a [Temporal Cloud](#temporal-cloud) Namespace that is configured to work across an active [region](/cloud/service-availability#regions) and a standby region. +Each region corresponds to a dedicated Temporal Cloud Service. +Temporal Cloud automatically replicates Workflow Executions and metadata from the active to the standby region. +MRNs are designed to respond to service issues as they arise. +In the event that the Namespace's performance is compromised, a [Failover](#failover) transfers control from the active to the standby region. + +_Tags: [term](/tags/term), [explanation](/tags/explanation)_ + #### [Namespace](/namespaces) A Namespace is a unit of isolation within the Temporal Platform diff --git a/docs/production-deployment/cloud/audit-logging.mdx b/docs/production-deployment/cloud/audit-logging.mdx index 54c9a114db..e3ceeb29c3 100644 --- a/docs/production-deployment/cloud/audit-logging.mdx +++ b/docs/production-deployment/cloud/audit-logging.mdx @@ -63,6 +63,8 @@ The following list specifies both the supported events and the Temporal APIs tha - Add custom Search Attributes: `UpdateNamespace` - Rename custom Search Attribute: `RenameCustomSearchAttribute` - Request increase in Retention Period: `UpdateNamespace` +- Multi-region Namespace + - Failover Namespace: `FailoverNamespace` ### API Key Operation events diff --git a/docs/production-deployment/cloud/introduction/pricing.mdx b/docs/production-deployment/cloud/introduction/pricing.mdx index 820a14b9be..73aa072b6b 100644 --- a/docs/production-deployment/cloud/introduction/pricing.mdx +++ b/docs/production-deployment/cloud/introduction/pricing.mdx @@ -1,7 +1,8 @@ --- id: pricing -title: Pricing - Temporal Cloud +title: Temporal Cloud Pricing sidebar_label: Pricing +sidebar_position: 6 description: Temporal Cloud pricing information slug: /cloud/pricing toc_max_heading_level: 4 @@ -27,35 +28,51 @@ tags: - term --- -Temporal Cloud is a consumption-based service; you pay only for what you need when you need it. Our [pricing](https://temporal.io/pricing) derives from your use of _actions_, _storage_, and _support_. Pricing is flexible, transparent, and predictable, so you know your costs and never pay for unused capacity. +Temporal Cloud is a consumption-based service. +You pay only for what you need, when you need it. +Our pricing reflects your use of _actions_, _storage_, and _support_. +It is flexible, transparent, and predictable, so you know your costs and never pay for unused capacity. -This page introduces the elements of Temporal Cloud pricing—[actions](#action), [storage](#storage), and [support](#support)—so that you can estimate costs for your implementation. To gain a reliable estimate, [contact our team](https://pages.temporal.io/contact-us). +This page introduces the elements of Temporal Cloud pricing for [actions](#action), [storage](#storage), and [services/support](#support). +It gives you the information you need to estimate costs for your implementation. +For more accurate estimates, [reach out to our team](https://pages.temporal.io/contact-us). -## What is an Action? {#action} +## Action pricing {#action} -Actions are the fundamental consumption pricing unit in Temporal Cloud. -An Action in Temporal occurs as part of an execution of your Workflow. -Each time you execute a Temporal Workflow (a Workflow Execution), the associated Actions are collected and ultimately represent the state and progress of your Temporal Application. +**What is an Action?** -Actions are collected and billed monthly for each Namespace. The base rate is \$25 per one million Actions per month (across all Namespaces), and you are billed only for the prorated amount of Actions you use. If you use fewer than one million Actions per month, your bill for Actions will be less than \$25 for that month. +Actions serve as the primary unit of consumption-based pricing for Temporal Cloud. +Temporal Actions are the building blocks for Workflow Executions. +When you execute a Temporal Workflow, its Actions create the ongoing state and progress of your Temporal Application. -| **Actions per month** | **Cost per 1M (USD)** | -| --------------------- | --------------------- | -| Any number | $25.00 (prorated) | +Actions are collected and billed monthly for each Namespace. +The base rate is $25 per one million Actions per month (across all Namespaces). +You are billed for a prorated portion of the Actions you use. +If you use fewer than one million Actions per month, your bill for Actions will be less than $25 for that month. -Alternatively, Temporal also offers a credit system. Credits provide an additional discount schedule for both billable Actions and [storage](#storage). + +| Actions | $25 per one million (prorated) | +| ------- | ------------------------------ | + +Temporal also offers a credit system. +Credits provide an additional discount schedule for billable Actions and [storage](#storage). Credits do not expire. -The following table outlines cost estimates for the credits system. -Please [reach out to the team](https://pages.temporal.io/contact-us) if you are interested in this option. - -| **Actions (Per Million Actions per Namespace)** | **Cost (USD)** | -| ----------------------------------------------- | -------------- | -| Up to 300M | $23.25 | -| Over 300M up to 1.5B | $18.80 | -| Over 1.5B up to 7.5B | $14.10 | -| Over 7.5B up to 30B | $10.50 | -| Over 30B up to 150B | $7.90 | -| Over 150B | $5.90 | + +The credits applied to a Temporal Cloud Account include volume-based discounts for each Namespace. +Pricing may not be uniform per Namespace. +For example, under one credit commitment, a Namespace may use up to 300 million Actions priced at $23.25 per million Actions. +Another Namespace with over 300 million Actions benefits from a reduced rate of $18.80 per million Actions for any Actions beyond the 300 million mark. + +The following table outlines the tiers for the credits system. + +|Actions per month (millions)|Cost per 1M (USD)|Cost band|Actions per second| +|----------------------------|-----------------|---------|------------------| +|0 to 299|$23.25|$0 - $7,500|~115| +|300 to 1,499|$18.80|$5,640 - $28,200|~570| +|1,500 to 7,499|$14.10|$21,150 - $105,750|~2,860| +|7,500 to 29,999|$10.50|$78,750 - $315,000|~11,400| +|30,000 to 149,999|$7.90|$237,000 - $1,185,000|~57,000| +|> 150,000|$5.90|$885,000 ->|–| The following operations result in Actions: @@ -82,14 +99,21 @@ The following operations result in Actions: **Child Workflows** -- The parent Workflow spawning a Child Workflow results in one Action. -- Execution of the Child Workflow results in one Action. +- **Child Workflows spawned.** + The parent Workflow spawning a Child Workflow results in one Action. +- **Child Workflows executed.** + Execution of the Child Workflow results in one Action. **Activities** - **Activity started or retried.** Occurs each time an Activity is started or retried. -- **Local Activity started.** Each [Local Activity](/activities#local-activity) associated with one Workflow Task will count as one Action. (Note: Each additional Workflow Task heartbeat after counts as an additional Action. Also, Local Activities retried following a Workflow Task heartbeat will count as one Action. +- **Local Activity started.** + All [Local Activities](/activities#local-activity) associated with one Workflow Task count as a single Action. + That's because Temporal Cloud counts all [RecordMarkers](https://docs.temporal.io/references/commands#recordmarker) from each Workflow Task as one action, and not _N_ actions. + Please note: + * Each additional Workflow Task heartbeat after counts as an additional Action. + * Local Activities retried following a Workflow Task heartbeat count as one Action (capped at 100 Actions). - **Activity Heartbeat recorded.** A Heartbeat call from Activity code counts as an Action only if it reaches the [Temporal Server](/clusters#temporal-server). Temporal SDKs throttle [Activity Heartbeats](/activities#activity-heartbeat). @@ -98,87 +122,136 @@ The following operations result in Actions: **Schedules** -[Schedules](/workflows#schedule) allows you to "schedule" a Workflow to start at a particular time. Each execution of a Schedule will accrue three actions: +[Schedules](/workflows#schedule) allows you to "schedule" a Workflow to start at a particular time. +Each execution of a Schedule will accrue three actions: -- **Schedule Start**. Will account for two actions -- **Workflow started**. One action to start the target workflow +- **Schedule Start**. + This accounts for two actions. +- **Workflow started**. + This is a single action to start the target workflow. **Export** -[Workflow History Export](/cloud/export) allows you to export closed Workflow Histories to a cloud storage sink of your choice. +[Workflow History Export](/cloud/export) enables you to export closed Workflow Histories to a cloud storage sink of your choice. -- **Workflow exported**. Each Workflow exported accrues a single action. +- **Workflow exported**. + Each Workflow exported accrues a single action. [Reach out to our team](https://pages.temporal.io/contact-us) to get more information or to help size your number of Actions. -## What are the Temporal Cloud storage prices? {#storage} +## Storage pricing {#storage} + +**What are the Temporal Cloud storage prices?** -An execution of a particular Workflow could exist for a few seconds, a day, month, or even forever. Temporal collects the Event History during this time and dispatches work when necessary. In this context, a Workflow Execution has only two states, open (active) or closed. +A particular Workflow's execution might exist for a few seconds, a day, month, or even forever. +Temporal collects the Workflow Execution's Event History during this time and dispatches work when necessary. +Under this context, a Workflow Execution has only two states, open (active) or closed. -Storage costs are measured in gigabyte-hours (GBh) and include charges for active Workflows, active storage, and the long-term, "retained" storage of Event Histories of closed Workflows. These are measured per Namespace. +Storage costs are measured in gigabyte-hours (GBh) and include charges for active Workflows, active storage, and the long-term, "retained" storage of Event Histories of closed Workflows. +These are measured per Namespace. -_Active storage_ is a measure of the amount of storage used to store active Workflows. +_Active storage_ measures the amount of storage used by active Workflows. -When the execution of a Workflow ends, Temporal Cloud stores Event History for a defined Retention Period, for historical use. This is _retained storage_. Typical uses include compliance, debugging, workload refresh, and business analytics. Both kinds of storage have fixed costs. +When the execution of a Workflow ends, Temporal Cloud stores Event History for a defined Retention Period, for historical use. +This is _retained storage_. +Typical uses include compliance, debugging, workload refresh, and business analytics. +Both kinds of storage have fixed costs. | **Storage** | **Cost per GBh (USD)** | | ----------- | ---------------------- | -| Retained | $0.00042 | -| Active | $0.042 | +| Retained | $0.00042 | +| Active | $0.042 | If you purchase Temporal Cloud credits (as [outlined earlier](#action)), active storage costs are tiered and measured in gigabyte-hours. | **Active Storage (Per GB Hour Per Namespace)** | **Cost (USD)** | | ---------------------------------------------- | -------------- | -| Up to 7,000 | $0.039 | -| Over 7,000 up to 30,000 | $0.031 | -| Over 30,000 up to 90,000 | $0.023 | -| Over 90,000 up to 400,000 | $0.018 | -| Over 400,000 up to 1,500,000 | $0.013 | -| Over 1,500,000 | $0.010 | +| Up to 7,000 | $0.039 | +| Over 7,000 up to 30,000 | $0.031 | +| Over 30,000 up to 90,000 | $0.023 | +| Over 90,000 up to 400,000 | $0.018 | +| Over 400,000 up to 1,500,000 | $0.013 | +| Over 1,500,000 | $0.010 | -## What kind of support do I get with Temporal Cloud? {#support} +## Multi-region Namespace pricing {#multi-region} -When signing up for Temporal Cloud, select a support plan—either Basic or Premium—and gain access to our support organization, which includes developer success engineers and solution architects. Our experts assist with a range of work streams, from Workflow design reviews to setting up observability, and from troubleshooting support to maintaining an agreed-upon set of SLAs. +For workloads with stringent high-availability requirements, Temporal Cloud provides multi-region Namespaces (MRNs), which add a failover capability. +Each multi-region Namespace automatically replicates Workflow Execution data and metadata to a standby region. +This keeps the standby region in sync, allowing for a near seamless failovers should regional outages occur.  -The Temporal Developer Success team brings a deep knowledge of how Temporal works and how you can optimize your deployment. This team is comprised of engineers who know Temporal inside and out, and continue to contribute to its development. +Multi-region Namespace costs align with the volume of your workloads. +After activation, the new MRN begins replicating data to the standby region. +Due to this replication, MRNs double your current action and storage costs. +To estimate costs for this deployment model, apply a 2x multiplier to your existing Actions and storage Namespace costs. -They also investigate your Workflows to optimize their environments and possibly reduce costs associated with Actions and storage. They ensure your instance is performance tuned and help with other ongoing maintenance, like upgrades of the Temporal software and maintenance of the platform. +When upgrading an existing Namespace, some points to consider: -If an issue occurs, the team provides support through our [support portal](/cloud/support#support-ticket), [community forum](https://community.temporal.io/), and (with Premium support) a dedicated Slack channel. Temporal offers two levels of support defined by their availability and SLAs. +* Temporal won't charge for historical Actions completed prior to upgrading to a multi-region Namespace. + Only ongoing (in-flight) and new Workflow Executions incur the \"2x\" charge, as these are actively replicated. +* Temporal charges for all Actions of existing (ongoing) and new Workflows from the point of adding a new region and provisioning multi-region Namespace service onward. +* Temporal charges for replicated storage of retained (historical), running (ongoing), and new Workflow Executions from the point of adding a new region and starting MRN service. -| | **Basic** | **Premium** | -| ----------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -| **Response times** | P0: 1 business hour
P1: 4 business hours
P2: 1 business day
P3: 2 business days | P0: 30 minutes
P1: 1 business hour
P2: 4 business hours
P3: 1 business day | -| **Pricing (per month)** | Greater of $200 or 10% of monthly usage | Greater of $2,000 or 10% of monthly usage | +Multi-region Namespace invoice charges will include: -**Business Hours**: 05:00-17:00 Mon-Fri US Pacific Time. -For P0 issues, coverage is 24×7. +* An Actions row, showing the Action count from the active region. +* A Replicated Actions row, showing the Action count from the standby region. -**Priority definitions** +The invoice follows a similar structure for Replicated Storage. +It breaks out storage and replicated storage into their own line items. -- **P0 - Critical** (Production impacted) - - The Temporal Cloud service is unavailable or degraded with a significant impact. -- **P1 - High** (Production issue) - - An issue related to production workloads running on the Temporal Cloud service, or a significant project is blocked. -- **P2 - Normal** (General issues) - - General Temporal Cloud service or other issues where there is no production impact, or a workaround exists to mitigate the impact. -- **P3 - Low** (General guidance) - - Questions or an issue with the Temporal Cloud service that is not impacting system availability or functionality. +:::tip Action counts for mid-month adoption +- If your multi-region Namespace adoption began mid-month, expect to see more Actions than Replicated Actions and more Storage than Replicated Storage on your invoice. -## What is the price for SSO and SAML? {#sso-and-saml} +::: + +## SSO and SAML pricing {#sso-and-saml} We offer single sign-on (SSO) integration using SAML at a monthly fixed fee based on the number of users registered in Temporal Cloud: | **Users** | **Cost per month** | | --------- | ------------------ | -| 0 to 25 | $200 | -| 26 to 50 | $300 | -| 51+ | $500 | +| 0 to 25 | $200 | +| 26 to 50 | $300 | +| 51+ | $500 | + + +## Services and Support {#support} + +**What kind of services and support do I get with Temporal Cloud?** -## How to estimate costs {#pricing-estimates} +Each Temporal Cloud receives access to expert services and support. +The Temporal Success team brings a deep knowledge of how Temporal works and how you can optimize your deployment. +Our team is made up of engineers who bring deep knowledge of Temporal, and help contribute to its development. -Temporal Cloud employs a consumption-based pricing model that's based on storage and execution, factors that vary from one Workflow to the next. You can estimate the cost of a specific Workflow by running it at a low volume and then using its storage and compute measurements to project your production-scale cost. Our team is happy to [help you estimate the cost](https://pages.temporal.io/contact-us) for your specific workloads. +The Success team provides troubleshooting and issue resolution as well as service work-streams like as design reviews, setting up observability, and worker tuning. +The team can help investigate your Workflows to optimize their environments and so you might reduce costs associated with Actions and storage. +They ensure your instance is performance-tuned and they help with other ongoing maintenance, like upgrades of the Temporal software and maintenance of the platform. + +Services and support costs are outlined in the table below. +A more complete description of each level can be found on our [Services and Support](https://docs.temporal.io/cloud/support) outline. + +| |Basic|Premium| +|---------------------|-----|-------| +| Pricing (per month) | Greater of $200 or 10% of monthly usage | Greater of $2,000 or 10% of monthly usage | + + +## Pricing Estimates + +Temporal Cloud uses a consumption-based pricing model based primarily on Actions and storage. +Each workload is different. +You can estimate the cost of a specific Workflow by running it at a low volume. +Use the resulting storage and compute measurements to project your production scale cost. + +The examples below provide general estimates based on workload size. +You can also use our calculator on the pricing page to build your estimate. +Our team is always happy to [help you estimate costs] (https://pages.temporal.io/contact-us) for your specific workloads and requirements. + +|Workload size|Cost (monthly)|Characteristics|Actions |Typical use cases| +|-------------|--------------|---------------|------------------|-----------------| +|Small|< $25.00|Modest / transient throughput|< 1M / month 
_(< 0.38 actions per second)_|General automation
Human dependent processes 
Data pipelines
Nightly batch processes | +|Medium|< $1K|Steady or burst throughput|< 40M / month 
_(< 15 actions per second)_ | Transaction & order systems
Infrastructure automation
Payment Processing
Batch processes | +|Large|< $10K|Sustained throughput or multiple use cases|< 400M / month 
_(< 150 actions per second)_ | Data processing / sync
Retail order system
KYC & fraud detection | +|Web Scale|$20K+|"Web scale" and / or numerous use cases |1B+ / month 
_(400+ actions per second)_ | Social media application
SaaS application service| ## Pricing FAQ {#pricing-faq} @@ -186,20 +259,14 @@ Temporal Cloud employs a consumption-based pricing model that's based on storage The Temporal Cloud service is consumption based. You pay only for what you need with no minimum. -Basic support has a minimum monthly fee of $200 per month. +Basic support has a minimum monthly fee of $200 per month. **How do I pay for Temporal Cloud?** Temporal sends a monthly bill based on your consumption. -You can pay this bill with a credit card, ACH, wire transfer, or Temporal Credits. +You can pay with a credit card, ACH, wire transfer, or Temporal Credits. **Can I purchase Temporal Cloud through my Amazon, Azure, or Google Cloud Platform marketplace?** You can purchase Temporal Cloud credits in the AWS Marketplace. -To learn more about our private offer on the AWS Marketplace, contact our team at sales@temporal.io. - -**Do you offer any discounts for startups?** - -Yes. -If you're part of a startup that has raised $30 million or less in funding and is not a current Temporal Cloud customer, you can apply for free Temporal Cloud credits. -To learn more and apply, see [Temporal Cloud for Startups](https://temporal.io/startup). +To learn more about our private AWS Marketplace offer, contact our team at sales@temporal.io. diff --git a/docs/production-deployment/cloud/introduction/sla.mdx b/docs/production-deployment/cloud/introduction/sla.mdx index 635e07b2c7..ea493d9f71 100644 --- a/docs/production-deployment/cloud/introduction/sla.mdx +++ b/docs/production-deployment/cloud/introduction/sla.mdx @@ -2,7 +2,7 @@ id: sla title: Service Level Agreement (SLA) - Temporal Cloud sidebar_label: SLA -description: Learn more about Temporal Cloud's Sevice Level Agreement (SLA). +description: Learn more about Temporal Cloud's Service Level Agreement (SLA). slug: /cloud/sla toc_max_heading_level: 4 keywords: @@ -17,15 +17,21 @@ tags: **What is Temporal Cloud's Service Level Agreement? SLA?** -Temporal Cloud provides 99.99% availability, and its service level agreement (SLA) has a 99.9% guarantee against service errors. +Temporal Cloud provides two availability levels: the [service availability](https://en.wikipedia.org/wiki/Reliability,_availability_and_serviceability) and the contractual [service level agreement](https://en.wikipedia.org/wiki/Service-level_agreement) (SLA). +These levels are set by your deployment mode: -To calculate the service-error rate, we capture all requests that arrive in a Namespace during a five-minute interval and record the number of gRPC service errors that occurred. -For each Namespace, we calculate the service-error rate as `1 - (count of errors / count of requests)`. -Rates are averaged per month and reset quarterly. +* **Temporal Cloud with standard single-region deployment**: + Standard Temporal Cloud deployment provides 99.99% availability and a contractual service level agreement (SLA) of 99.9% guarantee against service errors. +* **Temporal Cloud with multi-region Namespace deployment**: + Temporal Cloud Namespaces that use the multi-region feature provide 99.99% availability and contractual service level agreement (SLA) of 99.99% guarantee against service errors. -Errors that are recorded against the SLA are service errors, such as the `UNAVAILABLE` [gRPC status code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). +To calculate the service-error rate, Temporal Cloud captures all requests that arrive in a Namespace during a five-minute interval. +We record the number of gRPC service errors that occurred. +For each Namespace, we calculate the service-error rate as 1 - (count of errors / count of requests). +Rates are averaged per month and reset quarterly. -Errors that aren't counted against the SLA include the following: +Errors are recorded against the SLA are service errors, such as the `UNAVAILABLE` [gRPC status code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). +The following errors are _not_ counted against the SLA: - `ClientVersionNotSupported` - `InvalidArgument` @@ -37,16 +43,18 @@ Errors that aren't counted against the SLA include the following: - `PermissionDenied` - `QueryFailed` - `RetryReplication` -- `ShardOwnershipLost` - `StickyWorkerUnavailable` - `TaskAlreadyStarted` - `Throttling (resources exhausted; triggers retry)` - `WorkflowExecutionAlreadyStarted` - `WorkflowNotReady` -Our internal alerting system is based on our service level objectives (SLO) for all errors, not just errors that count against the SLA. +Our internal alerting system is based on a [service level objective](https://en.wikipedia.org/wiki/Service-level_objective) (SLO) for all errors, not just errors that count against the SLA. When we receive an alert that an SLO is not being met, we page our on-call engineers, which often means that issues are resolved before they become noticeable. Internally, our components are distributed across a minimum of three availability zones per region. +We implement a cell architecture. +Each cell contains the software and services necessary to host a Namespace. +Within each cell, the components are distributed across a minimum of three availability zones per region. For current system status and information about recent incidents, see [Temporal Status](https://status.temporal.io). diff --git a/docs/production-deployment/cloud/metrics/index.mdx b/docs/production-deployment/cloud/metrics/index.mdx index 28038a4353..c8e894c789 100644 --- a/docs/production-deployment/cloud/metrics/index.mdx +++ b/docs/production-deployment/cloud/metrics/index.mdx @@ -30,7 +30,7 @@ tags: Beyond the [metrics](/references/sdk-metrics) provided by the Temporal SDKs, some key metrics exist only in Temporal Cloud. You can use your own observability tool to query an endpoint and review Namespace metrics. -Metrics for all Namespaces in your account are available from the metrics endpoint. +Metrics for all Namespaces in your account are available from your metrics endpoint. Metrics lag real-time performance by about one minute. @@ -39,7 +39,7 @@ Temporal Cloud retains raw metrics for seven days. To ensure security of your metrics, a CA certificate dedicated to observability is required. Only clients that use certificates signed by that CA, or that chain up to the CA, can query the metrics endpoint. For more information about CA certificates in Temporal Cloud, see [Certificate requirements](/cloud/certificates#certificate-requirements). - + - [General setup](/production-deployment/cloud/metrics/general-setup) - [Available metrics](/production-deployment/cloud/metrics/reference) - [Prometheus & Grafana setup](/cloud/metrics/prometheus-grafana) diff --git a/docs/production-deployment/cloud/metrics/reference.mdx b/docs/production-deployment/cloud/metrics/reference.mdx index bc985a96d2..ad8d38d8d1 100644 --- a/docs/production-deployment/cloud/metrics/reference.mdx +++ b/docs/production-deployment/cloud/metrics/reference.mdx @@ -94,6 +94,18 @@ Tasks that are successfully sync matched to a poller. When no tasks are available for a poller before timing out. +### temporal_cloud_v0_replication_lag_bucket + +A histogram of [replication lag](/cloud/multi-region#replication-lag) during a specific time interval for a [multi-region Namespace](/cloud/multi-region). + +### temporal_cloud_v0_replication_lag_count + +The [replication lag](/cloud/multi-region#replication-lag) count during a specific time interval for a [multi-region Namespace](/cloud/multi-region). + +### temporal_cloud_v0_replication_lag_sum + +The sum of [replication lag](/cloud/multi-region#replication-lag) during a specific time interval for a [multi-region Namespace](/cloud/multi-region). + ### temporal_cloud_v0_resource_exhausted_error_count gRPC requests received that were rate-limited by Temporal Cloud, aggregated by cause. diff --git a/docs/production-deployment/cloud/multi-region.mdx b/docs/production-deployment/cloud/multi-region.mdx new file mode 100644 index 0000000000..d612ab6485 --- /dev/null +++ b/docs/production-deployment/cloud/multi-region.mdx @@ -0,0 +1,512 @@ +--- +id: multi-region +slug: /cloud/multi-region +title: Multi-region Namespace - Temporal Cloud feature guide +sidebar_label: Multi-region Namespaces +tags: + - multi-region + - multi-region namespace + - high-availability + - failover + - availability + - namespaces +--- + +:::tip Support, stability, and dependency info +- Multi-region Namespaces are in [Public Preview](/evaluate/release-stages#public-preview) for Temporal Cloud. + +::: + +Temporal Cloud's multi-region Namespaces offer disaster-tolerant deployment for workloads with high availability requirements. +With the multi-region feature enabled, Temporal Cloud automates failover and synchronizes data between Namespaces in different regions. +You benefit from easy deployment and operational continuity to ensure data integrity, even in the face of unexpected challenges such as regional disruptions. + +For each multi-region Namespace, you designate a standby region in addition to your primary active region. +The Temporal Cloud Service replicates data from the active region to the standby region. +This means your standby region is always available and synchronized with the active region. +In the event of network service and performance issues in the active region, your standby region is ready to take over. +When necessary, Temporal Cloud smoothly transitions control from the active to the standby region using a process called "failover". + +Temporal Cloud's multi-region Namespace feature includes a 99.99% contractual Service Level Agreement ([SLA](https://docs.temporal.io/cloud/sla)). +It provides 99.99% availability and 99.99% guarantee against service errors. + +This feature guide covers the following topics: + +- [Choosing multi-region](/cloud/multi-region#multi-region-intro) +- [Failovers](/cloud/multi-region#failovers) +- [Multi-region Namespace SLA](/cloud/multi-region#sla) +- [Architecture](/cloud/multi-region#architecture) +- [Pricing](/cloud/multi-region#pricing) +- [Operations](/cloud/multi-region#operations) + +## Choosing multi-region {#multi-region-intro} + +**Why choose a multi-region Namespace?** + +A high-availability multi-region Namespace (MRN) creates a single logical Namespace that operates across two physical regions: one active and one standby. +MRNs streamline access for both regions to a unified Namespace endpoint. +As Workflows progress in the active region, history events asynchronously replicate to the standby region, ensuring continuity and data integrity. + +In the event of an incident or outage in the active region, Temporal Cloud will seamlessly failover to your standby region. +After failover, the roles of the active and standby regions switch. +Failovers enable already existing Workflow Executions to keep running and new Workflow Executions to be started. + +In traditional active/active replication, multiple nodes serve requests and accept writes simultaneously with strong synchronous data consistency. +With a Temporal Cloud multi-region Namespace, two replicated Namespaces (one active, one standby) accept requests, but not simultaneously. +Requests are processed and Workflow history events are initially written to the active region, then asynchronously replicated to the standby region. + + +Before failover | After failover +:-------------------------:|:-------------------------: +![Before failover](/img/multi-region/before-failover.png) | ![After failover](/img/multi-region/after-failover.png) + +Should you be using multi-region Namespaces? +It depends on your availability requirements: + +* Multi-region Namespaces offer a 99.99% contractual SLA for workloads with strict high-availability requirements. + Multi-region Namespaces (MRNs) use two Namespaces in two deployment regions, to support standby recovery. + In the event of a regional failure, Temporal Cloud automatically fails over the MRN Namespace to the standby region. +* Single-region Namespaces offer a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)). + In single-region use, Temporal clients connect to a single Namespace in one deployment region. + For many applications, this provides sufficient availability. + +Temporal Cloud provides 99.99% service availability for all Namespaces, both single-region and multi-region. + +**Upgrading to multi-region:** + +Enable the multi-region Namespace feature for your existing single-region Namespace by adding a second region to your Namespace. +Once provisioned, Temporal Cloud automatically begins data replication for your new standby region. + +**Advantages of using a multi-region Namespace:** + +* No manual deployment or configuration needed, just simple push-button operation. +* Open Workflows continue in the standby region with minimal interruption and data loss. +* No changes needed for Worker and Workflow code during setup or failover. +* 99.99% Contractual SLA. + +## Failovers {#failovers} + +**What happens during the failover process?** + +Consider an incident or outage that elevates error rates or latency in the active region of a multi-region Namespace. +Under these circumstances, Temporal Cloud will initiate a Namespace failover. +This operation shifts Workflow processing to a standby region, unaffected by the incident. +This ensures that existing Workflows can progress and new Workflows can be created as the incident is addressed. + +:::info +- You can test your multi-region Namespace by manually triggering failovers using the UI page or the 'tcld' CLI utility. + In most scenarios, we recommend you let Temporal handle failovers for you. + +::: + +### Replication lag {#replication-lag} + +Multi-region Namespaces use asynchronous replication between regions. +Workflow updates in the active region, along with associated history events, are transmitted to the standby region with a short delay. +This delay is called the replication lag. +Temporal Cloud strives to maintain a P95 replication delay of less than 1 minute. +In this context, P95 means 95% of requests are processed faster than this specified limit. + +Replication lags mean a [forced failover](/cloud/multi-region#forced-failover) may cause Workflows to rollback in progress. +Lags may also cause recently started Workflows to be temporarily unavailable until the active region recovers. +Temporal event versioning and [conflict resolution mechanisms](/cloud/multi-region#conflict-resolution) help guarantee that the Workflow Event History can be replayed. +Critical operations like Signals won't get lost. + +### Failover scenarios + +The Temporal Cloud failover mechanism supports several modes to execute Namespace failovers. +These modes include graceful failover ("handover"), forced failover, and a hybrid mode. +The hybrid mode is Temporal Cloud’s default Namespace behavior. + +#### Graceful failover (handover) {#graceful-failover} + +In this mode, replication tasks are fully processed and drained. +Temporal Cloud pauses traffic to the Namespace before the failover. +This prevents the loss of progress and avoids data conflicts. +The Namespace experiences a short period of unavailability, defaulting to 10 seconds. + +During this period, existing Workflows stop progress. +State transitions will not happen and tasks are not dispatched. +User requests like start/signal workflow will be rejected while operations are paused during handover. + +This mode favors _consistency_ over availability. + +#### Forced failover {#forced-failover} + +In this mode, a Namespace immediately activates in the standby region. +Events not yet replicated due to [replication lag](/cloud/multi-region#replication-lag) will undergo [conflict resolution](/cloud/multi-region#conflict-resolution) upon reaching the new active region. + +This mode prioritizes _availability_ over consistency. + +#### Hybrid failover mode {#hybrid-failover} + +While graceful failovers are preferred for consistency, they aren’t always practical. +Temporal Cloud’s hybrid failover mode (the default mode) limits an initial graceful failover attempt to 10 seconds or less. +If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. +This strategy balances consistency and availability requirements. + +See the sections on [triggering a failover](/cloud/multi-region#triggering-failovers), [Worker deployment](/cloud/multi-region#worker-deployment), and [routing](/cloud/multi-region#routing) for more information. + +## Multi-region Namespace SLA {#sla} + +**What guarantees does Temporal offer for multi-region Namespaces?** + +Multi-region Namespaces offer 99.99% availability, enforced by Temporal Cloud's [service error rates SLA](https://docs.temporal.io/cloud/sla). +Our system is designed to limit data loss after recovery when the incident triggering the failover is resolved. + +Our recovery point objective ([RPO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective)) is near-zero. +There may be a short period of time during the incident, the [replication lag](/cloud/multi-region#replication-lag), when some data is unavailable. +When the active region is unable to reach the standby region, that data will be unavailable until the network issue is fixed and data has finished replicating. + +We've designed Temporal Cloud to proactively respond to incidents by triggering failovers. +Our recovery time objective ([RTO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective)) is 20 minutes or less per incident. + +:::info +- During a disaster scenario in which the data on the hard drives in the active region cannot be recovered, the duration of data loss may be as high as the replication lag at the time of disaster. + +::: + +### Regional availability {#regional-availability} + +Multi-region Namespaces are available in all existing [Temporal Cloud regions](/cloud/service-availability#regions). + +:::tip +- Namespace pairing is currently limited to regions within the same continent. +South America is excluded as only one region is available. +::: + +## Architecture {#architecture} + +**How do multi-region Namespaces work?** + +Multi-region Namespaces replicate Namespace metadata and Workflow Executions across connected regions. +This redundancy, plus the added failover capability, provides measurable stability when dealing with outages. + +A multi-region Namespace is normally active in a single region at any moment. +The passive region assumes a standby role. +An exception to this only occurs in the event of a network partition. +In this case, you may elect to promote a standby region to active status. +Caution: this action will temporarily result in both regions being active. +Once the network partition resolves and communication between the regions is restored, a conflict resolution algorithm determines which region continues as the active one. +This ensures only one region remains active. + +### Metadata replication {#metadata-replication} + +Updates to multi-region Namespace records automatically replicate across regions. +This metadata includes configurations such as retention periods, Search Attributes, and other settings. +Temporal Cloud ensures that all regions will eventually share a consistent and unified view of the Namespace metadata. + +:::info +- A Namespace failover, which changes the "active region" field of a Namespace record, is an update. + This update is replicated via the Namespace metadata mechanism. + +::: + +### Workflow Execution replication {#workflow-execution-replication} + +Temporal Cloud restricts certain Workflow operations to the active region: + +* You may only update Workflows in the active region. +* You may only dispatch Workflow Tasks and Activity Tasks from the active region. Forward progress in a Workflow Execution can therefore only be made in the active region. + +These limits mean that certain requests, such as Start Workflow and Signal Workflow, are processed by and limited to the active region. +Standby regions may receive API requests from Clients and Workers. +They automatically forward these requests to the active Namespace for execution. + +Multi-region Namespaces provide an “all-active” experience for Temporal users. +This helps limit or eliminate downtime during Namespace failover. +There's a short time window from when a standby region becomes the active region to when Clients and Workers receive a DNS update. +During this time requests forward from the now passive (formerly active) region to the newly active (formerly standby) region. + +As Workflow Executions progress and are operated on, replication tasks created in the active region are dispatched to the standby region. +Processing these replication tasks ensures that the standby region undergoes the same state transitions as the active region. +This enables replicated tasks to synchronize and achieve the same state as the original tasks. + +Standby regions do not distribute Workflow or Activity Tasks. +Instead, they perform verification tasks to confirm that intended operations are executed so Workflows reach the desired state. +This mechanism ensures consistency and reliability in the replication process across Temporal regions. + +### Conflict Resolution {#conflict-resolution} + +Multi-region Namespaces rely on asynchronous event replication across Temporal regions. +In the event of a non-graceful failover, replication lag may result in a temporary setback in workflow progress. + +Single-region Namespaces provide _at-most-once_ semantics for Activities execution (when [Maximum Attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0). +Multi-region Namespaces provide _at-least-once_ semantics for execution of Activities. +Completed Activities _may_ be re-dispatched in a newly active region, leading to repeated executions. + +When a Workflow Execution is updated in a new region following a failover, events from the previously active region that arrive after the failover can't be directly applied. +At this point, Temporal Cloud has forked the Workflow History. + +After failover, Temporal Cloud creates a new branch history for execution, and begins its conflict resolution process. +The Temporal Service ensures that Workflow Histories remain valid and are replayable by SDKs post-failover or after conflict resolution. +This capability is crucial for Workflow Executions to continue their forward progress. + +## Pricing {#pricing} + +**How does adding a multi-region Namespace affect my costs?** + +For pricing details, please visit Temporal Cloud's [Pricing page](/cloud/pricing). + +## Manage your multi-region Namespace {#management} + +**How do you create, enable, and manage your multi-region Namespace?** + +Temporal enables you to create and manage your multi-region Namespace using the Temporal Cloud Web UI, the command line 'tcld' CLI utility, and the [Cloud Ops API](/ops). +Use these tools to create, upgrade, and discontinue your multi-region Namespace. + +* [Create a multi-region Namespace](/cloud/multi-region#create) +* [Upgrade a single-region Namespace to multi-region](/cloud/multi-region#add-regions) +* [Discontinuing multi-region service](/cloud/multi-region#discontinuing) + +:::warning +- Only [Global Admins](https://docs.temporal.io/cloud/users#account-level-roles) and [Namespace Admins](https://docs.temporal.io/cloud/users#namespace-level-permissions) may create an multi-region Namespace (MRN), upgrade an existing namespace to MRN, or trigger an MRN failover. + +::: + +### Create a multi-region Namespace {#create} + +The following sections explain how to create a new multi-region Namespace (MRN). +MRNs provide multi-region deployment backed by Temporal's data replication and active-standby features. + +:::tip +- While reading through this coverage, remember that pairing is currently limited to regions within the same continent. + +::: + +#### Temporal Cloud Web UI + +During Namespace creation, specify the first region for the Namespace. +Then, select the “Add a region” option. +Adding a second region enables multi-region Namespace capabilities. + +#### Temporal 'tcld' CLI + +Start with the following command to create the new multi-region Namespace: + +``` +tcld namespace create --namespace --region --region +``` + +Include both regions by specifying the [region codes](/cloud/service-availability) as arguments to the `--region` flags. +Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. + +### Upgrade an existing single-region Namespace to multi-region functionality {#add-regions} + +You can upgrade existing single-region Namespaces to a multi-region by adding a standby region. +The following sections show you how. + +#### Temporal Cloud Web UI + +To upgrade an existing Namespace to a multi-region Namespace: + +1. Visit Temporal Cloud [Namespaces](https://cloud.temporal.io/namespaces) in your Web browser +1. Navigate to the Namespace details page +1. Select the “Add a region” button. +1. Select the standby region you want to add to this Namespace + +You will see an estimated time for replication. +This time is based on your selection and the size and scale of Workflows in your Namespace, +An email alert is sent once your multi-region Namespace is ready for use. + +#### Temporal 'tcld' CLI + +At the command line, enter: + +``` +tcld namespace add-region --namespace --region +``` + +Specify the region code for the new region to add. +Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. +An email alert is sent once your multi-region Namespace is ready for use. + + +### Discontinuing multi-region availability {#discontinuing} + +Disabling multi-region removes the high availability and automatic failover features that provide Temporal's highest service level agreement. +To disable the feature and end charges, users must contact [Temporal Support](https://support.temporal.io) directly. +MRN-specific charges for replication will stop once this decommissioning procedure completes. + +* When making your request you must let us know which region you want the Namespace to land in after removing the standby region. +* If you cease services in the middle of the month, your Namespace will be converted to a single region Namespace within 1 business day. +* Temporal won't retain replicated data in the standby region once multi-region has been disabled. +* After disabling multi-region, Temporal Cloud cannot re-enable the feature for a given Namespace for seven days. + +## Operations {#operations} + +**How do you trigger failovers and observe Workflow Executions?** + +This section provides how-to instructions for the following operations tasks: + +* [Triggering failovers](/cloud/multi-region#triggering-failovers) +* [Metrics](/cloud/multi-region#metrics-operations) +* [Monitoring and observability](/cloud/multi-region#observability) + +### Triggering failovers {#triggering-failovers} + +Temporal is responsible for detecting and triggering a failover for a multi-region namespace in the event of an outage or a disaster. +Users also can trigger a failover for testing purposes. + +The following sections show you how to perform this action, moving your active Namespace to the standby region. +You'll also read about what to expect after a failover. + +:::warning Check Your Replication Lag +- Always check the [metric replication lag](https://temporal-documentation-clljyhxqv.preview.thundergun.io/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket) before initiating a failover. + A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. + +::: + +#### Temporal Cloud Web UI + +Visit the [Namespace Web UI page](https://cloud.temporal.io/namespaces). +Navigate to the Namespace details page and select the menu option to “Trigger a failover”. +After confirming the operation, the failover is initiated. + +#### Temporal 'tcld' CLI + +To manually trigger a failover, issue the following from your command line: + +``` +tcld namespace failover --namespace --region +``` + +Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. + +#### After failover + +After any failover, whether triggered by you or by Temporal, event information appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. +The audit log entry for Failover uses the `"operation": FailoverNamespace"` event. +After failover, the namespace is active in the new region. + +You needn't monitor Temporal Cloud's failover response in real-time. +Whenever there is a failover event, you'll receive an alert email. + +More information is available in [Monitoring and Observability](/cloud/multi-region#observability). + +### Metrics {#metrics-operations} + +Replication lag refers to the transmission delay of Workflow updates and history events from the active region to the standby region. +A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress, so always check the metric replication lag before initiating a failover. +Temporal Cloud emits three replication lag-specific [metrics](https://temporal-documentation-clljyhxqv.preview.thundergun.io/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket). +The following samples demonstrate how you can use these metrics to explore replication lag. + +**P99 replication lag histogram** + +``` +histogram_quantile(0.99, sum(rate(temporal_cloud_v0_replication_lag_bucket[1m])) by (temporal_namespace, le)) +``` + +**Average replication lag** + +``` +sum(rate(temporal_cloud_v0_replication_lag_sum[1m])) by (temporal_namespace) +/ +sum(rate(temporal_cloud_v0_replication_lag_count[1m])) by (temporal_namespace) +``` + +### Monitoring and observability {#observability} + +You can view and alert on key cloud metrics using the Web UI, the 'tcld' CLI utility, and Temporal Cloud APIs. +For example, during the process of adding a region to a Namespace, you can see the progress of Workflow replication. +Errors--if any--will also surface in the Namespace Web UI. + +:::info +- You may notice that multi-region Namespace shows twice (2x) the Action count in `temporal_cloud_v0_total_action_count`. + This doubling happens due to regional replication. + +::: + +### Auditing operational events {#auditing} + +Temporal Cloud provides several ways to audit events: + +* When Temporal triggers failovers, the audit log updates with details. + Look specifically for `"operation": "FailoverNamespace"` in the logs. +* You can set alerts for Temporal-initiated failover events. +* After a failover, you can check that the Namespace is active in the new region using the Temporal Cloud Web UI. + +### Worker Deployment {#worker-deployment} + +Enabling the multi-region Namespace does not require specific Worker configuration. +The process is invisible to the Workers. +When a Namespace fails over to the standby region, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. +More details are available in the [Routing](/cloud/multi-region#routing) section below. + +:::info +- When a Namespace fails over to a standby region, Workers will be communicating cross-region. + +- In case of a complete regional outage, Workers in the original region may fail alongside the original Namespace. + To keep Workflows moving during this level of outage, deploy a second set of Workers to your standby region. + +::: + +### Routing {#routing} + +When using multi-region for a Namespace, the Namespace's DNS record `..` targets a regional DNS record in the format `.region.`. +In this format, `` is the currently active region for your namespace. +Clients resolving the Namespace’s DNS record are directed to connect to the active region for that Namespace, thanks to the regional DNS record. + +During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another. +Namespace DNS records are configured with a 15 seconds TTL. +Any DNS cache should re-resolve the record within this delay. As a rule of thumb, DNS reconciliation takes no longer than twice (2x) the TTL. +Clients should converge to the newly targeted region within, at, most a 30-second delay. + +#### PrivateLink routing {#privatelink-routing} + +:::important +- At failover, routing is completely transparent for multi-region customers _unless_ you use PrivateLink. + This section describes how to configure routing for multi-region Namespaces for PrivateLink customers only. + +::: + +PrivateLink customers may need to change certain configurations for multi-region Namespace use. +Routing configuration depends on networking setup and use of PrivateLink. +You may need to: + +* override a DNS zone; and +* ensure the network connectivity between the two regions. + +![Customer side solution example](/img/multi-region/private-link.png) + +When using PrivateLink, you connect to Temporal Cloud using IP addresses local to your network. +The `region.` zone is configured in the Temporal systems as an independent zone. +This allows you to override it to make sure traffic is routed internally for the regions in use. +You can check the Namespace's active region using the Namespace record CNAME, which is public. + +To set up the DNS override, you override specific regions to target the relevant IP addresses (e.g. aws-us-west-1.region.tmprl.cloud to target 192.168.1.2). +Using AWS, this can be done using a private hosted zone in Route53 for `region.`. +Link that private zone to the VPCs you use for Workers. + +When your Workers connect to the Namespace, they first resolve the ` ..` record. +This targets `.region.` using a CNAME. Your private zone overrides that second DNS resolution, leading traffic to reach the internal IP you're using. + +Consider how you'll configure Workers to run in this scenario. +You might set Workers to run in both regions at all times. +Alternately, you could establish connectivity between the regions to redirect Workers once failover occurs. + +The following table lists Temporal's available regions, PrivateLink endpoints, and DNS record overrides. +The `sa-east-1` region listed here is not yet available for use with multi-region Namespaces. + +| Region | PrivateLink VPC Endpoint | DNS Record Override | +|---|---|---| +| `ap-northeast-1` | `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` | `aws-ap-northeast-1.region.tmprl.cloud` | +| `ap-northeast-2` | `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` | `aws-ap-northeast-2.region.tmprl.cloud` | +| `ap-south-1` | `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` | `aws-ap-south-1.region.tmprl.cloud` | +| `ap-southeast-1` | `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` | `aws-ap-southeast-1.region.tmprl.cloud` | +| `ap-southeast-2` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` | `aws-ap-southeast-2.region.tmprl.cloud` | +| `ca-central-1` | `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` | `aws-ca-central-1.region.tmprl.cloud` | +| `eu-central-1` | `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` | `aws-eu-central-1.region.tmprl.cloud` | +| `eu-west-1` | `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` | `aws-eu-west-1.region.tmprl.cloud` | +| `eu-west-2` | `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` | `aws-eu-west-2.region.tmprl.cloud` | +| `sa-east-1` | `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` | `aws-sa-east-1.region.tmprl.cloud` | +| `us-east-1` | `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` | `aws-us-east-1.region.tmprl.cloud` | +| `us-east-1` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b06bbbd81fd41222` | `aws-us-east-1.region.tmprl.cloud` | +| `us-east-2` | `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` | `aws-us-east-2.region.tmprl.cloud` | +| `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0473f53028c65d8fb` | `aws-us-west-2.region.tmprl.cloud` | +| `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` | `aws-us-west-2.region.tmprl.cloud` | + +:::tip Learn more about multi-region Namespaces +- If you have more questions or feedback about this feature, please reach out to the product team. + +::: diff --git a/docs/production-deployment/cloud/tcld/namespace.mdx b/docs/production-deployment/cloud/tcld/namespace.mdx index 6a162e2f5e..7108da1d16 100644 --- a/docs/production-deployment/cloud/tcld/namespace.mdx +++ b/docs/production-deployment/cloud/tcld/namespace.mdx @@ -17,8 +17,10 @@ The `tcld namespace` commands enable [Namespace](/namespaces) operations in Temp Alias: `n` +- [tcld namespace add-region](#add-region) - [tcld namespace create](#create) - [tcld namespace delete](#delete) +- [tcld namespace failover](#failover) - [tcld namespace get](#get) - [tcld namespace list](#list) - [tcld namespace export](#export) @@ -28,6 +30,57 @@ Alias: `n` - [tcld namespace retention](#retention) - [tcld namespace update-codec-server](#update-codec-server) +## add-region + +Use `tcld namespace add-region` to add a standby region to an existing Temporal Cloud [Namespace](/namespaces). +Enrolling a second region upgrades the Namespace to [multi-region](/cloud/multi-region). +Once provisioned, the multi-region Namespace enables Temporal Cloud to start replicating Workflow Execution data from the active to the standby region and trigger failover during adverse conditions. + +`tcld namespace add-region` + +Alias: _none_ + +:::caution + +To end multi-region service and discontinue replication charges, contact [Temporal Support](https://support.temporal.io) directly. +You can't use `tcld` to remove a region from a Namespace. + +::: + +The following modifiers control the behavior of the command. + +#### --ca-certificate + +_Required modifier unless `--ca-certificate-file` is specified_ + +A base64-encoded CA certificate. + +If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used. + +Alias: `-c` + +#### --namespace + +Specify a Namespace hosted on Temporal Cloud. +If not specified, the value of the environment variable `$TEMPORAL_CLOUD_NAMESPACE` is used. + +Alias: `-n` + +_Required modifier_ + +#### --region + +The region to add to the existing Namespace. + +Valid options: `ap-northeast-1` | `ap-northeast-2` | `ap-south-1` | `ap-southeast-1` | `ap-southeast-2` | `eu-central-1` | `eu-west-1` | `eu-west-2` | `ca-central-1` | `us-east-1` | `us-east-2` | `us-west-2` + +See [Service Availability](/cloud/service-availability). +The `sa-east-1` region is not supported at this time. + +Alias: `--re` + +_Required modifier_ + ## create The `tcld namespace create` command creates a Temporal [Namespace](/namespaces) in Temporal Cloud. @@ -57,22 +110,36 @@ Alias: `-c` #### --namespace -_Required modifier_ - -Specify the name of the Namespace to create. +Specify a Namespace hosted on Temporal Cloud. +If not specified, the value of the environment variable $TEMPORAL_CLOUD_NAMESPACE is used. Alias: `-n` -#### --region - _Required modifier_ +#### --region + The region to create the Namespace in. +When one `--region` flag is used, `tcld` provisions a single-region namespace. +In single-region use, Temporal clients connect to a single Namespace in one deployment region. + +When two `--region` flags are provided, your Namespace is provisioned to use Temporal Cloud [multi-region Namespace](/cloud/multi-region) (MRN) service. +Once provisioned, MRN enables Temporal Cloud to start replicating Workflow Execution data from the active to the standby region and trigger failover during adverse conditions. + Valid options: `ap-northeast-1` | `ap-southeast-1` | `ap-southeast-2` | `ca-central-1` | `eu-central-1` | `eu-west-1` | `eu-west-2` | `us-east-1` | `us-west-2` Alias: `--re` +_Required modifier_ + +:::caution + +To end multi-region service and end charges, contact [Temporal Support](https://support.temporal.io) directly. +You can't use `tcld` to remove a region from a namespace. + +::: + #### --retention-days The number of days that data about closed Workflow Executions will be retained (default: 30). @@ -171,12 +238,12 @@ The following modifiers control the behavior of the command. #### --namespace -_Required modifier_ - Specify the Namespace hosted on Temporal Cloud to be deleted. Alias: `-n` +_Required modifier_ + #### --request-id The request identifier to use for the asynchronous operation. @@ -197,6 +264,56 @@ Alias: `-v` tcld namespace delete --namespace ``` +## failover + +Failover a Temporal Namespace. +A failover switches a Namespace region from the active region to the standby region. + +#### --request-id + +Specify a request identifier to use for the asynchronous operation. +If not specified, the server assigns a request identifier. + +Alias: `-r` + +#### --namespace + +Specify a Namespace hosted on Temporal Cloud. +If not specified, the value of the environment variable $TEMPORAL_CLOUD_NAMESPACE is used. + +Alias: `-n` + +_Required modifier_ + +#### --region + +The region to failover _to_. + +Valid options: `ap-northeast-1` | `ap-northeast-2` | `ap-south-1` | `ap-southeast-1` | `ap-southeast-2` | `eu-central-1` | `eu-west-1` | `eu-west-2` | `ca-central-1` | `us-east-1` | `us-east-2` | `us-west-2` + +See [Service Availability](/cloud/service-availability). +The `sa-east-1` region is not supported at this time. + +Alias: `--re` + +_Required modifier_ + +#### --ca-certificate + +_Required modifier unless `--ca-certificate-file` is specified_. + +A base64-encoded CA certificate. + +If both `--ca-certificate` and `--ca-certificate-file` are specified, only `--ca-certificate` is used. + +Alias: `-c` + +#### --cloud-provider + +The cloud provider of the region to failover to. + +Default: aws (default: "aws") + ## get The `tcld namespace get` command gets information about the specified [Namespace](/namespaces) in Temporal Cloud. diff --git a/sidebars.js b/sidebars.js index e0f5379082..e470d69f22 100644 --- a/sidebars.js +++ b/sidebars.js @@ -26,6 +26,37 @@ module.exports = { "evaluate/development-features/throughput-composability", ], }, + { + type: "category", + label: "Production features", + collapsed: true, + link: { + type: "doc", + id: "evaluate/production-features/index", + }, + items: [ + { + type: "category", + label: "Temporal Cloud", + collapsed: true, + link: { + type: "doc", + id: "evaluate/production-features/temporal-cloud/index", + }, + items: ["evaluate/production-features/temporal-cloud/high-availability"], + }, + { + type: "category", + label: "Self-hosted", + collapsed: true, + link: { + type: "doc", + id: "evaluate/production-features/self-hosted/index", + }, + items: ["evaluate/production-features/self-hosted/high-availability"], + }, + ], + }, "security", "evaluate/release-stages", { @@ -250,6 +281,7 @@ module.exports = { "production-deployment/cloud/metrics/prometheus-grafana", ], }, + "production-deployment/cloud/multi-region", "production-deployment/cloud/worker-health", "production-deployment/cloud/saml", "production-deployment/cloud/operation-api", diff --git a/static/img/multi-region/active-active.png b/static/img/multi-region/active-active.png new file mode 100644 index 0000000000..01c3186ce1 Binary files /dev/null and b/static/img/multi-region/active-active.png differ diff --git a/static/img/multi-region/active-passive.png b/static/img/multi-region/active-passive.png new file mode 100644 index 0000000000..52b8ece6ee Binary files /dev/null and b/static/img/multi-region/active-passive.png differ diff --git a/static/img/multi-region/after-failover.png b/static/img/multi-region/after-failover.png new file mode 100644 index 0000000000..078fec102e Binary files /dev/null and b/static/img/multi-region/after-failover.png differ diff --git a/static/img/multi-region/before-failover.png b/static/img/multi-region/before-failover.png new file mode 100644 index 0000000000..ca49e5e256 Binary files /dev/null and b/static/img/multi-region/before-failover.png differ diff --git a/static/img/multi-region/cell-with-multiple-namespaces.png b/static/img/multi-region/cell-with-multiple-namespaces.png new file mode 100644 index 0000000000..e2b0a0c74f Binary files /dev/null and b/static/img/multi-region/cell-with-multiple-namespaces.png differ diff --git a/static/img/multi-region/customer-environment.png b/static/img/multi-region/customer-environment.png new file mode 100644 index 0000000000..73991465d8 Binary files /dev/null and b/static/img/multi-region/customer-environment.png differ diff --git a/static/img/multi-region/customer-side-example.png b/static/img/multi-region/customer-side-example.png new file mode 100644 index 0000000000..de3104ca9a Binary files /dev/null and b/static/img/multi-region/customer-side-example.png differ diff --git a/static/img/multi-region/failover-to-secondary-region.png b/static/img/multi-region/failover-to-secondary-region.png new file mode 100644 index 0000000000..c4b851263d Binary files /dev/null and b/static/img/multi-region/failover-to-secondary-region.png differ diff --git a/static/img/multi-region/private-link.png b/static/img/multi-region/private-link.png new file mode 100644 index 0000000000..de3104ca9a Binary files /dev/null and b/static/img/multi-region/private-link.png differ diff --git a/static/img/multi-region/single-region.png b/static/img/multi-region/single-region.png new file mode 100644 index 0000000000..d142eec119 Binary files /dev/null and b/static/img/multi-region/single-region.png differ