Skip to content

Commit

Permalink
Deploy preview for PR #42
Browse files Browse the repository at this point in the history
  • Loading branch information
@actions-user
actions-user committed Jan 2, 2025
1 parent 2c38a9d commit 34b16b2
Show file tree
Hide file tree
Showing 8 changed files with 639 additions and 294 deletions.
69 changes: 25 additions & 44 deletions pr-previews/pr-42/deployment/time_series_api_limits/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -504,6 +504,15 @@
</span>
</a>

</li>

<li class="md-nav__item">
<a href="#tips" class="md-nav__link">
<span class="md-ellipsis">
Tips
</span>
</a>

</li>

<li class="md-nav__item">
Expand Down Expand Up @@ -570,15 +579,6 @@
</ul>
</nav>

</li>

<li class="md-nav__item">
<a href="#tips" class="md-nav__link">
<span class="md-ellipsis">
Tips
</span>
</a>

</li>

</ul>
Expand Down Expand Up @@ -670,6 +670,15 @@
</span>
</a>

</li>

<li class="md-nav__item">
<a href="#tips" class="md-nav__link">
<span class="md-ellipsis">
Tips
</span>
</a>

</li>

<li class="md-nav__item">
Expand Down Expand Up @@ -736,15 +745,6 @@
</ul>
</nav>

</li>

<li class="md-nav__item">
<a href="#tips" class="md-nav__link">
<span class="md-ellipsis">
Tips
</span>
</a>

</li>

</ul>
Expand Down Expand Up @@ -1207,21 +1207,26 @@
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput" data-mime-type="text/markdown">
<h1 id="time-series-api-limits">Time series API limits<a class="anchor-link" href="#time-series-api-limits">¶</a></h1><p>The <code>titiler-cmr</code> API can be deployed as a Lambda function in AWS. Since requests to the time series endpoints will make recursive requests to the Lambda function for the lower-level time step operations, there are some limits in place to avoid making large requests that are likely to overwhelm the API.</p>
<h1 id="time-series-api-limits">Time series API limits<a class="anchor-link" href="#time-series-api-limits">¶</a></h1><p>Last updated: 2025-01-02</p>
<p>The <code>titiler-cmr</code> API can be deployed as a Lambda function in AWS. Since requests to the time series endpoints will make recursive requests to the Lambda function for the lower-level time step operations, there are some limits in place to avoid making large requests that are likely to overwhelm the API.</p>
<h2 id="highlights">Highlights<a class="anchor-link" href="#highlights">¶</a></h2><ul>
<li>Maximum of 995 discrete points or intervals in a time series request (due to Lambda concurrency limits)</li>
<li>You can use the length of the time series, the AOI size, and the resolution of the dataset to calculate the number of total pixels (<code>x_pixels * y_pixels * n_time</code>) which is helpful for determining if a request will succeed</li>
<li>The <code>/timeseries/bbox</code> endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to <strong>100,000,000 (<code>1e8</code>) total pixels</strong>. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.</li>
<li>The <code>/timeseries/statistics</code> endpoint can handle larger requests than the <code>/timeseries/bbox</code> endpoint Based on a coarse evaluation of the API, requests are limited to <strong>15,000,000,000 (<code>1.5e10</code>) total pixels</strong> as long as the individual images read by the <code>/statistics</code> endpoint are smaller than 56,250,000 (<code>5.625e7</code>) pixels.</li>
</ul>
<h2 id="tips">Tips<a class="anchor-link" href="#tips">¶</a></h2><ul>
<li>If you hit an error because the total size of the request is too large, try reducing the temporal resolution of the time series, e.g. from daily (<code>P1D</code>) to weekly (<code>P7D</code>) or greater (<code>P10D</code>)</li>
<li>If you need higher temporal resolution but the full request is not able handle it, split the request into multiple smaller requests and merge the results yourself!</li>
</ul>
<h2 id="background">Background<a class="anchor-link" href="#background">¶</a></h2><p>The time series API provides rapid access to time series analysis and visualization of collections in the CMR catalog, but there are some limitations to the API deployment that require some care when making large requests.</p>
<p>There are several factors that must be considered in order to make a successful time series request:</p>
<ul>
<li>Spatial resolution of the dataset (especially for the xarray backend)</li>
<li>Request AOI size</li>
<li>Number of points/intervals in the time series</li>
</ul>
<p>These factors all influence the runtime and memory footprint of the initial <code>/timeseries</code> request and requests that are too large in any of those dimensions can result in an API failure. Here are a few guidelines to help you craft successful time series requests.</p>
<p>These factors all influence the runtime and memory footprint of the initial <code>/timeseries</code> request and requests that are too large in any of those dimensions can result in an API failure.</p>
</div>
</div>
</div>
Expand All @@ -1233,16 +1238,6 @@ <h2 id="background">Background<a class="anchor-link" href="#background">¶</a></
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput" data-mime-type="text/markdown">
<h2 id="details">Details<a class="anchor-link" href="#details">¶</a></h2><h3 id="number-of-pointsintervals-in-the-time-series">Number of points/intervals in the time series<a class="anchor-link" href="#number-of-pointsintervals-in-the-time-series">¶</a></h3><p>The top factor that determines if a request will succeed or fail is the number of points in the time series. In the default deployment, there is a hard cap of 995 time points in any time series request. This cap is in place because there is a concurrency limit of 1000 on the Lambda function that executes the API requests.</p>
</div>
</div>
</div>
</div>
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell" id="cell-id=4da8f9c2-fe34-4d16-a72a-04b50decd2c5">
<div class="jp-Cell-inputWrapper" tabindex="0">
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput" data-mime-type="text/markdown">
<h3 id="spatial-resolution-and-aoi-size">Spatial resolution and AOI size<a class="anchor-link" href="#spatial-resolution-and-aoi-size">¶</a></h3><p>For datasets that use the <code>rasterio</code> backend, there will be very few limitations on maximum array size as long as the data are COGs and you specify a reasonable output image size (or use the <code>max_size</code> parameter) in your request.</p>
<p>For datasets without overviews/pyramids, <code>titiler-cmr</code> will need to read all of the bytes that overlap the request AOI even if the resulting image is going to be downsampled for a GIF. Therefore, if the area of interest for a <code>/timeseries/statistics</code> or <code>/timeseries/bbox</code> request will create a large array that is likely to exceed the capacity of the Lambda function, the request will fail fast.</p>
<p>The limits for the <code>xarray</code> backend are:</p>
Expand Down Expand Up @@ -1469,20 +1464,6 @@ <h3 id="examples">Examples<a class="anchor-link" href="#examples">¶</a></h3><p>
</div>
</div>
</div>
<div class="jp-Cell jp-MarkdownCell jp-Notebook-cell" id="cell-id=4033af4c-6c85-45d5-9e5e-f2a15af471ab">
<div class="jp-Cell-inputWrapper" tabindex="0">
<div class="jp-Collapser jp-InputCollapser jp-Cell-inputCollapser">
</div>
<div class="jp-InputArea jp-Cell-inputArea"><div class="jp-InputPrompt jp-InputArea-prompt">
</div><div class="jp-RenderedHTMLCommon jp-RenderedMarkdown jp-MarkdownOutput" data-mime-type="text/markdown">
<h2 id="tips">Tips<a class="anchor-link" href="#tips">¶</a></h2><ul>
<li>If you hit an error because the total size of the request is too large, try reducing the temporal resolution of the time series, e.g. from daily (<code>P1D</code>) to weekly (<code>P7D</code>) or greater (<code>P10D</code>)</li>
<li>If you need higher temporal resolution but the full request is not able handle it, split the request into multiple smaller requests and merge the results yourself!</li>
</ul>
</div>
</div>
</div>
</div>
</div> <!-- jp-Notebook -->
</div> <!-- jupyter-wrapper -->

Expand Down
30 changes: 10 additions & 20 deletions pr-previews/pr-42/deployment/time_series_api_limits/time_series_api_limits.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
"source": [
"# Time series API limits\n",
"\n",
"Last updated: 2025-01-02\n",
"\n",
"The `titiler-cmr` API can be deployed as a Lambda function in AWS. Since requests to the time series endpoints will make recursive requests to the Lambda function for the lower-level time step operations, there are some limits in place to avoid making large requests that are likely to overwhelm the API.\n",
"\n",
"## Highlights\n",
Expand All @@ -15,6 +17,11 @@
"- The `/timeseries/bbox` endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to **100,000,000 (`1e8`) total pixels**. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.\n",
"- The `/timeseries/statistics` endpoint can handle larger requests than the `/timeseries/bbox` endpoint Based on a coarse evaluation of the API, requests are limited to **15,000,000,000 (`1.5e10`) total pixels** as long as the individual images read by the `/statistics` endpoint are smaller than 56,250,000 (`5.625e7`) pixels.\n",
"\n",
"## Tips\n",
"\n",
"- If you hit an error because the total size of the request is too large, try reducing the temporal resolution of the time series, e.g. from daily (`P1D`) to weekly (`P7D`) or greater (`P10D`)\n",
"- If you need higher temporal resolution but the full request is not able handle it, split the request into multiple smaller requests and merge the results yourself!\n",
"\n",
"## Background\n",
"The time series API provides rapid access to time series analysis and visualization of collections in the CMR catalog, but there are some limitations to the API deployment that require some care when making large requests.\n",
"\n",
Expand All @@ -23,7 +30,7 @@
"- Request AOI size\n",
"- Number of points/intervals in the time series\n",
"\n",
"These factors all influence the runtime and memory footprint of the initial `/timeseries` request and requests that are too large in any of those dimensions can result in an API failure. Here are a few guidelines to help you craft successful time series requests."
"These factors all influence the runtime and memory footprint of the initial `/timeseries` request and requests that are too large in any of those dimensions can result in an API failure."
]
},
{
Expand All @@ -35,14 +42,8 @@
"\n",
"### Number of points/intervals in the time series\n",
"\n",
"The top factor that determines if a request will succeed or fail is the number of points in the time series. In the default deployment, there is a hard cap of 995 time points in any time series request. This cap is in place because there is a concurrency limit of 1000 on the Lambda function that executes the API requests."
]
},
{
"cell_type": "markdown",
"id": "4da8f9c2-fe34-4d16-a72a-04b50decd2c5",
"metadata": {},
"source": [
"The top factor that determines if a request will succeed or fail is the number of points in the time series. In the default deployment, there is a hard cap of 995 time points in any time series request. This cap is in place because there is a concurrency limit of 1000 on the Lambda function that executes the API requests.\n",
"\n",
"### Spatial resolution and AOI size\n",
"\n",
"For datasets that use the `rasterio` backend, there will be very few limitations on maximum array size as long as the data are COGs and you specify a reasonable output image size (or use the `max_size` parameter) in your request.\n",
Expand Down Expand Up @@ -275,17 +276,6 @@
"\n",
"However, there is a maximum image size that we can read with the `xarray` backend, so we cannot increase the bounding box indefinitely. The limit imposed on the API at this time is `5.6e7` pixels (7500 x 7500 pixels). In the case of MUR-SST, that is a bounding box of roughly 75 x 75 degrees."
]
},
{
"cell_type": "markdown",
"id": "4033af4c-6c85-45d5-9e5e-f2a15af471ab",
"metadata": {},
"source": [
"## Tips\n",
"\n",
"- If you hit an error because the total size of the request is too large, try reducing the temporal resolution of the time series, e.g. from daily (`P1D`) to weekly (`P7D`) or greater (`P10D`)\n",
"- If you need higher temporal resolution but the full request is not able handle it, split the request into multiple smaller requests and merge the results yourself!"
]
}
],
"metadata": {
Expand Down
Loading

0 comments on commit 34b16b2

Please sign in to comment.