-
Notifications
You must be signed in to change notification settings - Fork 87
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Problem
The purpose of this PR is to introduce a class, `AsyncioIndex` that
provides an async version of the functionality found in the `Index`
client. This includes standard index data plane operations such as
`upsert`, `query`, etc as well as bulk import operations
(`start_import`, `list_imports`, etc).
## Solution
This is a very complex diff with many moving parts.
- New dependency on `aiohttp`, an asyncio-compatible http client.
- New dev dependency on `pytest-asyncio` to support async testing
- Heavy refactoring in `pinecone/openapi_support` to introduce
asyncio-variants of existing classes: `AsyncioApiClient`,
`AsyncioEndpoint`, and `AiohttpRestClient`. I don't love the way any of
these are currently laid out, but for simplicity sake I decided to hew
close to the existing organization since this was already going to be a
complex change.
- Adjustments to our private python openapi templates in order to
generate asyncio versions of api client (e.g.
`AsyncioVectorOperationsApi`) objects and reference the objects named
above.
- Create a new class, `AsyncioIndex` that uses these asyncio variant
objects. Since the majority of the logic (validation, etc) inside each
data plane method of `Index` was previously extracted into
`IndexRequestFactory`, the amount of actual new code needed inside this
class was minimal async from signature changes to use `async` / `await`.
- Add new integration test covering asyncio usage with both sparse and
dense indexes.
- Very mechnical refactoring to also bring bulk import functionality
into the AsyncioIndex class as a mixin. I did not add automated tests
for these due to the external dependencies required to properly
integration test this (e.g. parquet files hosted on S3). Will need to
manually verify these in testing.
Also:
- Drop python 3.8, which is now end of life
- Removed `ddtrace` dev dependency for logging test info in datadog.
This was giving me a lot of annoying errors when running tests locally.
I will troubleshoot and bring it back in later.
- Updated `jinja` and `virtualenv` versions in our poetry.lock file to
resolve dependabot alerts
- Work to implement the asyncio codepath for GRPC was previously handled
in a different diff
## Usage
In a standalone script, you might do something like this:
```python
import random
import asyncio
from pinecone import Pinecone
async def main():
pc = Pinecone(api_key="key")
async with pc.AsyncioIndex(name="index-name") as index:
tasks = [
index.query(
vector=[random.random()] * 1024,
namespace="ns1",
include_values=False,
include_metadata=True,
top_k=2
) for _ in range(20)]
# Execute 20 queries in parallel
results = await asyncio.gather(*tasks)
print(results)
asyncio.run(main())
```
## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] This change requires a documentation update
- [x] Infrastructure change (CI configs, etc)
- [ ] Non-code change (docs, etc)
- [ ] None of the above: (explain here)
## Test Plan
Describe specific steps for validating this change.- Loading branch information
Showing
94 changed files
with
6,715 additions
and
906 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| name: 'Test Asyncio' | ||
| description: 'Runs tests on the Pinecone data plane' | ||
|
|
||
| inputs: | ||
| spec: | ||
| description: 'The deploy spec of the index' | ||
| required: true | ||
| use_grpc: | ||
| description: 'Whether to use gRPC or REST' | ||
| required: true | ||
| freshness_timeout_seconds: | ||
| description: 'The number of seconds to wait for the index to become fresh' | ||
| required: false | ||
| default: '60' | ||
| PINECONE_API_KEY: | ||
| description: 'The Pinecone API key' | ||
| required: true | ||
| python_version: | ||
| description: 'The version of Python to use' | ||
| required: false | ||
| default: '3.9' | ||
|
|
||
| runs: | ||
| using: 'composite' | ||
| steps: | ||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ inputs.python_version }} | ||
|
|
||
| - name: Setup Poetry | ||
| uses: ./.github/actions/setup-poetry | ||
| with: | ||
| include_grpc: ${{ inputs.use_grpc }} | ||
| include_dev: 'true' | ||
|
|
||
| - name: Run data plane tests | ||
| id: data-plane-asyncio-tests | ||
| shell: bash | ||
| run: poetry run pytest tests/integration/data_asyncio -s -vv | ||
| env: | ||
| PINECONE_API_KEY: ${{ inputs.PINECONE_API_KEY }} | ||
| USE_GRPC: ${{ inputs.use_grpc }} | ||
| SPEC: ${{ inputs.spec }} | ||
| FRESHNESS_TIMEOUT_SECONDS: ${{ inputs.freshness_timeout_seconds }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule apis
updated
from 8562ca to 63e97d
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule python-oas-templates
updated
from c37f9f to 123b0b
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.