T40464 Upgrade API to python 3.11 #504
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JenySadadia
force-pushed
the
upgrade-to-python-11
branch
2 times, most recently
from
c0ad0b9 to
1227a11
Compare
JenySadadia
changed the title
JenySadadia
force-pushed
the
upgrade-to-python-11
branch
2 times, most recently
from
39e2693 to
0de1b76
Compare
|
Seems like API tests are not happy with this upgrade. Looking into the issues. |
|
As discussed offline today (btw we run into |
pawiecz
reviewed
Mar 20, 2024
Nice, let's use it then. I'll test it. |
JenySadadia
force-pushed
the
upgrade-to-python-11
branch
2 times, most recently
from
fb76fcd to
1250def
Compare
Upgrade Python version to 3.11 for API. Update
docker recipe that is being used for Developer mode
to use `Python 3.11` image.
Update requirements version to enable support for Python 3.11.
Below are the required changes:
1. `httptools` that is a requirement for `uvicorn`, failed to
build for existing package version that was `0.13.4`.
```
gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.11 -I/tmp/pip-install-o_24u7rm/httptools_8955220ea3f34d0bae14ed2b27b468ae/vendor/http-parser -c httptools/parser/parser.c -o build/temp.linux-x86_64-cpython-311/httptools/parser/parser.o -O2
httptools/parser/parser.c:212:12: fatal error: longintrepr.h: No such file or directory
212 | #include "longintrepr.h"
| ^~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for httptools
```
Fix it by using `uvicorn` version with Python 3.11 support i.e. `0.29.0`.
2. Since `uvicorn` is one of the requirements of `fastapi`, we
also need to update its version to make it compatible with `uvicorn`.
The fastapi version `0.99.1` has been selected to avoid dependency conflicts with
existing `fastapi-users` version i.e. `10.4.0`. Otherwise the below build failure
will take place:
```
ERROR: Cannot install fastapi-users, fastapi-users[beanie,oauth]==10.4.0 and fastapi[all]==0.110.0 because
these package versions have conflicting dependencies.
```
3. `fastapi==0.99.1` also removes the requirement of pinning `pyyaml` version.
Dropped: 473bd7b ("docker/api: pin PyYAML version to 5.3.1")
4. Also, upgrade `motor` version to use it with python 3.11.
5. Upgrade `pylint` to the latest version i.e. `3.1.0`to resolve
below incompatibility issue from `wrapt` package:
```
File "/usr/local/lib/python3.11/site-packages/wrapt/decorators.py", line 34, in <module>
from inspect import ismethod, isclass, formatargspec
ImportError: cannot import name 'formatargspec' from 'inspect' (/usr/local/lib/python3.11/inspect.py)
```
Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
Added description to commit message. |
added 3 commits
March 21, 2024 10:39
Upgrade python package version to make them compatible with Python 3.11. Match versions with requirements specified in `docker/api/requirements.txt` and `docker/api/requirements-dev.txt`. Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
As a part of adding support for Python 3.11 in API, add `3.11` version for `check` job. Since GH action for linter annotation is not tested with `python 3.11` yet, I am keeping its version `3.10` as of now. Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
Add python 3.11 to supported version for running Github actions for tests. Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
JenySadadia
added
the
staging-skip
JenySadadia
force-pushed
the
upgrade-to-python-11
branch
5 times, most recently
from
c4a6bca to
d8279cc
Compare
added 4 commits
March 26, 2024 17:15
FastAPI doesn't validate `None` when handler has specified
a response model. Fix the below validation error for GET node by ID
and GET group by ID requests:
```
{"detail":"1 validation error for UserGroup\nresponse\n none
is not an allowed value (type=type_error.none.not_allowed)"}
```
The change has been effective from `fastapi 0.80.0` version.
Reference: fastapi/fastapi#2725.
Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
The `fastapi` version upgrade affected unit tests as `TestClient` has been upgraded to use `httpx`. Reference to the release note: https://fastapi.tiangolo.com/release-notes/#0870 This introduced the below failure in the API unit tests: ``` > raise RuntimeError('Event loop is closed') E RuntimeError: Event loop is closed /usr/local/lib/python3.11/asyncio/base_events.py:520: RuntimeError ``` Based on the observation, it seems like somehow `TestClient` is closing event loop after executing the first test. It is backed by the GH issue: encode/starlette#2069 Fix the issue by using the same test client for all the synchronous tests. Fixture `event_loop` is not required anymore after changing `TestClient` scope to `session`. Hence, drop it. Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
A minor fixup related to comment placement. Fixes: 18b106c ("tests/unit_tests: update user related tests") Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
Due to `fastapi` version change, the newer `TestClient` from `fastapi` is causing issues by creating its own different event loop (Reference: https://www.starlette.io/testclient/). That conflicts with the event loop of database client created in the app startup handler. Fix the below error by using async test client for all the tests: ``` @pytest.fixture(scope='session') def test_client(): """Fixture to get FastAPI Test client instance""" with TestClient(app=versioned_app, base_url=BASE_URL) as client: tests/e2e_tests/conftest.py:36: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ api/main.py:87: in initialize_beanie await db.initialize_beanie() api/db.py:51: in initialize_beanie await init_beanie( .local/lib/python3.11/site-packages/beanie/odm/utils/init.py:750: in init_beanie await Initializer( .local/lib/python3.11/site-packages/beanie/odm/utils/init.py:128: in __await__ yield from self.init_class(model).__await__() .local/lib/python3.11/site-packages/beanie/odm/utils/init.py:713: in init_class await self.init_document(cls) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <beanie.odm.utils.init.Initializer object at 0x7457d1843cd0> cls = <class 'api.models.User'> > build_info = await self.database.command({"buildInfo": 1}) E RuntimeError: Task <Task pending name='anyio.from_thread.BlockingPortal._call_func' coro=<BlockingPortal._call_func() running at /home/kernelci/.local/lib/python3.11/site-packages/anyio/from_thread.py:217> cb=[TaskGroup._spawn.<locals>.task_done() at /home/kernelci/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py:699]> got Future <Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/lib/python3.11/asyncio/futures.py:387]> attached to a different loop .local/lib/python3.11/site-packages/beanie/odm/utils/init.py:545: RuntimeError ``` Signed-off-by: Jeny Sadadia <jeny.sadadia@collabora.com>
JenySadadia
force-pushed
the
upgrade-to-python-11
branch
from
d8279cc to
4fb627b
Compare
JenySadadia
removed
the
staging-skip
pawiecz
reviewed
Mar 27, 2024
|
Tested OK on staging. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #479
Upgrade Python version to 3.11 for API. Update docker recipe that is being used for Developer mode to use
Python 3.11image.Update requirements version to enable support for Python 3.11.
Below are the required changes:
httptoolsthat is a requirement foruvicorn, failed tobuild for existing package version that was
0.13.4.Fix it by using
uvicornversion with Python 3.11 support i.e.0.29.0.uvicornis one of the requirements offastapi, we also need to update its version to make it compatible withuvicorn. The fastapi version0.99.1has been selected to avoid dependency conflicts withexisting
fastapi-usersversion i.e.10.4.0. Otherwise the below build failure will take place:fastapi==0.99.1also removes the requirement of pinningpyyamlversion.Dropped: 473bd7b ("docker/api: pin PyYAML version to 5.3.1")
motorversion to use it with python 3.11.pylintto the latest version i.e.3.1.0to resolve below incompatibility issue fromwraptpackage: