[pull] master from ray-project:master #140

pull · 2023-06-29T21:04:01Z

See Commits and Changes for more details.

Can you help keep this open source service alive? 💖 Please sponsor : )

…mark (#50399) Adds a `.skip_training.fault_tolerance` variant to the image classification training ingest release test which **kills a node every N seconds and tests the worker recovery.** --------- Signed-off-by: Justin Yu <justinvyu@anyscale.com>

## Why are these changes needed? This is the first step towards #47933 It is not very tested at the moment (on Python 3.13), but it compiles locally (with `pip install -e . --verbose`) and can execute a simple workload like ``` >>> import ray >>> ray.init() 2024-10-10 16:03:31,857 INFO worker.py:1799 -- Started a local Ray instance. RayContext(dashboard_url='', python_version='3.13.0', ray_version='3.0.0.dev0', ray_commit='{{RAY_COMMIT_SHA}}') >>> @ray.remote ... def f(): ... return 42 ... >>> ray.get(f.remote()) 42 >>> ``` (and similar for actors). The main thing that needed to change to make Ray work on Python 3.13 was to upgrade Cython to 3.0.11 which seems to be the first version of Cython to support Python 3.13. Unfortunately it has a compiler bug cython/cython#3235 (the fix is not released yet) that I had to work around. I also had to work around cython/cython#5750 by changing some typing from `float` to `int | float`. ## Related issue number  ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Philipp Moritz <pcmoritz@gmail.com> Co-authored-by: pcmoritz <pcmoritz@anyscale.com> Co-authored-by: srinathk10 <68668616+srinathk10@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…#50462)   ## Why are these changes needed? Next step after #50160 to make it more convenient to use UV with Ray. This is a useful runtime environment hook for mirroring the environment of `uv run` to the workers (currently the args to uv run and the working_dir). This is useful because it will allow people to intuitively use `uv run` in a distributed application with the same behavior as for a single python process. This only modifies the environment if the driver was run with `uv run` and could conceivably become the default for drivers run with uv run. This is currently a developer API as implied by the fact that it is in the `_private` namespace. It is currently for experimentation and can needs to be opted in via ```shell export RAY_RUNTIME_ENV_HOOK=ray._private.runtime_env.uv_runtime_env_hook.hook ``` If it works well, we might make it the default in the `uv run` case. ## Related issue number  ## Checks - [ ] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [ ] I've run `scripts/format.sh` to lint the changes in this PR. - [ ] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Philipp Moritz <pcmoritz@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>

especially on building c/c++ parts. this should makes the use of build caches much more efficient Signed-off-by: Lonnie Liu <lonnie@anyscale.com>

… Don't broadcast more than once per `training_step()`. (#50477)

## Why are these changes needed? This is part of #48133. Continuing the approach taken in #49426, make all the encoders work in append mode ## Related issue number #49426 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [x] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Martin Bomio <martinbomio@spotify.com> Signed-off-by: Richard Liaw <rliaw@berkeley.edu> Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

The `map_batches.numpy` release test previously ran slowly and encountered OOM errors because of a memory leak with PyArrow. Since the issue has been worked around by #50403, this PR marks the test as stable again. Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>