[ENH] estimators failing check_regressor_against_expected_results

[TonyBagnall]

Describe the feature or idea you want to propose

no idea on these

• "RDSTRegressor"
• "RISTRegressor"

Describe your proposed solution

fix or create issue

Describe alternatives you've considered, if relevant

No response

Additional context

No response

[ricor07]

Hello, I'd like to join the project by working on this issue.

[baraline]

Hi @ricor07, just a heads-up, this issue is a bit tricky, so if you are new to python/numba/git CI, this might not be the easiest place to start with.

The problem we're facing is that these two estimators (RDST and RIST) are failing a test (check_regressor_against_expected_results, see this file and this one to reproduce the results) on some of the platforms we are testing on (Ubuntu/Windows/MacOS).

The problem is that the expected results can differ (despite fixing a random seed) from platform to platform. The goal of the issue is to understand and fix this, but we didn't have much success on that for now. You can go in the test_config file to make these test run on the CI, otherwise there are excluded.

[SalmanDeveloperz]

Hi @baraline , I'd like to work on this issue. I have experience with Python, Git CI, and debugging cross-platform. I'll start by reproducing the test failures and investigating potential causes. Could you assign me this issue? Also, do you have any specific insights or previous debugging attempts that might help narrow down the problem?
Looking forward for your kind response.
Best

[MatthewMiddlehurst]

Just noting these tests will only run on Ubuntu in the future.

[SalmanDeveloperz]

Thanks for the update! I have an Ubuntu setup and would like to test and debug the issue there. Could you guide me on how to properly set up and run the tests on Ubuntu? Any specific steps or configurations I should be aware of?
Best Regards

[baraline]

You should find everything you need to setup your environnement and run test in the documentation.
Althought the issue might be fixed by running the tests on Ubuntu only, but we need to verify that.

[SalmanDeveloperz]

Hi @baraline, @MatthewMiddlehurst , @TonyBagnall ,
I have set up the environment on Ubuntu and tested the estimators. Initially, the tests did not run, but after commenting out the following lines in testing_config.py:

# "RDSTRegressor": ["check_regressor_against_expected_results"],  
# "RISTRegressor": ["check_regressor_against_expected_results"], 

all tests passed successfully (pytest aeon/testing/estimator_checking/ -v).

This suggests that restricting the tests to Ubuntu may resolve the issue, as previously noted. However, I’d like to confirm whether removing these exclusions is the intended fix or just a temporary workaround.

Please advise on the next steps, should I proceed with a PR or investigate further?

Check the screenshot below for reference.

[baraline]

You can proceed with a PR that would remove the two test exclusion

"RDSTRegressor": ["check_regressor_against_expected_results"],  
"RISTRegressor": ["check_regressor_against_expected_results"], 

This way we can test it on the GitHub CI to confirm that its working as intended now. Note that this PR will depend on #2486 getting in, as it is that PR that modifies the testing behaviour.

[MatthewMiddlehurst]

I have merged #2486