NeuralForecast throws "segmentation fault" exception when Statsforecast[dask, spark], MLForecast[dask, spark] are also installed

[chenx26]

What happened + What you expected to happen

I am trying to use statsforecast, mlforecast and neuralforecast in one notebook to compare the performance of different types of models. The vanilla versions of these package work well together, but when I try to add on dask and spark modules of statsforecast and mlforecast, neuralforecast stops working properly. Trying to create a NeuralForecast object throws segmentation fault exception.

I expect statsforecast[dask, spark] and mlforecast[dask, spark] to work as normal with neuralforecast

Versions / Dependencies

Click to expand

Dependencies: python: 3.10.10 adagio==0.2.6 aiohappyeyeballs==2.4.4 aiohttp==3.11.11 aiosignal==1.3.2 alembic==1.14.1 appdirs==1.4.4 async-timeout==5.0.1 attrs==25.1.0 bokeh==3.6.2 certifi==2025.1.31 charset-normalizer==3.4.1 click==8.1.8 cloudpickle==3.1.1 colorlog==6.9.0 contourpy==1.3.1 coreforecast==0.0.15 dask==2025.1.0 distributed==2025.1.0 filelock==3.17.0 frozenlist==1.5.0 fs==2.4.16 fsspec==2024.12.0 fugue==0.9.1 idna==3.10 importlib_metadata==8.6.1 Jinja2==3.1.5 joblib==1.4.2 jsonschema==4.23.0 jsonschema-specifications==2024.10.1 lightgbm==4.5.0 lightning-utilities==0.12.0 llvmlite==0.44.0 locket==1.0.0 lz4==4.4.3 Mako==1.3.8 MarkupSafe==3.0.2 mlforecast==1.0.1 mpmath==1.3.0 msgpack==1.1.0 multidict==6.1.0 networkx==3.4.2 neuralforecast==2.0.1 numba==0.61.0 numpy==2.1.3 optuna==4.2.0 packaging==24.2 pandas==2.2.3 partd==1.4.2 patsy==1.0.1 pillow==11.1.0 propcache==0.2.1 protobuf==5.29.3 psutil==6.1.1 py4j==0.10.9.7 pyarrow==19.0.0 pyspark==3.5.4 python-dateutil==2.9.0.post0 pytorch-lightning==2.5.0.post0 pytz==2025.1 PyYAML==6.0.2 ray==2.41.0 referencing==0.36.2 requests==2.32.3 rpds-py==0.22.3 scikit-learn==1.6.1 scipy==1.15.1 six==1.17.0 sortedcontainers==2.4.0 SQLAlchemy==2.0.37 statsforecast==2.0.0 statsmodels==0.14.4 sympy==1.13.1 tblib==3.0.0 tensorboardX==2.6.2.2 threadpoolctl==3.5.0 toolz==1.0.0 torch==2.6.0 torchmetrics==1.6.1 tornado==6.4.2 tqdm==4.67.1 triad==0.9.8 typing_extensions==4.12.2 tzdata==2025.1 urllib3==2.3.0 utilsforecast==0.2.11 xgboost==2.1.3 xyzservices==2025.1.0 yarl==1.18.3 zict==3.0.0 zipp==3.21.0

Reproducible example

# pip install "statsforecast[dask,spark]==2.0.0" "mlforecast[dask,spark]==1.0.1" "neuralforecast==2.0.1"

from mlforecast import MLForecast
from statsforecast import StatsForecast
from neuralforecast import NeuralForecast
from neuralforecast.models import LSTM, NHITS, RNN, NBEATS
from neuralforecast.losses.pytorch import DistributionLoss
print(NeuralForecast(
                                   models=[NBEATS(h=7, input_size=14,
                                                  loss=DistributionLoss(distribution='Poisson', level=[80, 90]),
                                                  stack_types=['identity', 'trend', 'seasonality'],
                                                  max_steps=100,
                                                  val_check_steps=10,
                                                  early_stop_patience_steps=2)], freq=1))

Issue Severity

High: It blocks me from completing my task.

[jmoralez]

Hey. Are you on macos? There can be an incompatibility between pytorch and lightgbm

[chenx26]

@jmoralez Yes, I am on macos, is there anyway to remedy this issue?