GPyTorch support for ELFI

[YannickNoelStephanKuhn]

Feature proposal

I believe it would be beneficial for ELFI to not only work on GPy as a backbone, but GPyTorch as well. My reasons are as follows:

• GPyTorch supports more involved Gaussian Process Regression modelling.
• It supports massively parallelized computations, for example on HPCs or GPUs.
• It makes its inner workings more transparent.
• It has more active support.

Work that has already been done

To understand ELFI, GPy, PyTorch, and GPyTorch better, I wrote a re-implementation of BOLFI within the ELFI framework that uses GPyTorch for its computations. You may find the implementation on my fork: https://github.com/YannickNoelStephanKuhn/elfi/tree/gpytorch-support
It consists of four files:

• elfi/methods/bo/botorch_acquisition.py re-implements the LCBSC acquisition function within BoTorch.
• elfi/methods/bo/gpytorch_bolfi_model.py re-implements the parabolic regression model of BOLFI within GPyTorch.
• elfi/methods/bo/gpytorch_regression.py wraps the Adam optimizer for Gaussian Process Regression.
• elfi/methods/inference/gpytorch_bolfi.py shows how the Gaussian Process hyperparameters may be tuned with Pyro.

Possible future avenues

The implementation of hyperparameter tuning with Pyro does not work as of now, since Pyro does not support unnormalized distributions. I could not normalize the Gaussian Process distribution or find a way to work around the Pyro limitation.

What happens now

I wanted to have this work freely available, in case someone finds it useful. It may fully integrate into ELFI, or it may be here in case someone else wants to combine the benefits of ELFI and GPyTorch on their own.

[hpesonen]

This is great work, I can't wait to get to testing this! It's been an ongoing discussion for a while now to either switch from GPy to GPyTorch within ELFI or to bring it as an option. Would you be willing to make an official PR for your work?

[YannickNoelStephanKuhn]

Gladly: #488

While I don't have unit tests, those for BOLFI should be suited for copy-pasting; after all, it should behave just the same.
But I have an integration test, and I did it for comparison for GPy as well:

• elfi/examples/example_bolfi_invocation.py
• elfi/examples/example_gpytorchbolfi_invocation.py