Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tidy up the use of interpolation #642

Merged
merged 25 commits into from
Feb 6, 2025
Merged

Tidy up the use of interpolation #642

merged 25 commits into from
Feb 6, 2025

Conversation

tsmbland
Copy link
Collaborator

@tsmbland tsmbland commented Jan 24, 2025

Description

I've always been a bit bugged by the over-reliance on temporal interpolation. After all, there's a well defined time framework, which most objects should follow, so interpolation should rarely be required. Yet the code is riddled with unnecessary uses of interp.

I've found that most of these interp calls can be changed to sel without problem (in the case that the objects in question do cover the appropriate years), whereas others uses of interp are still necessary (i.e. where the object doesn't cover the appropriate years).

I think it's far better to be explicit about this, as using interp where it's not necessary could hide genuine bugs.

I've also got rid of filter_input, since, without interpolation, this is equivalent to sel (although I'm keeping AbstractAgent.filter_input for now as this serves another purpose)

Interpolation is still required for the following:

  • reading the initial market. The method is configurable with the interpolation_mode parameter. I've updated the documentation for this parameter
  • technologies parameters. This previously had no method specified, although I believe it should have been using the interpolation method specified in the settings file. It now is and I've updated the documentation
  • sometimes interpolation is still required for capacity datasets, as the time framework of these is based on the lifetime of assets which may not match the time framework - we should always use linear interpolation here, as explained in cliff_retirement_profile, so there's no need for sectors and agents to have an interpolation attribute. I've created the helper function interpolate_capacity to make this consistent
  • some interpolation remains in the regressions module - I'm not really sure what this module is doing and don't want to change it, but I think it's fine for now

Not sure why, but this appears to have fixed a bug with prices in the trade model (at least I think it was a bug - R2 prices were suspiciously high before). Won't think too much about it now as that model still needs a lot of work, but encouraging

Fixes #265

Type of change

  • New feature (non-breaking change which adds functionality)
  • Optimization (non-breaking, back-end change that speeds up the code)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (whatever its nature)

Key checklist

  • All tests pass: $ python -m pytest
  • The documentation builds and looks OK: $ python -m sphinx -b html docs docs/build

Further checks

  • Code is commented, particularly in hard-to-understand areas
  • Tests added that prove fix is effective or that feature works

@tsmbland tsmbland changed the base branch from foresight to remove_forecast February 4, 2025 14:54
@tsmbland tsmbland requested a review from dalonsoa February 4, 2025 22:42
@tsmbland tsmbland changed the title Remove interpolation Tidy up the use of interpolation Feb 4, 2025
@tsmbland tsmbland marked this pull request as ready for review February 5, 2025 08:46
@tsmbland tsmbland linked an issue Feb 5, 2025 that may be closed by this pull request
Copy link
Collaborator

@dalonsoa dalonsoa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've a question, but the changes make total sense to me. Performance wise, it might even be faster as, presumably, a lookup of the value(s) will be faster than interpolating. In any case, it is more explicit, for sure.

@@ -437,6 +402,25 @@ def avoid_repetitions(data: xr.DataArray, dim: str = "year") -> xr.DataArray:
return data.year[years]


def interpolate_capacity(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conceptually, I'm not entirely sure if a linear interpolation of the capacity makes much sense. Capacity should only change when there's an investment in the relevant technology, so it should look very much staircase, going up only in the investment years. Wouldn't previous make more sense?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're completely right, yes. If the capacity profile was created with cliff_retirement_profile, which I believe most are, then there will be no difference, but in general forward filling makes more sense than linear interpolation.

I've created an issue about this (#656) and will return to it later. It's an easy change to make, but some of the results are affected and I'd like to understand why before merging.

@tsmbland tsmbland changed the base branch from remove_forecast to main February 6, 2025 14:12
@tsmbland tsmbland merged commit c6453d7 into main Feb 6, 2025
11 of 14 checks passed
@tsmbland tsmbland deleted the interpolation branch February 6, 2025 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

interpolation_mode parameter needs clarification (9.2)
2 participants