Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spc nonpublic data loading #1070

Merged
merged 1 commit into from
Jan 8, 2025
Merged

Conversation

STBadman
Copy link
Contributor

@STBadman STBadman commented Jan 2, 2025

Fix, simplify, and make more transparent the logic for loading non-public SWEAP/SPC data

New logic is

  1. try downloading from sweap server looking for files with a "psp" prefix
  2. If not enough files downloaded, raise an Exception as such and try looking for remaining files with a "spp" prefix
  3. If no files found, raise a further Exception try one more time in the public directory (spdf).

This also enforces that the try/except handling checks the error is the one raised on purpose (for any future debugging) and prints the error message, handles the case when the user calls a trange with datetimes rather than strings.

I also removed the if/else statement for last_version to simplify things - I couldn't see how it was necessary as last_version was being passed to the download function in all cases.

@jameswilburlewis jameswilburlewis self-assigned this Jan 8, 2025
@jameswilburlewis jameswilburlewis added refactoring Data Servers Issues with remote data servers (SPDF, MMS, MAVEN, JAXA, etc) labels Jan 8, 2025
@jameswilburlewis
Copy link
Contributor

One of the unit tests failed for this change:

Traceback (most recent call last):
  File "/home/runner/work/pyspedas/pyspedas/pyspedas/projects/psp/tests/tests.py", line 22, in test_unpublished_data
    spc = pyspedas.projects.psp.spc(trange=['2018-11-5', '2018-11-5/06:00'], username='hello', password='world')
  File "/home/runner/work/pyspedas/pyspedas/pyspedas/projects/psp/spc.py", line 113, in spc
    return load(instrument='spc', trange=trange, datatype=datatype, level=level, suffix=suffix, prefix=prefix,
        get_support_data=get_support_data, varformat=varformat, varnames=varnames, downloadonly=downloadonly,
        notplot=notplot, time_clip=time_clip, no_update=no_update, username=username, *** last_version=last_version, force_download=force_download)
  File "/home/runner/work/pyspedas/pyspedas/pyspedas/projects/psp/load.py", line 327, in load
    trange_temp = [remote_dates[len(files)],t_end]    # set new trange to check for spp prefixes
                                            ^^^^^
UnboundLocalError: cannot access local variable 't_end' where it is not associated with a value

t_end isn't defined here -- it should be probably be replaced with trange[1] in line 327 to match the previous code. I think I'll go ahead and merge as-is, then fix the bug and make sure the tests pass. You'll probably want to propagate that change back to your fork.

@jameswilburlewis jameswilburlewis merged commit b70995f into spedas:master Jan 8, 2025
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Data Servers Issues with remote data servers (SPDF, MMS, MAVEN, JAXA, etc) refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants