Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updates to flowsa metadata #399

Merged
merged 24 commits into from
Jan 18, 2024
Merged

updates to flowsa metadata #399

merged 24 commits into from
Jan 18, 2024

Conversation

catherinebirney
Copy link
Collaborator

@catherinebirney catherinebirney commented Jan 5, 2024 •
edited
Loading

  • option to specify git version/hash when returning an FBA
  • new fxn compare_FBA_results() to compare the results of two FBA dataframes
  • new github action to compare an FBA method to an FBA parquet on Data Commons
  • updates to FBS metadata - output json captures the metadata for all FBAs identified in an FBS yaml for all recrusive/sequential datasets
  • drop FBS in source_catalog. Update all calls of Employment FBS in method yamls to include data_format definition
  • updates to FBA publication and download dates

@catherinebirney
updates to FBA bib info - urls and dates
fb4fcc6
@catherinebirney
Merge branch 'develop' into bib
fc308c9 
# Conflicts:
#	flowsa/methods/flowbyactivitymethods/BEA_Make_AR.yaml
#	flowsa/methods/flowbyactivitymethods/BEA_Make_Detail_BeforeRedef.yaml
@catherinebirney
Merge branch 'develop' into bib
8db66ea 
# Conflicts:
#	flowsa/methods/flowbyactivitymethods/CalRecycle_WasteCharacterization.yaml
@catherinebirney
Merge branch 'develop' into bib
458ffcd 
# Conflicts:
#	flowsa/methods/flowbyactivitymethods/BEA_GDP_GrossOutput.yaml
#	flowsa/methods/flowbyactivitymethods/BEA_Make_Detail_BeforeRedef.yaml
#	flowsa/methods/flowbyactivitymethods/BEA_Use_Detail_PRO_BeforeRedef.yaml
@catherinebirney
Merge branch 'develop' into bib
28f6078
@catherinebirney
Merge branch 'develop' into bib
09b9f8a
@catherinebirney
option to specify git version when returning FBA
0c06114
@catherinebirney
new fxn compare_FBA() to compare two FBA datasets - can specify df ve…
77e3164 
…rsion and hash
@catherinebirney
drop old code
e853250
@catherinebirney
rename fxn to comapre_FBA_results()
e2a5c43
@catherinebirney
move defining Employment_location_year as an FBS out of source_catalo…
9b8a3fc 
…g and into FBS method yamls
@catherinebirney
update FBS metadata to capture nested and sequential datasources and …
3ad2a02 
…their combinations
@catherinebirney
address duplicate assignment of _sector_fields in flowby_config - ren…
30511b1 
…ame one to _fba_sector_fields and _fbs_sector_fields
@catherinebirney
specify Employment FBS as FBS in method yamls
421cd43
@catherinebirney
look for attribution sources under "clean_source"
1ac5f25
@catherinebirney
Merge branch 'develop' into bib
35daf49 
# Conflicts:
#	flowsa/methods/flowbyactivitymethods/BEA_GDP_GrossOutput.yaml
#	flowsa/methods/flowbyactivitymethods/BEA_Make_AR.yaml
@catherinebirney
update FBA download and publication dates
2ddc54e
@catherinebirney
Merge pull request #275 from USEPA/bib
58e4afd 
add source publication dates to FBAs - bibliography not yet updated to include
@catherinebirney
revert reassignment of _sector_fields - duplication necessary for col…
5d3164c 
…umn check in FBA
@catherinebirney catherinebirney marked this pull request as ready for review January 12, 2024 23:09
@catherinebirney catherinebirney marked this pull request as draft January 13, 2024 01:10
@catherinebirney catherinebirney marked this pull request as ready for review January 13, 2024 01:11
@catherinebirney catherinebirney marked this pull request as draft January 17, 2024 16:38
@catherinebirney catherinebirney marked this pull request as ready for review January 17, 2024 16:38
@catherinebirney
Copy link
Collaborator Author

@bl-young Once the checks complete I'd like to pull this PR into master to check the new compare FBA github action is working correctly

@bl-young
Copy link
Collaborator

You can also commit .github/workflows/compare_single_FBA.yml by itself directly to master and then test on this branch without pulling everything else in first

@catherinebirney
Copy link
Collaborator Author

You can also commit .github/workflows/compare_single_FBA.yml by itself directly to master and then test on this branch without pulling everything else in first

That is a better idea, I will do that, thanks!

@catherinebirney
Copy link
Collaborator Author

@bl-young The new github action works. I am done with changes on this branch, so this PR is ready to be pulled in

@bl-young
Copy link
Collaborator

Awesome - any ideas of an FBA that has changed that we could test this on? Nothing immediate comes to mind for me.

@catherinebirney
Copy link
Collaborator Author

I locally tested on FBAs that have changed - Census_PEP_Poplulation changed between v1.3.0 and v2.0.0 but there won't be any differences in the FBAs flowsa creates right now vs what is on data commons, so you would need to download those files

bl-young
bl-young reviewed Jan 17, 2024
View reviewed changes
Copy link
Collaborator

@bl-young bl-young left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is source_publication_date meant to appear in the FBA metadata?

@catherinebirney catherinebirney marked this pull request as draft January 17, 2024 22:11
@catherinebirney catherinebirney marked this pull request as ready for review January 17, 2024 22:11
@catherinebirney
Copy link
Collaborator Author

yes, thanks - I added the source pub date to FBA metadata json. I'll pull this in when the checks are done

@catherinebirney catherinebirney merged commit 037fd96 into develop Jan 18, 2024
12 checks passed
@catherinebirney catherinebirney deleted the metadata branch January 18, 2024 01:54
@bl-young bl-young mentioned this pull request Jan 18, 2024
Closed
@bl-young bl-young added this to the 2.0.x milestone Mar 25, 2024
@bl-young bl-young modified the milestones: 2.0.x, 2.0.2 Jun 11, 2024
@bl-young bl-young mentioned this pull request Jun 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bl-young bl-young bl-young left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.0.2
Development

Successfully merging this pull request may close these issues.
2 participants
@catherinebirney @bl-young