Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Additional initializer #23

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Additional initializer #23

wants to merge 14 commits into from
+832 −374

Conversation

Andrey170170
Copy link
Collaborator

@Andrey170170 Andrey170170 commented Feb 4, 2025 •
edited by egrace479
Loading

Added new initializers - fathom_net, EoL and Lila

Small fixes:

  • added MATERIAL_CITATION filtering for gbif initializer, issue described here
  • formatting updates
  • consolidate dependencies in pyproject
  • adds descriptions of downloaded data format

Andrey170170 and others added 10 commits August 7, 2024 21:04
@Andrey170170
Fathom_net initializer
1e18716
@Andrey170170
Fathom_net update
84adde8 
Small fixes
Also adjusted tools to use `source_id` instead of `gbif_id`
@Andrey170170
Small fixes
95e8bef 
Added tool_name_override option for Tools, to be able to use custom tools
@Andrey170170
Small fixes
ddad5f2 
Added the use of `verification_scheme` instead of hard coded column names for some of the parts of the runner
@Andrey170170
EoL download
bff688f 
plus some minor fixes
@Andrey170170
ReadMe file update
0edeeb8 
Updated readme - added `how to access data` section.

Updated pyproject.toml - added dependency libraries directly into this file, instead of link to `requirements.txt`.
@Andrey170170
Update README.md
049e086
@Andrey170170
Lila download
035839e
@Andrey170170
Updated tools checkpoint options
08f7a62 
Now there is a distinction between scheduled filtering or scheduling jobs and completed ones.

Adjusted logic of scripts according to this change.
@Andrey170170
Added filtration to gbif initializer using basisOfRecords
bbe77c4
@egrace479 egrace479 added documentation Improvements or additions to documentation enhancement New feature or request labels Feb 4, 2025
@Andrey170170
Initializers refactoring
34ecfd7 
Extracted initializers into a class structure
Rewrote initialization calling file to have a dict with initialization types.
Added `initializer_type` in mandatory config fields
@Andrey170170
Initializers refactoring
307054b 
Small fix
@egrace479
Copy link
Member

Please add description of base initializer, inheritance to child initializers, and considerations for making a custom child initializer. Use existing as examples, e.g., filters that could be applied (GBIF excluding MATERIAL_CITATION). This would be a good place to note the importance of understanding the metadata coming from the source before creating a child initializer and to not rely on source IDs to be persistent (also check uniqueness if relying on them to map to additional metadata)---considering EOL content IDs, which may be unique but it's the page ID that maps to the taxa information.

As discussed, put this description into a README in the initializer/ folder and link to it from the root repo README.

Andrey170170 added 2 commits February 9, 2025 23:47
Documentation
03b265f 
Added README.md to initializer.
Made small code quality adjustments to initializers
Documentation
b37de48 
Added doc strings to `base_initializer.py`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thompsonmj thompsonmj Awaiting requested review from thompsonmj

@egrace479 egrace479 Awaiting requested review from egrace479

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Andrey170170 @egrace479