CF Roadmap: Kicking off work on Provenance and Lineage theme

[erget]

Topic for discussion

Do you have ideas around data provenance and lineage, and would you like to be involved in ensuring that these themes are represented in a way that's beneficial to the CF Roadmap? This is the discussion for you.

At this year's CF Workshop we divided the roadmap preparations into multiple themes. This is one such theme. As you can see from the table, right now we're working on a green field with a blue sky - nothing really more than 2 keywords to start with. Behind that are a whole bunch of ideas and opportunites to shape where we go with it.

I'm setting up a call for later this month. If you want to be involved, please indicate your availability in this survey by the end of 14 Nov 2024.

Topics:

• what are our core concerns?
• who needs to be involved?
• what other technologies / projects should we be aware of and might we align with or adopt?
• anything else we should be aware of?

From there we can agree how to organise the work moving forward.

If you're not available on those dates but still want to be involved, that's no problem - let me know in the comments or drop me a mail and I'll keep you looped in :)

@pagecp

[JonathanGregory]

Does "provenance and lineage" mean discovery metadata?

[erget]

No, this is focused on being able to inspect data and find out how it was made.

[DocOtak]

Would this possibly be a revival of work on ACDD? I've also come across W3C's PROV a few times.

[erget]

I don't think it's got a good connect to ACDD, because it's not discovery but rather lineage. I like W3C's PROV a whole lot and agree that it seems like a good fit but we should analyse and verify.

[JonathanGregory]

Is it related to cell_methods, and the ways in which cell_methods needs to be enhanced, overhauled or replace e.g. to describe climatologies and anomalies wrt climatology? These matters are being discussed in Discussion 305 opened by @TomLav, following discuss issue 252, opened by @sethmcg, now superseded.

(The present discussion so far reads a bit like Twenty questions. 😃)

[sethmcg]

I think it is at least partially related, yes.

[larsbarring]

My understanding is that many of the mentioned issues and external resources have some overlap. I think that Daniel @erget captures it well:

... focused on being able to inspect data and find out how it was made.

At this general level the W3C's PROV that Andrew @DocOtak points at is the comprehensive resource. The abstract of the web page states:
"Provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability or trustworthiness."

For users requiring this kind all-embracing machinery, maybe the best thing CF can do is to point at PROV in the conventions document, and by including a suitable string in theConventions global attribute.

Personally I think that the 2024 CF Workshop presentations by David Huard here, in the Uncertainty session, and by José Manuel Gutiérrez here, in the Statistical Processing session, offers a lot of food for thought.

Many users do not need e.g. manual quality judgement, expert group voting, and much more, but still do need something more than current cell_methods can offer. As Jonathan @JonathanGregory writes

... ways in which cell_methods needs to be enhanced, overhauled or replace ...

I think that by enhancing the cell method machinery we can cover several existing use cases, but I also think that we need to come up with something new that complements cell methods to cover even more complex use cases and data manipulation/processing. I have for some time been playing with the idea of creating some kind of "pseudo-language". That is, something similar to what is now used for parametric vertical coordinates but with freedom to also describe the equations. Well, still just some wild ideas without much substance behind ....

[erget]

Yes, in the end I reckon it's likely lthat PROV is the spec to use, not to re-invent - but it's still not clear to me how users should do that in a netCDF file, or if we think beyond netCDF in other CF-compliant data. So providing some guidance here so that there's 1 clear way to do it would probably be good. That's where I'm thinking the discussion will go - not necessarily rounding this out for before the roadmap exists but in the roadmap we can set forth what approach we want to use.

[larsbarring]

I fully agree! 👍 +1:

[sethmcg]

Over in discussion #305, we reached the conclusion that there are two levels of provenance to consider: there's the level that tells you how the data was changed such that you can understand what it means now, and there's the level that tracks what happened to the data such that you could completely regenerate it from the original sources by following the chain of provenance.

The first is what we're trying to figure out how we can accomplish with an overhaul to cell_methods. The second is like what gets recorded in the history attribute, but with maybe even more detail, and is (I believe) what we're contemplating in this discussion. The important distinction between the two is that the second is much, MUCH more complicated and detailed than the first, and involves not just abstract geophysical quantities, but entities like files and software and versions and configurations.

I agree that it's good to avoid re-inventing the wheel, and that it's probably best to make use of an existing spec like PROV, but if we're talking about recording a chain of provenance that is sufficiently detailed and specific to let you regenerate the end result completely, I would argue that it should not be mandatory, and definitely should NOT be incorporated into the netCDF file itself, because it can very rapidly become entirely unmanageable.

Recording every step in detail works fine if you're talking about a relatively short and linear chain of derivation, but if it's something more complex where it's branched or if you have a large number of input files, then things start to get out of hand really quickly. If you have multiple files as inputs, you have multiple history attributes (or the equivalent) to track / merge / reconcile / deal with. NCO takes the approach of adding them all to a history_of_appended_files attribute, and if you have more than just a couple input files, you end up with a gigantic header that consists almost entirely of history attributes.

This is not an idle concern; I've already run into problem cases where I had to turn off all history tracking while post-processing model output because I had files generated from hundreds if not thousands of parent files, and the resulting histories were preposterous. And that's just for doing data cleanup to make model output CF-compliant; the problem gets exponentially worse if you're doing things like bias-correcting data or calculating derived quantities with multiple inputs and intermediate outputs. (And if you really want to have nightmares, consider what happens if you do any of this in parallel by splitting files apart then recombining them...)

So I think this is an important concern for the roadmap discussion, and that we want to make sure we're not navigating toward the iceberg of metadata that's bigger than the data it describes.

[erget]

I'm totally on board with your concerns here - this is not something to be underestimated. I've been involved in some work in this area that I think might be useful for the community, because we tried to tackle many of the problems that you're referring to. I'm not claiming this is the only way to do it, or that the solutions we developed were bulletproof, but I think that they could be a useful starting point.

For the purpose of the roadmap the important questions in my mind are what we consider in the scope of our roadmap, i.e. what is important, where do we want to develop a stance, and what do we know about how that would look. And then some kind of horizon when we want to be there and whom we engage with.

More than willing to share the mysterious docs I'm referring to but I'd prefer to do that at a time when we meet - surely there'll be a time before end of Q1 2025 - so I can provide some context and we can discuss, otherwise it turns into a data dump.

But yes, my focus for this discussion is more on the history attribute rather than cell_methods, but I could imagine there are synergies there - I'm not an expert on cell_methods in any way!

[erget]

Hi folks, particularly @sethmcg and @pagecp - the survey has spoken, unfortunately there's no appointment for the near-term that fits for all of us but @sethmcg we'll fill you in and hopefully you can join later.

Anybdoy who wants on the invite let me know and I'll add you. @pagecp you're already on it. It'll be 2024-11-27T13:00Z on Teams.

[sethmcg]

@erget Go ahead and send me an invite. It's possible I'll be awake early enough to hop on.

[feggleton]

@erget do you have any notes from the meeting that you can share?

[erget]

Hi @feggleton , @sethmcg - sorry, I got a bit behind here in responding as I was out out office for a bit.

In the end it was just me in the meeting room so I figured I'd reach out again after New Year's when scheduling's a bit easier and give it another go. Hopefully participation will be easier then as well. I take it you're both keen to be involved? Then I'll reach out again and check your availability before setting a time.

[sethmcg]

@erget Yes, i'm keen to be involved. Thanks!

[JonathanGregory]

I'm interested too, especially to see whether it interacts with cell_methods.

[erget]

Hi folks, we had an appointment last year to roadmap what we want to do with provenance and lineage... In the end it was a veritable echo chamber - I was the only one there! 😱

Do we want to try it again?

Mark your availability here by EOB on 12 Feb (next Wednesday) and we'll set a date.

[sethmcg]

Hi Daniel,
I'm interested, but it looks like you're 8 hours ahead of us, which makes scheduling pretty difficult.

I have a couple specific concerns, so maybe it would make more sense for me to just contribute to online discussions?

[davidhassell]

Hi Daniel, I've not expressed an interest before, but I'd like to join now. I've filled in my availability.

I have acquired an interest in how uncertainty can be recorded (if at all!) from different stages of previous data processing. Regardless of the answer to that, it's be great to see progress on this issue.

[erget]

@sethmcg of course - yes and sorry about the time zone issues, I think there's a few slots in the afternoon that might fit your schedule so you're welcome to join if you can. Otherwise yes, it's very useful if you join the online discussions - we need to get things kicked off though and talk about what our priorities are, and who does what, and therefore it makes sense I think to get the working group together before we do the bulk of the work here on GitHub.

@davidhassell - excellent, glad to hae you on board! Will look forward to working with you on it :)

[erget]

Hi all, the survey has spoken - we'll meet on 20 Feb 2025 at 15 UTC. @davidhassell @JonathanGregory @larsbarring I've put you on the invite. Anybody else who wants to be involved send me an email and I'll rope you in!

[erget]

We met yesterday (sorry @sethmcg for leaving you out in the dark, that was definitely not intentional and it's my fault we missed out on you 😥) and had a first meeting to discuss our intents. We are meeting again on 12 March at 16 CET - if you aren't on the invite and would like to be, let me know and I'll add you.

What use cases did we discuss?

We identified the following reasons why somebody using CF data may want to have provenance baked in:

1. For users of data: Checking if the products in their hands are the latest and greatest version
2. For upstream providers: Being credited when their products are incorporated into downstream products
3. For data providers: Potentially restoring processing chains, potentially including processors and data used

Not all of these use cases need to be fulfilled for things to be useful. The W3C PROV standard provides tools that service all of them.

What do we propose incorporating into the CF Roadmap?

In this order:

1. Being able to put in provenance records into CF data. This might be a point-out to an external file. It might be embedded in the data file itself. Dunno.
2. Linking provenance records in different CF files so that we can have chains that aren't overburdened by metadata.
3. Cryptographically signing it so you can be the nonrepudiable, authoritative author of data products.

Any design principles we want to propose?

We want to re-use PROV rather than adapt it or invent something, because using an existing standard has a lot of advantages. The issue that we see here is how to represent the provenance data in a way that works with CF so we don't have a multutide of implementations.

What are we doing between now and the next meeting?

I made some technical documentation available to the people in the meeting to mull over. This is related to a prototype implementation that we had at EUMETSAT for inspiration, it's not something that we could just take off the shelf and use. So we're thinking about

1. Do we have the right people involved?
2. Are there other communities that we should engage with?
3. Are there other examples we could learn from?
4. Is the PROV data model indeed what we want to use?
5. How could the PROV data model be used in the CF data model?

In particular, Lars kindly volunteered to attempt to provide an overview of uses of PROV in IPCC and other settings, this may or not be ready at that point.

Looking forward to seeing you all at the next meeting :)

[sethmcg]

I had a brief look at PROV, and it seems like a reasonable tool to use. It's very general, and I think we'd want to define something more specific - a dialect of it, I guess. Sort of like how the netCDF format is actually very general, and CF defines a specific and more limited way of using it.

The one thing that I think is very important if we use PROV is that the basic PROV 'entity' for data needs to be an entire dataset not a single data file. CF is in many ways defined at the file level, so there may be some tension there, and it may create limitations with regard to supporting use case no. 3. You also have to figure out where the boundaries of the dataset are, which is a lot more ambiguous than what a file is. But I think this is important because I have a real-world use case where the difference between the two is vast.

I recently did a simple aggregation operation on a dataset where I pulled a subset of variables from some files and concatenated them, going from hourly organization (one timestep per file) to monthly (~360 timesteps per file). This dataset covers several decades and has more than 750,000 individual files in it. Described at the dataset level, this is just a couple steps, probably not more than a half-page of PROV information. Described at the file level, it's more than 66MB of metadata just for the filenames.

This may seem simple and obvious, but I think the hitch is that people always want to do the easy thing, and it's less effort to have software automatically dump a bunch of commands into a file than it is to have a human in the loop describing what happened in a meaningful way at a more abstract level. Mindless recording of commands works fine for small data, but climate and weather data is big data, and if we want our provenance metadata to be useful and usable, we'll need to steer the community away from that approach.

[erget]

Interesting use case! Yes, for me the concept of datasets and data files is important too, whereby in my view data files are members of datasets. Clearly not a use case for on the fly processing because there is overhead in describing a dataset.

[davidhassell]

Clearly not a use case for on the fly processing because there is overhead in describing a dataset.

Possibly a use case for the upcoming aggregation conventions provide a way of recording aggregations of files in a CF-netCDF file without copying the data, and removing the need to repeatedly apply your aggregation algorithm every time you access the aggregated dataset.

[TomLav]

Described at the file level, it's more than 66MB of metadata just for the filenames.

Files pertaining to a dataset often have filenames following a convention. The commands run on these files are often the same. I wonder if a simple grammar with regexp and loops could compress most of this repeated information in much fewer characters. This would be less human-readable.

Some time ago I made the loop_wrapper tool. We still use it in our internal processing of files across dates. Maybe documenting the processing of zillion files could be stored in a loop/regexpr form in PROV instead of explicitly. For example:

loop_wrapper --every 1M 18500101 20200101 ./command -t -H -a auxFile.nc -f dataFile_{d:%Y%m}15.nc

would actually stand for

./command -t -H -a auxFile.nc -f dataFile_18500115.nc
./command -t -H -a auxFile.nc -f dataFile_18500215.nc
...
./command -t -H -a auxFile.nc -f dataFile_20191215.nc;

PS : For the avoidance of doubt I support that we first attempt describing operations on Datasets rather than files, as @sethmcg advocates for. But if this is too cumbersome, maybe something along the line of the above, somehow mixed with PROV, could help compress the size of file-level provenance information.

[davidhassell]

We discussed the use case

3. For data providers: Potentially restoring processing chains, potentially including processors and data used

I'd like to add that this use case could well cover my use case of recording the uncertainty of data used at different parts of the processing chain, assuming that each contributing dataset includes its own uncertainty description (which is not in any way a part of this discussion!).