Git LFS usage

[jaimergp]

This repository is using a lot of LFS storage above the org quota:

I see that #512 tried to deal with this, but the history still has some. Maybe someone still had a local copy and accidentally reuploaded the content with a push for something else.

[baszalmstra]

As far as I understand, old commits (before #512) still contain references to LFS. We would have to rewrite the entire history to completely get rid of that which would be quite problematic for existing checkouts.

[jaimergp]

Yea, the script would need to export SHA-like tags for at least the release tags but it's not perfect. And this repo has a few tags already 😬 Just flagging this for awareness in case it was an omission. Not sure if it'll become a problem in the future or not.

[baszalmstra]

Yeah, we are aware but simply didn't want to rewrite history. Should we close this issue for now and reopen when it becomes a problem?

[jaimergp]

One "problem" already is the influx of billing warnings in the notifications for org admins. When this moves to conda it might incur in actual charges too.

[baszalmstra]

@jaimergp Do you have a suggestion on how we could "fix" this without having to rewrite all history?

@jezdez Is this a serious problem we have to tackle before graduating?

[jaimergp]

FYI, upon transfer we got this notification

Git LFS has been disabled on the organization conda-incubator because you’ve exceeded your data plan by at least 150%. Please purchase additional data packs to cover your bandwidth and storage usage:

https://github.com/organizations/conda-incubator/billing/data/upgrade

Current usage as of 13 Aug 2024 03:25PM CEST:

Bandwidth: 54.71 GB / 1 GB (5471%)
Storage: 0.0 GB / 1 GB (0%)

[wolfv]

Is this still a problem?

[jaimergp]

I don't have admin access in this org, so I can't tell. Someone would need to go and visit https://github.com/organizations/conda/billing/data/upgrade to tell us. If you didn't do anything about it, the LFS traffic is still there and someone is paying for it.

[wolfv]

We did something about it at some point, but I am not sure if we still pull historic LFS data.

[wolfv]

Going to close this as I didn't hear any further from @jezdez or others.

[jezdez]

We're using 1.21 of 1 GB of data storage and 4 of 1 GB of traffic. So this isn't resolved as such, just not breaking the conda org. I'll buy a data pack now to cover for this.

[jezdez]

Done, this is now 50 GB for traffic and storage for the conda org per month. Please let me know if you need more, preferable ahead of time via a ticket in conda/infra so we have a paper trail.

Also kind reminder that these resources are shared with all conda org projects.

[wolfv]

Maybe we should add "lfs: false" to the ci pipelines