feat: add a rattler_pypi_interop crate

[travishathaway]

Description

Fixes: #1044

This is still a draft and so far, I have just copied over the rattler_installs_package crate from rip. I'm initially planning to prune this crate and remove some unnecessary pieces. So far, this consists of the pieces of the code which perform solving of pypi dependencies. This would presumably be most things in the src/resolve directory of this crate.

[baszalmstra]

I really hope you can trim it down to the bare essentials. This is a lot of code to maintain.

[iamthebot]

As a suggestion... since this may still be a lot of code even after pruning, this might be easier to submit as a sequence of PRs that incrementally adds the needed functionality. That'll be easier to review + merge as well.

[travishathaway]

@iamthebot,

I'll explore that. I recently removed a bunch of modules we initially won't need, but if that's too much still, I'll come up with a game plan for making this review (and maybe more) more manageable.

[tdejager]

Hey @travishathaway, excellent initial work on the first steps so far. I still feel as @iamthebot said, that it might be better to split up the parts into different pypi-* crates that we could review better and integrate better. Browsing through the PR now, its quite hard to have a high quality review, and I even helped write some of that code.

I think a structure could potentially be:

• pypi-types: containing artifact types and such.
• pypi-gateway: for working with the PyPI indexes.
• python-env (not sure about this name) for inspecting the python environment.

We might need to also write down the goals for the crates. AFAIK from your current work you need to mostly translate an existing environment into a locked down version, in which case you would I think only need to hit the PyPI index for a simple request or not at all. That said having functionality to hit the index, cache the results, and build wheels might be interesting for other use cases. So I'd guess its just a matter of what we want in rattler not really sure if thats been written down yet? Like what actual functionality do we actually require in rattler.

We might also want a test that verifies if the metadata can be correctly retrieved for the 500 most popular python projects ore something.

A last note, not sure if I'm reading the code correctly, but it seems that with the current code we won't be able to retrieve metadata for sdists that do not have these statically available right?

Feel free to chime in @baszalmstra @wolfv

[travishathaway]

Hi @tdejager,

To answer you first question, the code that retrieved sdist metadata from the index relied on building to the sdist in order to get it. I think this is because it needed to do that in order to transform the sdist into a wheel. I could go back and look at this and see how I could re-write the code in order to make that not necessary.

I do like your idea about coming up with a design document for this as something that would give us a high level overview of these crates and what they are meant to be used for. I would be happy to write something like this. Is there a particular format you prefer for this? I usually use hackmd.io

Regarding the adding additional crates, I'm not opposed to it, but I'm also not sure if it's necessary. I could create a series of separate pull requests though. I'll just have to do it in the right order because the modules within the crates depend on each other.