How to (re)use external types for Python? (Missing exports)

[tmathern]

I've been reading through the documentation for the use of foreign types: https://mozilla.github.io/uniffi-rs/0.28/udl/ext_types_external.html

I am trying to reuse types defined in this crate, here: https://github.com/contentauth/c2pa-python/tree/mathern/updates, namely this SigningAlg type.

I am trying to start another new project, a consuming_crate, which is a Rust+Python codebase, that would reuse that SigningAlg type.

My Cargo.toml to import the needed dependency in the consuming_crate looks like this:

c2pa-python = { git = "https://github.com/contentauth/c2pa-python.git", branch = "mathern/updates" }

This is what my UDL for this type looks like in my consuming crate:

[External="c2pa_python"]
typedef extern SigningAlg;

I also have the needed pub use statements in my consuming crate's lib.rs file, as well as the scaffolding:

pub use c2pa_python::SigningAlg;
uniffi::include_scaffolding!("my_new_project");

Then, in my consuming crate, I will get a generated Python file with imports on top looking like this:

from __future__ import annotations
import os
import sys
import ctypes
import enum
import struct
import contextlib
import datetime
import threading
import itertools
import traceback
import typing
import platform
from c2pa import SigningAlg
from c2pa import _UniffiConverterTypeSigningAlg
from c2pa import _UniffiRustBuffer as _UniffiRustBufferSigningAlg

Note the last two exports of _UniffiConverterTypeSigningAlg and _UniffiRustBuffer.
The thing is that the generated Python code generated in the crate I want to consume types from, c2pa-python/tree/mathern/updates, does not export these types (anywhere).

I verified that, if I manually go into the installed packages and add the needed exports, things works.

I am attaching two zip files containing the Python code:

• what-I-get.zip is what is generated by default for the c2pa repo (which doesn't work, note the exports where _UniffiConverterTypeSigningAlg and _UniffiRustBuffer are missing).
• what-I-need.zip with fixed exports, which is what I think need to actually be generated to be able to reuse types?
  what-I-get.zip
  what-I-need.zip

So I wonder if I am missing a configuration somewhere so the generation and build would work? I cannot ask everyone who installs the future Python bindings to manually do these export changes.

[mhammond]

It looks like SigningAlg is created in a crate that doesn't use UniFFI? If so, I think our docs are a little poor on that (although should be improved by #1865) but we have a couple of examples.

Eg, this enum is defined in a non-uniffi crate, and consumed by a uniffi crate. The enum needs to be duplicated (eg,

uniffi-rs/fixtures/ext-types/lib/src/ext-types-lib.udl

Line 69 in 2450637

enum ExternalCrateNonExhaustiveEnum {

) but then can be used directly (eg, see

uniffi-rs/fixtures/ext-types/lib/src/lib.rs

Line 4 in 2450637

ExternalCrateDictionary, ExternalCrateInterface, ExternalCrateNonExhaustiveEnum,

). I think there's support for that using proc-macros (ie, so instead of duplicating it in UDL you can in Rust), which I can try to dig up once I know this is the actual problem you have.

[tmathern]

Both crates use uniffi (or you need to define what "using uniffi" means), both crate are even on same maturin and uniffi versions.

[tmathern]

Also: I am not looking to reuse the type in Rust that much (that piece works), but in-between the two Python codebases.

[mhammond]

Sorry, I misunderstood.

The thing is that the generated Python code generated in the crate I want to consume types from, c2pa-python/tree/mathern/updates, does not export these types (anywhere).

You should not need to use any Python code generated by the crate you are consuming types from - the generation of your final crate (the "consuming" crate, which depends on the crate you are consuming types from) should generate all the bits correctly.

Our example at https://github.com/mozilla/uniffi-rs/tree/245063790fc16731d720dfa23b0c180029cff06b/fixtures/ext-types/lib is all about this scenario - it uses types from https://github.com/mozilla/uniffi-rs/tree/245063790fc16731d720dfa23b0c180029cff06b/fixtures/ext-types/uniffi-one but doesn't need any Python generated from it. When that final "consuming crate" has Python code generated, it should notice the multiple crates and namespaces - eg, here is our Python test for our "consuming crate" which imports the code generated for the other crates - but the code behind those imports was still generated by that final consuming crate.

[mhammond]

Note also that Python will not actually generate a valid package in this scenario as it doesn't know what should be in init.py etc. It will be necessary for you to move the .py files from their generated location into however you want your package laid out - https://mozilla.github.io/uniffi-rs/latest/python/configuration.html#external-packages talks a little about that.

[tmathern]

Note also that Python will not actually generate a valid package in this scenario as it doesn't know what should be in init.py etc. It will be necessary for you to move the .py files from their generated location into however you want your package laid out - https://mozilla.github.io/uniffi-rs/latest/python/configuration.html#external-packages talks a little about that.

Do you have working examples of that? I tried that - but could not get it to work either.

[tmathern]

Sorry, I misunderstood.

The thing is that the generated Python code generated in the crate I want to consume types from, c2pa-python/tree/mathern/updates, does not export these types (anywhere).

You should not need to use any Python code generated by the crate you are consuming types from - the generation of your final crate (the "consuming" crate, which depends on the crate you are consuming types from) should generate all the bits correctly.

Our example at https://github.com/mozilla/uniffi-rs/tree/245063790fc16731d720dfa23b0c180029cff06b/fixtures/ext-types/lib is all about this scenario - it uses types from https://github.com/mozilla/uniffi-rs/tree/245063790fc16731d720dfa23b0c180029cff06b/fixtures/ext-types/uniffi-one but doesn't need any Python generated from it. When that final "consuming crate" has Python code generated, it should notice the multiple crates and namespaces - eg, here is our Python test for our "consuming crate" which imports the code generated for the other crates - but the code behind those imports was still generated by that final consuming crate.

What if I want to make my two Python packages compatible with each other?

Both my crates use Rust in the underlying code (same deps) to share code. I was hoping there would be ways the Python packages could therefore easily work together.

When I saw the UDL doc for https://mozilla.github.io/uniffi-rs/0.28/udl/ext_types_external.html, I thought it meant that the shared types would be shared also by the generated Python?

[mhammond]

Do you have working examples of that? I tried that - but could not get it to work either.

Not directly, but https://github.com/mozilla/uniffi-rs/blob/main/fixtures/ext-types/lib/uniffi.toml is what that example uses to make itself work. In that scenario we are fine with all modules being "top-level", but expect most real consumers are likely to want a Python package.

What if I want to make my two Python packages compatible with each other?

I'm not quite sure what you mean exactly, but in your scenario, you are ending up with a single .so which contains both compiled crates - and both Python packages are created from that single .so.

Because Rust doesn't support dynamic linking (ie, it's not really possible to create your external lib as its own .so, and have that .so consumed by your "consuming" .so) we only support generation from the final target.

I thought it meant that the shared types would be shared also by the generated Python?

They are - it's just that they are shared by virtue of the fact that have been compiled into the same .so.

[tmathern]

Thanks @mhammond.
It seems though I have a follow-up question/issue: #2313