Common system for remote type handling (mozilla#1865)

Remote types present an issue because we can't implement FFI traits like
`Lower` and `Lift` on them directly.  The workaround is to only
implement the FFI trait for the current crate.  If another crate
wants to use that impl, we need to implement it again by forwarding to
the initial implementation.

Historically, UDL and proc-macros have approached this differently which
has lead to a lot of extra complexity and confusion.  This change aligns
them on a common system:  By default, we implement traits for all crates
(a "blanket impl").  However, users can opt-in to only implementing it
for the current crate ("a local impl`)

See the doc changes for a description of the new system.

This required a bunch of changes:

- UDL now creates a blanket impl by default. The [Remote] attribute can
  be used to switch to a local impl.
- Proc-macros get the `#[remote]` attribute macro, which allows users to
  create a local impl.
- Custom types now create a blanket impl by default.  The `remote`
  parameter can be used to create a local impl.
- Added the `remote_type!` macro to which handles the case where you
  want to use a remote type and another crate has already created a
  local impl.  This creates a local impl for your crate by forwarding to
  the other impl.  Removed the `use_udl_*` macros, which were a kludge
  for doing the same thing.

Added a motivating example where `anyhow::Error` is used as an error
type.  Changed the error type bound to `Display + Debug` rather than
`Error`. For some reason `anyhow::Error` doesn't actually implement
error.

One feature that we've lost in defining callback/trait interfaces using
traits defined in a remote crate.  I think this is okay because I doubt
anyone is using that feature.

CHANGELOG.md

@@ -13,6 +13,13 @@
   We did this to help fix some edge-cases with custom types wrapping types from other crates (eg, Url).
   See https://mozilla.github.io/uniffi-rs/next/Upgrading.html for help upgrading and https://mozilla.github.io/uniffi-rs/next/udl/custom_types.html for details.
 
+- Handling of remote and external types has changed significantly:
+    - UDL users need to add the `[Remote]` attribute for remote types
+    - The `use_udl_*` macros are no longer needed and have been removed.
+    - To share remote type implementations between crates, use the `use_remote_type` macro.
+    - The UDL syntax for external types in the UDL has been changed.
+    - See https://mozilla.github.io/uniffi-rs/udl/remote_ext_types.html for details
+
 ### What's fixed?
 
 - Fixed bug in metadata extraction with large ELF files.

Cargo.toml

@@ -18,6 +18,7 @@ members = [
   "examples/custom-types",
   "examples/futures",
   "examples/geometry",
+  "examples/remote-types",
   "examples/rondpoint",
   "examples/sprites",
   "examples/todolist",

docs/manual/src/internals/lifting_and_lowering.md

@@ -108,9 +108,10 @@ To work around this we do several things:
  - We generate a unit struct named `UniFfiTag` in the root of each UniFFIed crate.
  - Each crate uses the `FfiConverter<crate::UniFfiTag>` trait to lower/lift/serialize values for its scaffolding functions.
 
-This allows us to work around the orphan rules when defining `FfiConverter` implementations.
- - UniFFI consumer crates can implement lifting/lowering/serializing types for their own scaffolding functions, for example `impl FfiConverter<crate::UniFfiTag> for serde_json::Value`.  This is allowed since `UniFfiTag` is a local type.
+This allows us to work around the orphan rules when defining ffi trait implementations.
  - The `uniffi` crate can implement lifting/lowering/serializing types for all scaffolding functions using a generic impl, for example `impl<UT> FfiConverter<UT> for u8`.  "UT" is short for "UniFFI Tag"
- - We don't currently use this, but crates can also implement lifting/lowering/serializing their local types for all scaffolding functions using a similar generic impl (`impl<UT> FfiConverter<UT> for MyLocalType`).
+ - UniFFI consumer crates usually implement lifting/lowering/serializing types the same way.
+ - However, for remote types, they must only implement ffi traits for their local tag, for example `impl FfiConverter<crate::UniFfiTag> for serde_json::Value`.  This is valid since `UniFfiTag` is a local type.
+ - If other crates also want to use the same remote type implementation, the need to implement the ffi traits for their local tag as well.  This is what the `use_remote_type!` macro does.
 
 For more details on the specifics of the "orphan rule" and why these are legal implementations, see the [Rust Chalk Book](https://rust-lang.github.io/chalk/book/clauses/coherence.html#the-orphan-rules-in-rustc)

docs/manual/src/kotlin/configuration.md

@@ -10,7 +10,7 @@ The generated Kotlin modules can be configured using a `uniffi.toml` configurati
 | `cdylib_name`                | `uniffi_{namespace}`[^1] | The name of the compiled Rust library containing the FFI implementation (not needed when using `generate --library`). |
 | `generate_immutable_records` | `false`                  | Whether to generate records with immutable fields (`val` instead of `var`). |
 | `custom_types`               |                          | A map which controls how custom types are exposed to Kotlin. See the [custom types section of the manual](../udl/custom_types.md#custom-types-in-the-bindings-code)|
-| `external_packages`          |                          | A map of packages to be used for the specified external crates. The key is the Rust crate name, the value is the Kotlin package which will be used referring to types in that crate. See the [external types section of the manual](../udl/ext_types_external.md#kotlin)
+| `external_packages`          |                          | A map of packages to be used for the specified external crates. The key is the Rust crate name, the value is the Kotlin package which will be used referring to types in that crate. See the [external types section of the manual](../udl/remote_ext_types.md#kotlin)
 | `android`                    | `false`                  | Used to toggle on Android specific optimizations
 | `android_cleaner`            | `android`                | Use the [`android.system.SystemCleaner`](https://developer.android.com/reference/android/system/SystemCleaner) instead of [`java.lang.ref.Cleaner`](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/ref/Cleaner.html). Fallback in both instances is the one shipped with JNA.
 | `kotlin_target_version`      | `"x.y.z"`                | When provided, it will enable features in the bindings supported for this version. The build process will fail if an invalid format is used.

docs/manual/src/proc_macro/index.md

@@ -454,39 +454,15 @@ pub trait Person {
 // }
 ```
 
-## Types from dependent crates
+## Conditional compilation
 
-When using proc-macros, you can use types from dependent crates in your exported library, as long as
-the dependent crate annotates the type with one of the UniFFI derives.  However, there are a couple
-exceptions:
-
-### Types from UDL-based dependent crates
-
-If the dependent crate uses a UDL file to define their types, then you must invoke one of the
-`uniffi::use_udl_*!` macros, for example:
-
-```rust
-uniffi::use_udl_record!(dependent_crate, RecordType);
-uniffi::use_udl_enum!(dependent_crate, EnumType);
-uniffi::use_udl_error!(dependent_crate, ErrorType);
-uniffi::use_udl_object!(dependent_crate, ObjectType);
-```
-
-### Non-UniFFI types from dependent crates
-
-If the dependent crate doesn't define the type in a UDL file or use one of the UniFFI derive macros,
-then it's currently not possible to use them in an proc-macro exported interface.  However, we hope
-to fix this limitation soon.
-
-## Other limitations
-
-In addition to the per-item limitations of the macros presented above, there is also currently a
-global restriction: You can only use the proc-macros inside a crate whose name is the same as the
-namespace in your UDL file. This restriction will be lifted in the future.
-
-### Conditional compilation
 `uniffi::constructor|method` will work if wrapped with `cfg_attr` attribute:
 ```rust
 #[cfg_attr(feature = "foo", uniffi::constructor)]
 ```
 Other attributes are not currently supported, see [#2000](https://github.com/mozilla/uniffi-rs/issues/2000) for more details.
+
+## Mixing UDL and proc-macros
+
+If you use both UDL and proc-macro generation, then your crate name must match the namespace in your
+UDL file. This restriction will be lifted in the future.

docs/manual/src/udl/remote_ext_types.md

@@ -0,0 +1,142 @@
+# Remote and External types
+
+Remote and external types can help solve some advanced use-cases when using UniFFI.
+They are grouped this section, since they're often used together.
+
+# Remote types
+
+"Remote types" refer to types defined in other crates that do not use UniFFI.
+This normally means types from crates that you depend on but don't control.
+Remote types require extra handling to use them in UniFFI APIs, because of Rust's [orphan rule](https://doc.rust-lang.org/book/traits.html#rules-for-implementing-traits).
+See https://github.com/mozilla/uniffi-rs/tree/main/examples/log-formatter for example code.
+
+In general, using remote types in UniFFI requires writing a type definition that mirrors the real definition found in the remote crate.
+
+## Proc-macros
+
+```rust
+
+// Type aliases can be used to give remote types nicer names when exposed in the UniFFI api.
+type LogLevel = log::Level;
+
+// Write a definition that mirrors the definition from the remote crate and wrap it with `[uniffi::remote(<kind>)]`.
+//
+// - UniFFI will generate the FFI scaffolding code for the item, but will not output the item itself
+//   (since the real item is defined in the remote crate).
+// - `<kind>` can be any parameter that's valid for `uniffi::derive()`.
+#[uniffi::remote(Enum)]
+enum LogLevel {
+    Error,
+    Warn,
+    Info,
+    Debug,
+    Trace,
+}
+```
+
+## UDL
+
+Wrap the definition with `[Remote]` attribute:
+
+```idl
+[Remote]
+enum LogLevel {
+    "Error",
+    "Warn",
+    "Info",
+    "Debug",
+    "Trace",
+};
+```
+
+# External Types
+
+"External types" refer to types defined in other crates that use UniFFI.
+This normally means types from other crates in your workspace.
+
+## Proc-macros
+
+Proc-macro-based code can use external types automatically, without any extra code.
+
+## UDL
+
+Suppose you depend on the `DemoDict` type from another UniFFIed crate in your workspace.
+You can reference this type by using the `[External]` attribute to wrap a typedef describing the concrete type.
+
+```idl
+[External]
+typedef dictionary One;
+
+// Now define our own dictionary which references the external type.
+dictionary ConsumingDict {
+  DemoDict demo_dict;
+  boolean another_bool;
+};
+```
+
+Supported values for the typedef type:
+
+* Enums: `enum`
+* Records: `record`, `dictionary` or `struct`
+* Objects: `object`, `impl` or `interface`
+* Traits: `trait`, `callback` or `trait_with_foreign`
+
+# Special cases for remote types
+
+There are a few cases where remote types require extra handling in addition to the rules above.
+
+## Remote + External types
+
+Types that are remote and external require a `use_remote_type!` macro call.
+
+If `crate_a` defines [IpAddr](https://doc.rust-lang.org/std/net/enum.IpAddr.html) as a remote type, then `crate_b` can use that type with the following Rust code:
+
+```rust
+uniffi::use_remote_type!(IpAddr, crate_a);
+```
+
+## UDL
+
+UDL-users will also need to add the external type definition:
+
+```idl
+[External]
+typedef enum IpAddr;
+```
+
+## Remote custom types
+
+Types that are remote and custom require using the `remote` attribute with the `custom_type!` macro.
+
+```rust
+
+uniffi::custom_type!(StructFromOtherCrate, String, {
+    remote,
+    lower: |s| s.to_string(),
+    try_lift: |s| StructFromOtherCrate::try_from_string(s),
+});
+```
+
+## Foreign bindings
+
+The foreign bindings will also need to know how to access the external type,
+which varies slightly for each language:
+
+### Kotlin
+
+For Kotlin, "library mode" generation with `generate --library [path-to-cdylib]` is recommended when using external types.
+If you use `generate [udl-path]` then the generated code needs to know how to import
+the external types from the Kotlin module that corresponds to the Rust crate.
+By default, UniFFI assumes that the Kotlin module name matches the Rust crate name, but this can be configured in `uniffi.toml` with an entry like this:
+
+```
+[bindings.kotlin.external_packages]
+# Map the crate names from [External={name}] into Kotlin package names
+rust-crate-name = "kotlin.package.name"
+```
+
+### Swift
+
+For Swift, you must compile all generated `.swift` files together in a single
+module since the generate code expects that it can access external types
+without importing them.