materializations: improved handling for "ambiguous" materialized fields

[williamhbaker]

Description:

The materializations for some fields may be ambiguous if the destination systems are for example case insensitive, and a field like "thisField" will be materialized as the same column as "thisfield". Presently this is a concern for materialize-redshift, which treats everything as lowercase, and materialize-snowflake because of the historical way we have quoted identifiers.

Historically we have not handled these cases well, and would end up with a error during apply because of trying to create a duplicate column. More recently, changes were made to outright forbid these ambiguous fields, and instead require alternate projections of them in order to be materialized. Forbidding the fields was simple, but excessive since users generally want to use field selection (rather than projections on collections) to pick which ambiguous field they actually want as the quick and easy way to get something working.

This change enables that by implementing the following behaviors:

• For a new binding of a materialization, ambiguous fields will all be optional if they aren't otherwise forbidden. The constraint description will say that only one of them should be selected to materialize, since we don't have any kind of mutual exclusion constraint. If a user selects more than one of these fields for a new materialization, it will fail to apply.
• Once an ambiguous field is selected for materialization, all others from that set of ambiguous fields will be forbidden. If a binding is published with no ambiguous field selected, they will all still be constrained as optional so that it is possible to select one for an existing binding.
• In the event of a pre-existing table with no previously persisted spec for that binding, the ambiguous fields will still be set as optional, because we don't know which projection the existing column should correspond to without a field selection. This is different than other pre-existing columns which we will recommend by default, but seems to be necessary given the ambiguous nature of the columns.

Also, a change is included to enforce that materialized collection keys are not null, or are nullable with a default value annotation as part of Validate calls. Materializations that use typed columns will currently fail in unspecified ways if a null value is actually present in a collection, and this ensures that Validate will not allow key fields that may be null.

Workflow steps:

Materialize a collection with fields that the destination system considers ambiguous. A blind publication of the materialization will not fail, but will also not include any of the ambiguous fields by default. Selecting one of them via field selection will work, and forbidden constraints will then result for other fields that are amiguous with the selected field.

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

There's some test snapshot update churn, since I re-ran all the tests with the latest binaries that include the flow_truncated metadata field.

---

This change is 

[jgraettinger]

LGTM

[jgraettinger]

👍 all this was no-doubt tedious to implement, but will be really helpful for users trying to understand what's happening and why.

[jgraettinger]

Noting this error is the same as the one in the new bindings case. Maybe it could be extracted into a common constant, but ... that confuses Go tooling and there's just one instance of duplication so probably best to leave as-is 🤷. Maybe just comment both to note that they're effectively the same reason and language should be changed together ?

[jgraettinger]

Lemme know if you think we need to implement this.