Update 'qualifiers' rule in core spec #382 #398
Conversation
Reference: #382 Signed-off-by: John M. Horan <johnmhoran@gmail.com>
johnmhoran
added
PURL core specification
PURL-SPECIFICATION.rst
Outdated
|
|
||
| - The ``key`` MUST be composed only of ASCII letters and numbers, '.', '-' and | ||
| '_' (period, dash and underscore). | ||
| - A ``key`` MUST start with an ASCII letter. |
There was a problem hiding this comment.
A
keyMUST start with an ASCII letter.
this is a change.
old:
A
keycannot start with a number
the new spec would prevent keys like _foo, -bar, .bazz
There was a problem hiding this comment.
I think this is a change I like and that makes sense, and I surmise that there are no sane qualifiers out there that do start with _, -, or .. For the sake of backward compatibility, we could either:
- keep the original, updated:
- A ``key`` MUST NOT start with a number. - keep this improved form:
- A ``key`` MUST start with an ASCII letter.
Unless there is a strong group of PURLs in the wild that exist, I would prefer the improved form.
There was a problem hiding this comment.
Given the comments thus far ^, I'm keeping as is unless/until I hear otherwise.
PURL-SPECIFICATION.rst
Outdated
| - The ``key`` MUST be composed only of ASCII letters and numbers, '.', '-' and | ||
| '_' (period, dash and underscore). | ||
| - A ``key`` MUST start with an ASCII letter. | ||
| - A ``key`` MUST NOT be percent-encoded. |
There was a problem hiding this comment.
A
keyMUST NOT be percent-encoded.
I think this is wrong, and against other existing spec
purl-spec/PURL-SPECIFICATION.rst
Lines 245 to 248 in 8040ff0
I think a key can be percent-encoded. and at some points, it must be percent encoded.
Otherwise, if percent-encoding MUST NOT happen, then i could not choose certain keys.
examples: foo&bar[]=bazz -> foo%26bar%5B%5D -- at least the & MUST be percent-encoded.
I might be wrong, please help me understand.
There was a problem hiding this comment.
foo&bar[]=bazz is not a valid key according to the preceding rules. The allowed characters shouldn't require percent encoding but I don't see why the spec would forbid percent encoding.
There was a problem hiding this comment.
The current spec already spells this out clearly, so this is not changing anything:
- A ``key`` must NOT be percent-encoded
Now since the % sign is not allowed in a key, technically, this is implied already and we could remove this sentence entirely. I like this to be explicit though.
There was a problem hiding this comment.
Given the comments thus far ^, I'm keeping as is unless/until I hear otherwise.
PURL-SPECIFICATION.rst
Outdated
| - The ``value`` MUST be composed only of the following characters, encoded | ||
| as described below and in keeping with RFC 3986 Section 2.2: |
There was a problem hiding this comment.
This is confusing and self contradictory. "The value MUST be composed only of the following characters", but then the following text defines a set containing all characters.
There was a problem hiding this comment.
Thanks @matt-phylum -- I don't think it's confusing but in any event the revised update will be simplified and clarified.
PURL-SPECIFICATION.rst
Outdated
|
|
||
| .. code-block:: none | ||
|
|
||
| '!', '$', '&', ''', '(', ')', '*', '+', ',', ';', '=' |
There was a problem hiding this comment.
The previous item lists characters that should not be encoded (MUST NOT for canonical PURLs), but this item lists some characters that MUST be encoded and some characters that should not be encoded together.
There was a problem hiding this comment.
Thanks, will look forward to your comments on the revised update once I've pushed it.
PURL-SPECIFICATION.rst
Outdated
| - The ``value`` MUST be composed only of the following characters, encoded | ||
| as described below and in keeping with RFC 3986 Section 2.2: | ||
|
|
||
| "If data for a URI component would conflict with a reserved character's | ||
| purpose as a delimiter, then the conflicting data must be percent-encoded | ||
| before the URI is formed." | ||
| https://datatracker.ietf.org/doc/html/rfc3986#section-2.2 | ||
|
|
||
| 1. **All US-ASCII characters defined as "unreserved"** in RFC 3986 Section 2.3 | ||
| (https://datatracker.ietf.org/doc/html/rfc3986#section-2.3): | ||
|
|
||
| .. code-block:: none | ||
|
|
||
| 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', '~' | ||
|
|
||
| - These 66 characters do not need to be percent-encoded. | ||
|
|
||
| 2. **All US-ASCII characters defined as "sub-delims"**, a subset of | ||
| the "reserved" characters, in RFC 3986 Section 2.2 | ||
| (https://datatracker.ietf.org/doc/html/rfc3986#section-2.2): | ||
|
|
||
| .. code-block:: none | ||
|
|
||
| '!', '$', '&', ''', '(', ')', '*', '+', ',', ';', '=' | ||
|
|
||
| - The '&' MUST be percent-encoded to avoid being incorrectly parsed | ||
| as a separator between multiple key-value pairs. See "How to parse | ||
| a purl string in its components" ("Split the qualifiers on '&'. | ||
| Each part is a key=value pair"). | ||
|
|
||
| - The other 10 characters do not need to be percent-encoded, including | ||
| the '=' -- the parser will not mistakenly treat a '=' in the value | ||
| as a separator because it splits each key-value pair just once, | ||
| from left-to-right, on the first '=' it encounters, and thus there | ||
| is no conflict: | ||
|
|
||
| .. code-block:: none | ||
|
|
||
| - For each pair, split the key=value once from left on '=': | ||
| - The key is the lowercase left side | ||
| - The value is the percent-decoded right side | ||
|
|
||
| 3. **Four additional US-ASCII characters** identified in the "query" | ||
| definition in RFC 3986 Section 3.4 (https://datatracker.ietf.org/doc/html/rfc3986#section-3.4) | ||
| and the "pchar" definition in RFC 3986 Appendix A (https://datatracker.ietf.org/doc/html/rfc3986#appendix-A): | ||
|
|
||
| .. code-block:: none | ||
|
|
||
| ':', '@', '/', '?' | ||
|
|
||
| - The '?' MUST be percent-encoded to avoid being incorrectly parsed | ||
| as a ``qualifiers`` separator -- in the right-to-left parsing | ||
| (see "How to parse a purl string in its components"), an unencoded | ||
| '?' in the ``value`` would be the first '?' encountered by the | ||
| parser and incorrectly treated as a ``qualifiers`` separator. | ||
|
|
||
| - The other three characters do not need to be percent-encoded. | ||
|
|
||
| 4. **All other US-ASCII characters**. | ||
|
|
||
| .. code-block:: none | ||
|
|
||
| - 33 control characters (ASCII codes 0-31 and 127) | ||
|
|
||
| - the 14 US-ASCII characters not covered in the preceding groups of US-ASCII characters: | ||
|
|
||
| ' ' [space], '"', '#', '%', '<', '>', '[', '\', ']', '^', '`', '{', '|', '}' | ||
|
|
||
| - Each of these 47 US-ASCII characters MUST be percent-encoded. | ||
|
|
||
| 5. **Any character that is not a US-ASCII character** | ||
| (i.e., characters with ASCII code > 127 and non-ASCII characters). | ||
|
|
||
| - All such characters MUST be UTF-8 encoded and then percent-encoded. |
There was a problem hiding this comment.
This is way too complicated to be one bullet point in the qualifiers section.
The encoding rules are:
- The ASCII control characters 0 through 31 and 127 and any non-ASCII character greater than or equal to 128 MUST always be encoded in any component of a PURL.
%which MUST always be encoded in any percent encoded string.- The additional characters
"<>SHOULD always be encoded in any component of a PURL. - The additional characters
#@?MUST be encoded for qualifier values.
Plus should also be encoded to avoid interoperability problems: #261
There was a problem hiding this comment.
Will be clarified in the update.
PURL-SPECIFICATION.rst
Outdated
| '?' in the ``value`` would be the first '?' encountered by the | ||
| parser and incorrectly treated as a ``qualifiers`` separator. | ||
|
|
||
| - The other three characters do not need to be percent-encoded. |
There was a problem hiding this comment.
This contradicts preexisting rules:
purl-spec/PURL-SPECIFICATION.rst
Lines 238 to 241 in 8040ff0
There was a problem hiding this comment.
Will be clarified in the update.
PURL-SPECIFICATION.rst
Outdated
| - ``purl`` parsers MUST return an error when the ``key`` or ``value`` contains | ||
| a prohibited character. |
There was a problem hiding this comment.
This is incorrect and incompatible with the preexisting spec.
In the unescaped form, no characters are prohibited, so you could have a valid PURL pkg:generic/name?key=%00 and the parser must handle this (or maybe it returns an error because it's string type can't represent null characters).
In the escaped form, some characters are escaped to avoid problems unrelated to the URL. http://example.com/?key=a "value" and http://example.com/?key=a%20%22value%22 are the same, but if you write Go to http://example.com/?key=a "value" or <a href="http://example.com/?key=a "value""> it doesn't work. At least I'm pretty sure why it's done. These characters have no meaning to the URL or PURL parsers so whether they are escaped or not doesn't make a difference to the parsing result.
There was a problem hiding this comment.
Seems to me that inclusion of a prohibited character could only be "normalized" by removing/discarding such a character, which does not sound to me like mere normalization. I look to feedback from @pombredanne and others on this point.
There was a problem hiding this comment.
Maybe I'm misunderstanding. There are no characters that are prohibited in the unencoded form of a qualifier value. In the encoded form, there are characters that are listed above as requiring escaping that don't actually require escaping in order for the PURL to be parsed successfully. If it can be unambiguously parsed and the result can be formatted into a PURL with the same meaning, I don't think that should be a "MUST return an error" condition. The characters do not need to be removed or discarded.
There was a problem hiding this comment.
Hmmm. I must admit that I don't yet have a good understanding of when a violation of the spec should be normalized (allowing the parsing process to continue) vs. treating a spec violation as invalid (i.e., "is_invalid": true, raising an error/exception and thus halting the parsing process).
I'd be interested in reading any authorities/resources you could point me to on this important topic -- and in all events I'll happily defer to whatever approach you, @pombredanne, @jkowalleck, @mprpic and others in the community ultimately agree to.
PURL-SPECIFICATION.rst
Outdated
|
|
||
| ' ' [space], '"', '#', '%', '<', '>', '[', '\', ']', '^', '`', '{', '|', '}' | ||
|
|
||
| - Each of these 47 US-ASCII characters MUST be percent-encoded. |
There was a problem hiding this comment.
This is a change to the PURL canonicalization rules and will cause problems for anyone who is using canonicalized PURLs as keys.
There was a problem hiding this comment.
Will be clarified in the update.
There was a problem hiding this comment.
@matt-phylum As you'll see elsewhere in these comments, I've deleted what was intended as a simple placeholder rule hoping people would provide language they wanted. Failing that, it's been deleted.
|
@jkowalleck @matt-phylum Thanks very much for your comments. After touching base with @pombredanne, I'm going to give some thought to a substantially simplified approach re permitted characters and required/prohibited encoding and will update this PR accordingly. This will include:
This next draft will be similar to the language I proposed early last week in the "Remove section on Character Encoding" PR -- see #389 (comment) |
There was a problem hiding this comment.
Thanks!
Here is some partial feedback. You may also want to consider moving the encoding details to an encoding section after all, that will be referenced by other components? I wonder also if we can streamline this section, as basically all non "unreserved" ASCII chars should be percent encoded
PURL-SPECIFICATION.rst
Outdated
| - The '?' separator is not part of the ``qualifiers`` component. | ||
| - The ``qualifiers`` component is a query string composed of one or more ``key=value`` | ||
| pairs, each of which is separated by an ampersand '&'. A ``key`` and ``value`` | ||
| are separated by the equal '=' character. |
There was a problem hiding this comment.
| are separated by the equal '=' character. | |
| - A ``key`` and ``value`` MUST be separated by the equal '=' character. |
PURL-SPECIFICATION.rst
Outdated
| - The '&' separator is not part of the ``key`` or the ``value``. | ||
| - The '=' separator is not part of the ``key`` or the ``value``. | ||
| - The ``key`` MUST be unique among the keys of the ``qualifiers`` string. | ||
| - The ``value`` MUST NOT be an empty string: a ``key=value`` pair with an empty ``value`` |
There was a problem hiding this comment.
| - The ``value`` MUST NOT be an empty string: a ``key=value`` pair with an empty ``value`` | |
| - A ``value`` MUST NOT be an empty string: a ``key=value`` pair with an empty ``value`` |
PURL-SPECIFICATION.rst
Outdated
|
|
||
| - The ``key`` MUST be composed only of ASCII letters and numbers, '.', '-' and | ||
| '_' (period, dash and underscore). | ||
| - A ``key`` MUST start with an ASCII letter. |
There was a problem hiding this comment.
I think this is a change I like and that makes sense, and I surmise that there are no sane qualifiers out there that do start with _, -, or .. For the sake of backward compatibility, we could either:
- keep the original, updated:
- A ``key`` MUST NOT start with a number. - keep this improved form:
- A ``key`` MUST start with an ASCII letter.
Unless there is a strong group of PURLs in the wild that exist, I would prefer the improved form.
PURL-SPECIFICATION.rst
Outdated
| - The ``key`` MUST be composed only of ASCII letters and numbers, '.', '-' and | ||
| '_' (period, dash and underscore). | ||
| - A ``key`` MUST start with an ASCII letter. | ||
| - A ``key`` MUST NOT be percent-encoded. |
There was a problem hiding this comment.
The current spec already spells this out clearly, so this is not changing anything:
- A ``key`` must NOT be percent-encoded
Now since the % sign is not allowed in a key, technically, this is implied already and we could remove this sentence entirely. I like this to be explicit though.
Reference: #382 Signed-off-by: John M. Horan johnmhoran@gmail.com
|
@pombredanne @jkowalleck @matt-phylum and other colleagues: As long as we're still trying to figure out what we intend the rules to permit, require, prohibit etc., what do you all think about this language I proposed a few weeks ago in PR 389? This could fit in the |
Reference: #382 Signed-off-by: John M. Horan <johnmhoran@gmail.com>
|
@pombredanne @jkowalleck @matt-phylum @mprpic I've just pushed an update with a simplified "qualifiers" subsection and a revised "Character encoding" section. Many of the "MUST"s etc. are mere placeholders (e.g., (Please don't hesitate to include proposed language where you see a need for corrections, clarifications and the like.) |
| * - '/' | ||
| - ``type``/``namespace``/``name`` and ``subpath`` segments separator | ||
| * - '=' | ||
| - ``qualifiers`` ``key``/``value`` separator |
There was a problem hiding this comment.
add
* - '&'
- ``qualifiers`` ``key=value`` separatorThere was a problem hiding this comment.
Thanks for catching this @jkowalleck -- 👍 I've added it to the update of this PR that I'll be pushing shortly.
PURL-SPECIFICATION.rst
Outdated
| MUST be percent-encoded as described in the "Character encoding" | ||
| section below[.] [, except that: | ||
|
|
||
| - *list exceptions to the updated "Character encoding" rules, e.g., if a |
There was a problem hiding this comment.
TODO // to be decided
colon ':' does not need to, or SHOULD NOT, or MUST NOT, be percent-encoded.*]
There was a problem hiding this comment.
If i remember correctly,
from existing rules, there is no "MUST NOT".
there was a rule that allowed to percent-encode everything, except the well-known delimiters in their explicit locations.
I personally liked that, since this made it easy to program/codify the rules - when in doubt, just percent-encode.
and in general: percent-decode everything.
There was a problem hiding this comment.
Yes. It's easy to implement non-canonical formatting if you're allowed to percent-encode additional characters, and it's easier to implement a parser if you are just decoding and not checking if there were extra characters that shouldn't have been encoded, but the implementation will fail the test suite and users that do string comparisons on PURLs will get false negatives when comparing the output of the implementation with the output of another implementation.
Should the canonical format stuff be separated out somehow? You MUST NOT encode : ever in the canonical form, but if you don't care about the canonical form it's a SHOULD NOT encode except for the first colon which remains a MUST NOT because the parsers really don't care about unnecessary encoding. It'd close a lot of implementation bugs where characters are not being encoded exactly right, but I worry about confusion and usability problems if people are using PURLs produced from multiple implementations. AFAIK URL does not have the same canonical form feature and instead you're expected to parse URL before comparing it. It'd be a much bigger change than this PR.
There was a problem hiding this comment.
A value in the colon has always been spec'ed as encoded
@matt-phylum re:
Should the canonical format stuff be separated out somehow?
can you move this to a new issue or a discussion?
There was a problem hiding this comment.
You mean a colon in a value has always been spec'ed as encoded? It has not. The spec has said that it is unencoded since at least 8 years ago, and there were tests that it was not to be encoded.
purl-spec/PURL-SPECIFICATION.rst
Lines 231 to 232 in 7f7e82f
The tests were broken by the recent #364.
There was a problem hiding this comment.
imho that language is not clear because line 231 refers expressly to the colon separator, not the literal colon character wherever it might be used in the purl. This could easily be clarified -- IF that's the intent. Is it? All we need is some agreement. 🙂
@johnmhoran I would avoid making breaking changes by adding "must" language for error handling that did not exist before at all. |
|
Thanks @pombredanne and that's fine, I'm just looking for agreement on what the language should be to accurately reflect the rule. Please provide your suggested language at your earliest opportunity. |
PURL-SPECIFICATION.rst
Outdated
| - A ``key`` MUST start with an ASCII letter. | ||
| - A ``key`` MUST NOT be percent-encoded. | ||
| - A ``key`` is case insensitive. The canonical form is lowercase. | ||
| - A ``value`` MAY be composed of any ASCII or non-ASCII character, and |
There was a problem hiding this comment.
@johnmhoran where does "any ASCII or non-ASCII character" comes from? Can you clarify?
There was a problem hiding this comment.
@pombredanne The "Character encoding" section already refers to ASCII and non-ASCII -- that's where my references come from. https://github.com/package-url/purl-spec/blob/main/PURL-SPECIFICATION.rst#character-encoding
There was a problem hiding this comment.
@pombredanne If I'm reading the "blame" data correctly, this language was already in the Character encoding section when you added your 2017-11-13 commit:
For clarity and simplicity a `purl` is always an ASCII string.
To ensure that there is no ambiguity when parsing a `purl`, separator
characters and non-ASCII characters must be UTF-encoded and then
percent-encoded as defined at::
See 840410b
There was a problem hiding this comment.
@pombredanne FWIW:
- RFC 3986 has 15 references to "ASCII", including 9 to "US-ASCII" (referring to DEC 0-127) and 4 to "non-ASCII" (e.g., "non-ASCII data", "non-ASCII registered names", "[n]on-ASCII characters").
- The "Normative Reference" for US-ASCII (RFC 3986 section 10.1) is "American National Standards Institute, "Coded Character Set -- 7-bit American Standard Code for Information Interchange", ANSI X3.4, 1986."
- There is no definition of "non-ASCII".
There was a problem hiding this comment.
@johnmhoran Thanks for all the details. Here is my take:
| - A ``value`` MAY be composed of any ASCII or non-ASCII character, and | |
| - A ``value`` MAY be composed of any character. A value MUST be percent-encoded as described in the "Character encoding" section. |
Then we can clarify the "Character encoding" in that section, not here.
There was a problem hiding this comment.
@pombredanne Done. As a result of this change, with the current language in "Character encoding", all characters used in a qualifiers value must be percent-encoded except:
(1) an unreserved character as defined in RFC 3986 section 2.3 (https://datatracker.ietf.org/doc/html/rfc3986#section-2.3), or
(2) expressly defined in this PURL-SPECIFICATION.rst as a purl separator (and only when used as such a separator).
That would mean that with the current "Character encoding" language, the colon ':', for example, MUST be percent-encoded in the qualifiers value. So, are you contemplating:
- adding exceptions to the
qualifierssection (that's the language I added and at your request just removed), or - adding exceptions to the "Character encoding" section (presumably this would include the exceptions all components that have 1+ exceptions), or
- rewriting the default
MUST be percent-encodedin the "Character encoding" section, or - some other approach?
We could also just direct users to read the wikipedia percent-encoding page and/or RFC 3986, but both contain ambiguities and we'd need to keep fielding new issues and PRs from puzzled users/potential users of the purl spec. I vote for clarity instead.
There was a problem hiding this comment.
Encoding colon in any part of the PURL (or any other currently unencoded character) changes the canonical form in a way that will cause problems for some users and shouldn't be done without good reason. One of the reasons I originally got involved with PURL implementations was related to problems caused by inconsistent colon encoding across the "canonical" forms of different implementations.
There was a problem hiding this comment.
Thanks @matt-phylum. I've just pushed my updates in response to several of @pombredanne's and other comments.
Can you please propose language for the qualifiers section and the "Character encoding" section that will (1) accomplish your goals and (2) add sufficient detail and breadth to enable a user to figure out the encoding of any permitted character without needing to open an issue?
Reference: #382 Signed-off-by: John M. Horan <johnmhoran@gmail.com>
|
@pombredanne Re
I've deleted that bullet I added in a recent update to this PR. As I noted when I commented on adding this new bullet a week or so ago, this was meant only as a placeholder, not a proposed rule, so that people could comment and provide suggested language. I still think clarity in error-handling would be helpful to ensure all implementations are in sync with one another (i.e., when to normalize and forgive, and maybe explain and educate the user on what we did vs. raising an exception and halting the process). As far as breaking changes, that's not my or our goal, but in bringing clarity to a somewhat ambiguous and incomplete spec, there are bound to be clarifications that are inconsistent with the range of interpretations people have been forced to make on their own. |
Reference: #382 Signed-off-by: John M. Horan <johnmhoran@gmail.com>
|
@pombredanne @jkowalleck @matt-phylum I've pushed a few requested changes to the |
|
This is how existing implementations handle escaping in qualifier values: Tables for implementationsalthonos/packageurl.rs
anchore/packageurl-go
package-url/packageurl-js
giterlizzi/perl-URI-PackageURL
phylum-dev/purl
package-url/packageurl-go
sonatype/package-url-javaDNF. An inappropriate exception is thrown during formatting. package-url/packageurl-python
package-url/packageurl-ruby
package-url/packageurl-php
maennchen/purl
package-url/packageurl-java
package-url/packageurl-dotnet
package-url/packageurl-swift
ComparisonThe value is shown when all implementations agree. A warning is shown when implementations differ. A bomb is shown when one or more implementation behaves incorrectly. packageurl-java produces incorrect percent encoding for any code point less than 0x10: package-url/packageurl-java#160 maenchenn/purl encodes 0x08 as if it were packageurl-dotnet, packageurl-ruby, packageurl-go incorrectly encode spaces (0x20) as plus signs: #261 package-url/packageurl-swift does not encode
Encoding problemsDetailsanchore/packageurl-go giterlizzi/perl-URI-PackageURL package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby are the only implementations to encode '!' '(' ')' (0x21 0x28 0x29). The PURL spec does not mention these characters. They are part of the RFC 3986 sub-delims set and are therefore not required to be encoded in a URI query string. They are not part of the WHATWG URL query percent-encode set and are therefore not required to be encoded in a URL query string. package-url/packageurl-swift is the only implementation to not encode '"' (0x22). The PURL spec does not mention this character. It is not part of either the RFC 3986 reserved or unreserved sets and is therefore supposed to be encoded in a URI query string. It is part of the WHATWG URL query percent-encode set and is therefore supposed to be encoded in a URL query string. package-url/packageurl-swift is also the only implementation that failed to encode '%' (0x25), which is a bug as mentioned previously. anchore/packageurl-go giterlizzi/perl-URI-PackageURL package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby package-url/packageurl-swift are the only implementations to encode '$' (0x24). The PURL spec does not mention this character. It is part of the RFC 3986 sub-delims set and is therefore not required to be encoded in a URI query string. It is not part of the WHATWG URL query percent-encode set and is therefore not required to be encoded in a URL query string. althonos/packageurl.rs phylum-dev/purl sonatype/package-url-java package-url/packageurl-swift are the only implementations to not encode '&' (0x26). The PURL spec does not mention this character, but it should because obviously '&' needs to be encoded within qualifier values to prevent ambiguity when parsing a PURL. RFC 3986 puts it in sub-delims and WHATWG URL does not include it in any percent-encode set, because in both of those specs the query part of the URL is left as instead of being parsed into a dictionary. anchore/packageurl-go giterlizzi/perl-URI-PackageURL package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby are the only implementations to encode "'" and ',' (0x27 and 0x2c). The PURL spec does not mention these characters. They are part of the RFC 3986 sub-delims set and are therefore not required to be encoded in a URI query string. They are not part of the WHATWG URL query percent-encode set, but 0x27 is part of the special-query percent-encode set, so whether these characters are supposed to be encoded if a PURL is a WHATWG URL depends on whether anchore/packageurl-go giterlizzi/perl-URI-PackageURL package-url/packageurl-go package-url/packageurl-java package-url/packageurl-php package-url/packageurl-python are the only implementations to encode '*' (0x2a). The PURL spec does not mention this character. It is part of the RFC 3986 sub-delims set and is therefore not required to be encoded in a URI query string. It is not part of the WHATWG URL query percent-encode set and is therefore not required to be encoded in a URL query string. althonos/packageurl.rs sonatype/package-url-java are the only implementations that do not encode '+' (0x2b). See #261. anchore/packageurl-go package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-ruby are the only implementations to encode '/' (0x2f). The PURL spec says that this character must not be encoded. It is part of the RFC 3986 query set and is therefore explicitly not required to be encoded in a URI query string. It is not part of the WHATWG URL query percent-encode set and is therefore not required to be encoded in a URL query string. anchore/packageurl-go package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-ruby are the only implementations to encode ':' (0x3a). The PURL spec says that this character must not be encoded. It is part of the RFC 3986 pchar set and is therefore not required to be encoded in a URI query string. It is not part of the WHATWG URL query percent-encode set and is therefore not required to be encoded in a URL query string. althonos/packageurl.rs anchore/packageurl-go giterlizzi/perl-URI-PackageURL package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby are the only implementations to encode ';' (0x3b). It is part of the RFC 3986 sub-delims set and is therefore not required to be encoded in a URI query string. It is not part of the WHATWG URL query percent-encode set and is therefore not required to be encoded in a URL query string. althonos/packageurl.rs anchore/packageurl-go giterlizzi/perl-URI-PackageURL package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby package-url/packageurl-swift are the only implementations to encode '=' (0x3d). Similar to '&', it is not required to be encoded by RFC 3986 or WHATWG URL because of differences in what it means to parse a PURL vs a URI or URL. PURL is written such that it is not required to be encoded but isn't very clear about whether it is supposed to be or not. althonos/packageurl.rs anchore/packageurl-go giterlizzi/perl-URI-PackageURL maennchen/purl package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby are the only implementations to encode '[' '' ']' '{' '}' (0x5b 0x5c 0x5d 0x7b 0x7d). The PURL spec does not mention these characters. RFC 3986 says that these characters should be encoded in a URI query string. WHATWG URL says that these characters are not required to be encoded in a URL query string. althonos/packageurl.rs anchore/packageurl-go giterlizzi/perl-URI-PackageURL maennchen/purl package-url/packageurl-dotnet package-url/packageurl-go package-url/packageurl-java package-url/packageurl-js package-url/packageurl-php package-url/packageurl-python package-url/packageurl-ruby package-url/packageurl-swift are the only implementations to encode '^' '`' '|' (0x5e 0x60 0x7c). The PURL spec does not mention these characters. They are not part of either the RFC 3986 reserved or unreserved sets and are therefore supposed to be encoded in a URI query string. They are not part of the WHATWG URL query percent-encode set and are therefore not required to be encoded in a URL query string. package-url/packageurl-dotnet package-url/packageurl-js package-url/packageurl-ruby package-url/packageurl-swift are the only implementations that escape ❔ Unspecified
'&' and '=' are marked as N/A because RFC 3986 and WHATWG URL parse the query string out of the URI or URL as a single value. These characters have special meaning to PURL because PURL continues to parse until the qualifiers are name value pairs. |
|
Thanks @matt-phylum -- this is very cool, and amazingly informative. Naturally, several questions come to mind, including:
|
|
@matt-phylum From your chart the descriptions for the x and checkmark are the same. Presumable the definition for the checkmark should be: "Specifies that the character should be encoded in a qualifier value or query string." |
Reference: #382
@pombredanne @jkowalleck @mprpic @matt-phylum I've updated the
qualifiersrule in the core spec's "Rules for each purl component" section, still need toqualifiers-related items need to be addressed infaq.rstqualifiers-related tests (those that already exist all look good, but I'll add a few"is_invalid": truetests for the encoding details I've included in thequalifiersupdate).Turning to these ^ in the morning.