fix(deps): update rust crate text-splitter to 0.22

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
text-splitter	dependencies	minor	0.17 -> 0.22

---

Release Notes

benbrandt/text-splitter (text-splitter)

v0.22.0

Compare Source

Breaking Changes

• Revert change to special token behavior in v0.21. This had many unintended side effects, and does not seem to be recommended for chunking.

v0.21.0

Compare Source

Breaking Changes

• Special tokens are now also encoded by both Huggingface and Tiktoken tokenizers. This is closer to the default behavior on the Python side, and should make sure if a model adds tokens at the beginning or end of a sequence, these are accounted for as well. This is especially important for embedding models that can add a special token to the beginning of the sequence, and the chunks generated didn't actually fit within the context window because of this.

What's New

Rust

• MSRV is now 1.80 to remove dependency on once_cell.

v0.20.1

Compare Source

Fixes

• Python: correctly specify version for compatibility with uv installations.

v0.20.0

Compare Source

Breaking Changes

• Switched backing Unicode segmentation implementation from unicode-segmentation to icu_segmenter. This brings some modest performance gains, along with being able to leverage the official Unicode crate. There may be slight differences in chunk behavior in some edge cases, so treating this as a breaking change.

v0.19.1

Compare Source

What's New

• Python splitters have new chunk_all and chunk_all_indices method so the multiple texts can be processed in parallel. (For Rust, you should be able to use rayon to do this already)

v0.19.0

Compare Source

Breaking Changes

• Update to tokenizers v0.21

v0.18.1

Compare Source

What's New

• Ensure tokenizer sizers with truncation parameters count their overflow encodings

v0.18.0

Compare Source

Breaking

• Change supported tiktoken-rs version to 0.6.x

---

Configuration

📅 Schedule: Branch creation - "after 1am every 3 weeks on Saturday" in timezone America/Los_Angeles, Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.