replace `InCB_Extend_table` search with simple predicates (#178) According to [UCD](https://www.unicode.org/reports/tr44/), the property `InCB=Extend` is defined as: ``` \p{InCB = Extend} ≔ [ \p{gcb=Extend} \p{gcb=ZWJ} - \p{InCB=Linker} - \p{InCB=Consonant} - [\u200C] ] ``` And no `InCB=Consonant` is `gcb=Extend` or `gcb=ZWJ`. So in can be determined with the grapheme category the caller already has and two equality. `scripts/unicode.py` now checks that derivation against the UCD. A future Unicode version that breaks either fact fails the generator instead of silently mis-segmenting Indic text. Also one fact that is `InCB=Linker` and `InCB=Extend` are subset of `gcb=Extend`, any other category rules out both without inspecting the codepoint. This change improves the overall performance of the grapheme APIs by skipping unnecessary category checks and table searches in the hot loop. On my machine, it measures approximately -15 to -20%.
Iterators which split strings on Grapheme Cluster, Word, or Sentence boundaries, according to the Unicode Standard Annex #29 rules.
use unicode_segmentation::UnicodeSegmentation; fn main() { let s = "a̐éö̲\r\n"; let g = s.graphemes(true).collect::<Vec<&str>>(); let b: &[_] = &["a̐", "é", "ö̲", "\r\n"]; assert_eq!(g, b); let s = "The quick (\"brown\") fox can't jump 32.3 feet, right?"; let w = s.unicode_words().collect::<Vec<&str>>(); let b: &[_] = &["The", "quick", "brown", "fox", "can't", "jump", "32.3", "feet", "right"]; assert_eq!(w, b); let s = "The quick (\"brown\") fox"; let w = s.split_word_bounds().collect::<Vec<&str>>(); let b: &[_] = &["The", " ", "quick", " ", "(", "\"", "brown", "\"", ")", " ", "fox"]; assert_eq!(w, b); }
unicode-segmentation does not depend on libstd, so it can be used in crates with the #![no_std] attribute.
You can use this package in your project by adding the following to your Cargo.toml:
[dependencies] unicode-segmentation = "1"
Yanked due to accidental breakage and MSRV mistag.
#[inline] opportunities, resulting in 15-40% performance improvement.GraphemeCursor API allows random access and bidirectional iteration.as_str methods to the iterator types.