Struct NlpToken 

#[non_exhaustive]
pub struct NlpToken {
    pub text: String,
    pub start: usize,
    pub end: usize,
    pub pos: Option<String>,
    pub ner: Option<String>,
    pub dep: Option<DepLink>,
    pub word_index: usize,
}

One token in an NlpResult.

Tokens are subword units; several may belong to one word (see word_index). Index-based references elsewhere (e.g. DepLink::head, SrlRole::span) are always token indices into NlpResult::tokens, not word indices.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

text: String

Surface form of the token.

start: usize

UTF-8 byte offset of the token start in the original input text.

end: usize

UTF-8 byte offset of the token end (exclusive).

pos: Option<String>

POS tag (typically Universal Dependencies tagset). None if POS not requested.

ner: Option<String>

Named-entity type (e.g. "PERSON", "ORG"). None outside any entity span.

dep: Option<DepLink>

Dependency-parse head and relation label. None if DEP not requested.

word_index: usize

Index of the word this subword token belongs to.

Dense and monotonically non-decreasing over NlpResult::tokens: consecutive tokens that form one word share a word_index, and each new word increments it by one starting at 0. Lets consumers regroup subwords into words without parsing tokenizer metaspace markers.

Trait Implementations

impl Clone for NlpToken

fn clone(&self) -> NlpToken

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NlpToken

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NlpToken

fn default() -> NlpToken

Returns the “default value” for a type.