lexnlp.nlp.en.transforms.tokens: Transforming text into token-oriented features

The lexnlp.nlp.en.transforms.tokens module contains methods that transform text into token distributions or related feature vectors.

Attention

The sections below are a work in progress. Thank you for your patience while we continue to expand and improve our documentation coverage.

If you have any questions in the meantime, please feel free to log issues on GitHub at the URL below or contact us at the email below:

lexnlp.nlp.en.transforms.tokens Module

Transforms related to tokens for English

Functions

get_bigram_distribution(text[, lowercase, …]) Get bigram distribution from text.
get_ngram_distribution(text, n[, lowercase, …]) Get n-gram distribution of text, potentially lowercasing and stopwording first.
get_skipgram_distribution(text, n, k[, …]) Get skipgram distribution from text.
get_token_distribution(text[, lowercase, …]) Get token distribution of text, potentially lowercasing and stopwording first.
get_tokens((text[, lowercase, stopword]) Get token generator from text.
get_trigram_distribution(text[, lowercase, …]) Get trigram distribution from text.

Variables

MODULE_PATH str(object=’‘) -> str