lexnlp.nlp.en.transforms.tokens
: Transforming text into token-oriented features¶
The lexnlp.nlp.en.transforms.tokens
module contains methods
that transform text into token distributions or related feature vectors.
Attention
The sections below are a work in progress. Thank you for your patience while we continue to expand and improve our documentation coverage.
If you have any questions in the meantime, please feel free to log issues on GitHub at the URL below or contact us at the email below:
- GitHub issues: https://github.com/LexPredict/lexpredict-lexnlp
- Email: support@contraxsuite.com
lexnlp.nlp.en.transforms.tokens Module¶
Transforms related to tokens for English
Functions¶
get_bigram_distribution (text[, lowercase, …]) |
Get bigram distribution from text. |
get_ngram_distribution (text, n[, lowercase, …]) |
Get n-gram distribution of text, potentially lowercasing and stopwording first. |
get_skipgram_distribution (text, n, k[, …]) |
Get skipgram distribution from text. |
get_token_distribution (text[, lowercase, …]) |
Get token distribution of text, potentially lowercasing and stopwording first. |
get_tokens ((text[, lowercase, stopword]) |
Get token generator from text. |
get_trigram_distribution (text[, lowercase, …]) |
Get trigram distribution from text. |
Variables¶
MODULE_PATH |
str(object=’‘) -> str |