get_tokens

lexnlp.nlp.en.tokens.get_tokens(text, lowercase=False, stopword=False, preserve_line=True) → Generator

Get token generator from text. :param text: :param lowercase: :param stopword: :param preserve_line: keep the preserve the sentence and not sentence tokenize it. :return: