get_token_list

lexnlp.nlp.en.tokens.get_token_list(text: str, lowercase: bool = False, stopword: bool = False, preserve_line: bool = True) → List

Get token list from text. :param text: :param lowercase: :param stopword: :param preserve_line: keep the preserve the sentence and not sentence tokenize it. :return: