Skip to content

Commit

Permalink
remove gensim dependency
Browse files Browse the repository at this point in the history
  • Loading branch information
djstrong committed Nov 29, 2023
1 parent 128e58f commit d422197
Showing 1 changed file with 0 additions and 4 deletions.
4 changes: 0 additions & 4 deletions tests/test_tokenizers.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,6 @@ def test_word_ninja_tokenizer2():
tokenized_names = tokenizer.tokenize('yorknewyork123')
assert ('york', 'newyork', '123') in tokenized_names

print(tokenizer.tokenize('vitalik'))
print(tokenizer.tokenize('avsa'))
print(tokenizer.tokenize('brantly'))


@mark.parametrize(
"name,tokenized_name", [
Expand Down

0 comments on commit d422197

Please sign in to comment.