Count the frequency of each tokens in the corpus

前端 未结 0 1411
夕颜
夕颜 2021-02-12 11:55

I am using the transformers library and have a tokenizer(BPE) and I know how to use it for my texts.

I wonder how can I count the frequency of each token in my text corpu

相关标签:
回答
  • 消灭零回复
提交回复
热议问题