Update C++ tokenizer to match rag_tokenizer.py#3357

yingfeng merged 1 commit intoinfiniflow:maininfiniflow/infinity:mainfrom

qinling0210:update_c++_tokenizerqinling0210/infinity:update_c++_tokenizerCopy head branch name to clipboard