Cap tiktoken input length in countTokens to avoid O(n^2) hang#332

stevegonzalez721-ux wants to merge 1 commit into

Conway-Research:mainConway-Research/automaton:mainfrom

stevegonzalez721-ux:fix/tiktoken-quadratic-tokenize-capstevegonzalez721-ux/automaton:fix/tiktoken-quadratic-tokenize-capCopy head branch name to clipboard