As a user, I expect to be able to extract all the tokens from UAST using helpers from #363 with something like: ```go var tokens []string iter := tools.NewIterator(root, tools.PreOrder) for n := range tools.Iterate(iter) { token := uast.TokenOf(n) if token != "" { tokens = append(tokens, token) } } ``` and then by printing all the tokens get a "tokenized" version of particular file. But [uast.TokenOf](https://godoc.org/github.com/bblfsh/sdk/uast#TokenOf) returns non-empty results e.g for position nodes, with is although documents, is not very intuitive.
As a user, I expect to be able to extract all the tokens from UAST using helpers from #363 with something like:
and then by printing all the tokens get a "tokenized" version of particular file.
But uast.TokenOf returns non-empty results e.g for position nodes, with is although documents, is not very intuitive.