将字符串和字符串列表提供给 keras 标记器有什么区别？

Question

I am working with keras.preprocessing for tokenize sentences, I encountered an unexpected case in keras.preprocessing.text.Tokenize . When I give it string, the output of word_index is a dictionary of single characters and their indexes but for list the output of word_index is dictionary of words (spllited by space).
Why this happen?
String for tokenizer input:

from keras.preprocessing.text import Tokenizer

text = "Keras is a deep learning and neural networks API by François Chollet" 
tokenizer = Tokenizer()
tokenizer.fit_on_texts(text) #input of tokenizer as string

print(tokenizer.word_index)

>>> {'e': 1, 'a': 2, 'n': 3, 'r': 4, 's': 5, 'i': 6, 'l': 7, 'o': 8, 'k': 9, 'd': 10, 'p': 11, 't': 12, 'g': 13,
     'u': 14, 'w': 15, 'b': 16, 'y': 17, 'f': 18, 'ç': 19, 'c': 20, 'h': 21}

List for tokenizer input:

from keras.preprocessing.text import Tokenizer

text = ["Keras is a deep learning and neural networks API by François Chollet"] 
tokenizer = Tokenizer()
tokenizer.fit_on_texts(text) #input of tokenizer as list

print(tokenizer.word_index)

>>> {'keras': 1, 'is': 2, 'a': 3, 'deep': 4, 'learning': 5, 'and': 6, 'neural': 7, 'networks': 8,
     'api': 9, 'by': 10, 'françois': 11, 'chollet': 12}

Answer 1

The docs state to use a list of strings or a list of list of strings. There is no mention of whether you are allowed to pass a string as input, so it's possible that what you're doing is undefined behaviour that isn't getting caught.

When you pass a string as input, it looks like Keras interprets it to be a character level tokenization. Either way, if you wanted to perform a character level tokenization, it's much better to pass char_level=True when you are instantiating the Tokenizer class.

TL;DR: Don't pass a string. The docs don't mention it as a legal argument. There exists a legal way of performing character level tokenization.