module `onnxrt.ops_cpu.op_tokenizer`#

Short summary#

module mlprodict.onnxrt.ops_cpu.op_tokenizer

Runtime operator.

class	truncated documentation
`Tokenizer`	See Tokenizer.
`TokenizerSchema`	Defines a schema for operators added in this package such as `TreeEnsembleClassifierDouble`.

property	truncated documentation
`args_default`	Returns the list of arguments as well as the list of parameters with the default values (close to the signature). …
`args_default_modified`	Returns the list of modified parameters.
`args_mandatory`	Returns the list of optional arguments.
`args_optional`	Returns the list of optional arguments.
`atts_value`	Returns all parameters in a dictionary.

method	truncated documentation
`__init__`
`__init__`
`_find_custom_operator_schema`
`_run`
`_run_char_tokenization`	Tokenizes y charaters.
`_run_regex_tokenization`	Tokenizes using separators. The function should use a trie to find text.
`_run_sep_tokenization`	Tokenizes using separators. The function should use a trie to find text.
`_run_tokenization`	Tokenizes a char level.