net.sf.classifier4J
Interface ITokenizer

All Known Implementing Classes:

DefaultTokenizer

public interface ITokenizer

Author:

Peter Leschev

Method Summary

java.lang.String[]

tokenize(java.lang.String input) Splits up the string passed into the tokens which have individual probabilities.

Method Detail

tokenize

public java.lang.String[] tokenize(java.lang.String input)

Splits up the string passed into the tokens which have individual probabilities.

Returns:

Should never return null, rather it should return an empty array of Strings if there aren't any elements to return.