Package opennlp.tools.tokenize
Class TokenSample
java.lang.Object
opennlp.tools.tokenize.TokenSample
- All Implemented Interfaces:
Serializable,Sample
A
TokenSample is text with token spans.- See Also:
-
Field Summary
Fields -
Constructor Summary
ConstructorsConstructorDescriptionTokenSample(String text, Span[] tokenSpans) Initializes ainstance.TokenSample(Detokenizer detokenizer, String[] tokens) Initializes ainstancevia aDetokenizer. -
Method Summary
-
Field Details
-
DEFAULT_SEPARATOR_CHARS
- See Also:
-
-
Constructor Details
-
TokenSample
Initializes ainstance.- Parameters:
text- The text which contains the tokens. Must not benull.tokenSpans- The spans which mark the start and end of the tokens. Must not benull.
-
TokenSample
Initializes ainstancevia aDetokenizer.- Parameters:
detokenizer- The text which contains the tokens. Must not benull.tokens- The tokens to be processed. Must not benull.
-
-
Method Details
-
getText
- Returns:
- Retrieves the text.
-
getTokenSpans
- Returns:
- Retrieves the token
spans.
-
toString
-
parse
Parses a string sample.- Parameters:
sampleString- The sample to be parsed. Must not benull.separatorChars- The characters to be considered separators. SeeDEFAULT_SEPARATOR_CHARS. Must not benull.- Returns:
- A valid
TokenSampleinstance.
-
hashCode
public int hashCode() -
equals
-