edu.stanford.nlp.process
Class TokenizerAdapter

java.lang.Object
  edu.stanford.nlp.process.AbstractTokenizer
      edu.stanford.nlp.process.TokenizerAdapter

All Implemented Interfaces:

Tokenizer, Iterator

Direct Known Subclasses:

PennTreebankTokenizer

public class TokenizerAdapter
extends AbstractTokenizer

This class adapts between a java.io.StreamTokenizer and a edu.stanford.nlp.process.Tokenizer.

Author:

Christopher Manning

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

TokenizerAdapter(StreamTokenizer st)
Create a new TokenizerAdaptor.

Method Summary

Object	getNext() Internally fetches the next token.
boolean	isEol(String str) Say whether the String is the end-of-line token for this tokenizer.
void	setEolString(String eolString) Set the String returned when the inner tokenizer returns an end-of-line token.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerAdapter

public TokenizerAdapter(StreamTokenizer st)

Create a new TokenizerAdaptor. In general, it is recommended that the passed in StreamTokenizer should have had resetSyntax() done to it, so that numbers are returned as entered as tokens of type String, though this code will cope as best it can.

Parameters:

st - The internal java.io.StreamTokenizer

Method Detail

getNext

public Object getNext()

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer

Returns:

the next token in the token stream, or null if none exists.

setEolString

public void setEolString(String eolString)

Set the String returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to eolIsSignificant(true).

Parameters:

eolString - The String used to represent eol. It is not allowed to be null (which would confuse line ends and file end)

isEol

public boolean isEol(String str)

Say whether the String is the end-of-line token for this tokenizer.

Parameters:

str - The String being tested

Returns:

Whether it is the end-of-line token