edu.stanford.nlp.process
Class LexerTokenizer

java.lang.Object
  edu.stanford.nlp.process.AbstractTokenizer
      edu.stanford.nlp.process.LexerTokenizer

All Implemented Interfaces:

Tokenizer, Iterator

Direct Known Subclasses:

NegraPennTokenizer

public class LexerTokenizer
extends AbstractTokenizer

An implementation of Tokenizer designed to work with Lexer implementing classes. Throw in a Lexer on construction and you get a Tokenizer.

Author:

Roger Levy

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

LexerTokenizer(Lexer l)

LexerTokenizer(Lexer l, Reader r)

Method Summary

protected Object	getNext() Internally fetches the next token.
static void	main(String[] args) for testing only

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LexerTokenizer

public LexerTokenizer(Lexer l)

LexerTokenizer

public LexerTokenizer(Lexer l,
                      Reader r)

Method Detail

getNext

protected Object getNext()

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer

Returns:

the next token in the token stream, or null if none exists.

main

public static void main(String[] args)
                 throws IOException

for testing only

Throws:

IOException