edu.stanford.nlp.io
Interface Lexer

public interface Lexer

A Lexer interface to be used with LexerTokenizer. You can put a Reader inside a Lexer with the yyreset(java.io.Reader) method. An easy way to build classes implementing this interface is with JFlex (http://jflex.de). Just make sure to include the following in the JFlex source file

In the Options and Macros section of the source file, include

%class JFlexDummyLexer
%standalone
%unicode
%int

%implements edu.stanford.nlp.io.Lexer

%{
public void pushBack(int n) {
yypushback(n);
}

public int getYYEOF() {
return YYEOF;
}
%}

Alternatively, you can customize your own lexer and get lots of flexibility out.

Author:

Roger Levy

Field Summary

static int	ACCEPT
static int	IGNORE

Method Summary

int	getYYEOF() returns value for YYEOF
void	pushBack(int length) Pushes back length character positions in the lexer.
int	yylex() Gets the next token from input and returns an integer value signalling what to do with the token.
void	yyreset(Reader r) put a Reader inside the Lexer.
String	yytext() returns the matched input text region

Field Detail

ACCEPT

static final int ACCEPT

See Also:

Constant Field Values

IGNORE

static final int IGNORE

See Also:

Constant Field Values

Method Detail

yylex

int yylex()
          throws IOException

Gets the next token from input and returns an integer value signalling what to do with the token.

Throws:

IOException

yytext

String yytext()

returns the matched input text region

pushBack

void pushBack(int length)

Pushes back length character positions in the lexer. Conventionally used to push back exactly one token.

getYYEOF

int getYYEOF()

returns value for YYEOF

yyreset

void yyreset(Reader r)
             throws IOException

put a Reader inside the Lexer.

Throws:

IOException