|
|||||||||
PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES |
See:
Description
Interface Summary | |
---|---|
DependencyGrammar | An interface for DependencyGrammars. |
Lexicon | An interface for lexicons interfacing to lexparser. |
TreebankLangParserParams | Contains language-specific methods necessary to get the parser to parse an arbitrary treebank. |
WordSegmenter | An interface for segmenting strings into words (in unwordsegmented languages). |
Class Summary | |
---|---|
AbstractDependencyGrammar | An abstract base class for dependency grammars. |
AbstractTreebankParserParams | An abstract class providing a common method base from which to
complete a TreebankLangParserParams implementing class. |
BaseLexicon | This is the default concrete instantiation of the Lexicon interface. |
BinaryGrammar | Maintains efficient indexing of binary grammar rules. |
BinaryRule | Binary rules (ints for parent, left and right children) |
ChineseCharacterBasedLexicon | |
ChineseLexicon | A lexicon class for Chinese. |
ChineseLexiconAndWordSegmenter | This class lets you train a lexicon and segmenter at the same time. |
ChineseMarkovWordSegmenter | Performs word segmentation with a hierarchical markov model over POS and over characters given POS. |
ChineseSimWordAvgDepGrammar | A Dependency grammar that smooths by averaging over similar words. |
ChineseTreebankParserParams | Parameter file for parsing the Penn Chinese Treebank. |
ChineseUnknownWordModel | Stores, trains, and scores with an unknown word model. |
CNFTransformers | |
EnglishTreebankParserParams | Parser parameters for the Penn English Treebank (WSJ, Brown, Switchboard). |
EnglishTreebankParserParams.EnglishTest | |
EnglishTreebankParserParams.EnglishTrain | |
ExhaustivePCFGParser | An exhaustive generalized CKY PCFG parser. |
FactoredParser | |
GermanUnknownWordModel | An unknown word model for German. |
GrammarCompactor | |
IntTaggedWord | |
IterativeCKYPCFGParser | Does iterative deepening search inside the CKY algorithm for faster parsing. |
LatticeReader | |
LatticeReader.LatticeWord | |
LexicalizedParser | This class provides the top-level API and command-line interface to a set of reasonably good treebank-trained parsers. |
MaxMatchSegmenter | A word-segmentation scheme using the max-match algorithm |
MLEDependencyGrammar | |
NegraPennTreebankParserParams | Parameter file for parsing the Penn Treebank format of the Negra Treebank (German). |
Options | Options to the parser which MUST be the SAME at both training and testing (parsing) time in order for the parser to work properly. |
Options.LexOptions | |
ParentAnnotationStats | See what parent annotation helps in treebank, based on support and KL divergence. |
ParserData | Stores the serialized material representing the grammar and lexicon of a parser, and an Options that specifies things like how unknown words were handled and how distances were binned that will also be needed to parse with the grammar. |
Rule | Parent class for unary and binary rules. |
SisterAnnotationStats | See what parent annotation helps in treebank, based on support and KL divergence. |
Test | Options to the parser which affect performance only at testing (parsing) time. |
Test.Constraint | |
TransformTreeDependency | |
TreeBinarizer | Binarizes trees in such a way that head-argument structure is respected. |
TueBaDZParserParams | TreebankLangParserParams for the German Tuebingen corpus. |
UnaryGrammar | Maintains efficient indexing of binary grammar rules. |
UnaryRule | Unary Rules (with ints for parent and child) |
This package contains implementations of three parsers for natural language text. There is an accurate unlexicalized probabilistic context-free grammar (PCFG) parser, a lexical dependency parser, and a factored, lexicalized probabilistic context free grammar parser, which does joint inference over the first two parsers. For many purposes, we recommend just using the unlexicalized PCFG. With a well-engineered grammar (as supplied for English), it is fast, accurate, requires much less memory, and in many real-world uses, lexical preferences are unavailable or inaccurate across domains or genres and it will perform just as well as a lexicalized parser. However, the factored parser will sometimes provide greater accuracy through knowledge of lexical dependencies. Using the dependency parser by itself is not very useful (its accuracy is much lower).
The factored parser and the unlexicalized PCFG parser are described in:
Much of the internal guts of the parser are in one file,
FactoredParser.java
, and are not exposed as public.
The class LexicalizedParser
provides an interface for
either
training a parser from a treebank, or parsing text using a saved
parser. It can be called programmatically, or the commandline main()
method supports many options.
The parser has been ported to multiple languages. German and Chinese grammars are included. The first publication below documents the Chinese parser. The German parser was developed for and used in the second paper (but the paper contains very little detail on it).
You need Java 1.5+ installed on your system, and
java
in your PATH where commands are looked for.
You need a machine with a fair amount of memory. Required memory depends on the choice of parser, the size of the grammar, and other factors like presence of numerous unknown words To run the PCFG parser on sentences of up to 40 words you need 100 Mb of memory. To be able to handle longer sentences, you need more (to parse sentences up to 100 words, you need 400 Mb). For running the Factored Parser, 600 Mb is needed for dealing with sentences up to 40 words (which are quite typical in newsire!). Training a new lexicalized parser requires about 1500m of memory; much less is needed for training a PCFG.
For just parsing text, you need a saved parser model (grammars, lexicon,
etc.), which can be
represented either as a text file or as a binary (Java serialized
object) representation. A number are
provided (some compressed) (in /u/nlp/data/lexparser
for local
users, or in the root directory
of the distributed version).
For instance, there is englishFactored.ser.gz
and englishPCFG.ser.gz
for English, and
chineseFactored.ser.gz
and
chinesePCFG.ser.gz
for Chinese.
You need the parser code
accessible. This can be done by having the supplied
stanford-parser.jar
in your CLASSPATH.
Then if you have some sentences in testsent.txt
(as plain
text), the following
commands should work.
Parsing a local text file:
java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser
englishPCFG.ser.gz testsent.txt
Parsing a document over the web:
java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser
-maxLength 40 englishPCFG.ser.gz http://nlp.stanford.edu/~danklein/project-parsing.shtml
Note the -maxLength
flag: this will set a maximum length
sentence to parse. If you do not set one, the parser will try to parse
sentences up to any length, but will usually run out of memory when
trying to do this. This is important with web pages with text that may
not be real sentences (or just with technical documents that turn out to
have 300 word sentences).
The parser just does very rudimentary stripping of HTML tags, and
so it'll work okay on plain text web pages, but it won't work
adequately on most complex commercial script-driven pages. If you
want to handle these, you'll need to provide your own preprocessor,
and then to call the parser on its output.
Parsing a Chinese sentence (in the default input encoding of GB18030 -- and you'll need the right fonts to see the output correctly):
java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser -tLPP
edu.stanford.nlp.parser.lexparser.ChineseTreebankParserParams
chinesePCFG.ser.gz chinese-onesent
or for Unicode (UTF-8) format files:
java -mx100m edu.stanford.nlp.parser.lexparser.LexicalizedParser -tLPP
edu.stanford.nlp.parser.lexparser.ChineseTreebankParserParams
-encoding UTF-8 chinesePCFG.ser.gz chinese-onesent-utf
The parser will send parse trees to stdout
and other
information on what it is doing to stderr
, so one commonly
wants to direct just stdout
to an output file, in the
standard way.
The program has many options. The most useful end-user option is
-maxLength n
which determines the maximum
length sentence that the parser will parser. Longer sentences are
skipped, with a message printed to stderr
.
The parser supports many different input formats: tokenized/not, sentences/not, and tagged/not.
The input may be
tokenized or not, and users may supply their own tokenizers. The input
is by default assumed to not be tokenized; if the
input is tokenized, supply the option -tokenized
. If the
input is not tokenized, you may supply the name of a tokenizer class
with -tokenizer tokenizerClassName
; otherwise the default
tokenizer (edu.stanford.nlp.processor.PTBTokenizer
) is
used. This tokenizer should perform well over typical plain
newswire-style text.
The
input may have already been split into sentences or not. The input is by
default assumed
to be not split; if sentences are split, supply the option
-sentences delimitingToken
, where the delimiting token
may be any string. As a special case, if the delimiting token
is newline
the parser will assume that each line of the
file is a sentence.
The input may be in XML. The main method does not incorporate an XML
parser, but one can fake certain simple cases with the
-parseInside regex
which will only parse the tokens inside
elements matched by the regular expression regex
. These
elements are assumed to be pure CDATA.
If you use -parseInside s
, then the parser will accept
input in which sentences are marked XML-style with
<s> ... </s> (the same format as the input to
Eugene Charniak's parser).
Finally, the input may be tagged or not. If it is tagged, the program
assumes that words and tags are separated by a non-whitespace
separating character such as '/' or '_'. You give the option
-tagSeparator tagSeparator
to specify tagged text with a
tag separator.
You can see examples of many of these options in the
test
directory.
As an example, you can parse the example file with partial POS-tagging
with this command:
java edu.stanford.nlp.parser.lexparser.LexicalizedParser -maxLength 20 -sentences newline -tokenized -tagSeparator / englishPCFG.ser.gz pos-sentences.txt
There are some restrictions on the interpretation of POS-tagged input:
For the examples in pos-sentences.txt
:
For Chinese, the package includes two simple word segmenters. One is a
lexicon-based maximum match segmenter, and the other uses the parser to
do Hidden Markov Model-based word segmentation. These segmentation
methods are okay, but if you would like a high quality segmentation of
Chinese text, you will have to segment the Chinese by yourself as a
preprocessing step. The supplied grammars assume that
Chinese input has already been word-segmented according to Penn
Chinese Treebank conventions. Choosing
Chinese with -tLPP
edu.stanford.nlp.parser.lexparser.ChineseTreebankParserParams
makes space-separated words the default tokenization.
To do word segmentation within the parser, give one of the options
-segmentMarkov
or -segmentMaxMatch
.
You can set how sentences are printed out by using the
-outputFormat format
option. The native and default format is as
trees are formatted in the Penn Treebank, but there are a number of
other useful options:
penn
The default.oneline
Printed out on one line.wordsAndTags
Use the parser as a POS tagger.latexTree
Help write your LaTeX papers (for use with
Avery Andrews' trees.sty
package.typedDependenciesCollapsed
Write sentences in a typed
dependency format that represents sentences via grammatical relations
between words. Suitable for representing text as a semantic network.You can get each sentence printed in multiple formats by giving a comma-separated list of formats.
LexicalizedParser
can be easily called
within a larger
application. It implements a couple of useful interfaces that
provide for simple use:
edu.stanford.nlp.parser.ViterbiParser
and edu.stanford.nlp.process.Function
.
The following simple class shows typical usage:
import java.util.*; import edu.stanford.nlp.trees.*; import edu.stanford.nlp.parser.lexparser.LexicalizedParser; class Demo { public static void main(String[] args) { LexicalizedParser lp = new LexicalizedParser("englishPCFG.ser.gz"); String[] sent = { "This", "is", "an", "easy", "sentence", "." }; Tree parse = (Tree) lp.apply(Arrays.asList(sent)); parse.pennPrint(); System.out.println(); TreePrint tp = new TreePrint("penn,typedDependenciesCollapsed"); tp.print(parse); } }
In a usage such as this, the parser expects sentences already tokenized according to Penn Treebank conventions. For arbitrary text, prior processing must be done to achieve such tokenization (the main method of LexicalizedParser provides an example of doing this).
The current version uses class objects as temporary objects to avoid short-lived object creation and as global numberer spaces. Because of this, the parser doesn't support concurrent usage in multiple threads.
A trained parser consists of grammars, a lexicon, and option values. Once a parser has been trained, it may be written to file in one of two formats: binary serialized Java objects or human readable text data. A parser can also be quickly reconstructed (either programmatically or at the command line) from files containing a parser in either of these formats.
The binary serialized Java
objects format is created using standard tools provided by the java.io
package, and is not text, and not human-readable. To train and then save a parser
as a binary serialized objects file, use a command line invocation of the form:
java -mx1500m edu.stanford.nlp.parser.lexparser.LexicalizedParser
-train trainFilePath [fileRange] -saveToSerializedFile outputFilePath
The text data format is human readable and modifiable, and consists of four sections, appearing in the following order:
Each section is headed by a line consisting of multiple asterisks (*) and the name of the section. Note that the file format does not support rules of arbitrary arity, only binary and unary rules. To train and then save a parser as a text data file, use a command line invocation of the form:
java -mx1500m edu.stanford.nlp.parser.lexparser.LexicalizedParser
-train trainFilePath start stop -saveToTextFile outputFilePath
To parse a file with a saved parser, either in text data or serialized data format, use a command line invocation of the following form:
java -mx500m edu.stanford.nlp.parser.lexparser.LexicalizedParser
parserFilePath test.txt
If you want to use the text grammars in another parser and duplicate our performance, you will need to know how we handle the POS tagging of rare and unknown words:
For more information, you should next look at the Javadocs for the LexicalizedParser class.
|
|||||||||
PREV PACKAGE NEXT PACKAGE | FRAMES NO FRAMES |