|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectsofie.parsing.Tokenizer
sofie.parsing.Tokenizer.WikiTokenizer
public static class Tokenizer.WikiTokenizer
Tokenizer for Wikipedia
Nested Class Summary |
---|
Nested classes/interfaces inherited from class sofie.parsing.Tokenizer |
---|
Tokenizer.HTMLTokenizer, Tokenizer.PatternRunner, Tokenizer.WikiTokenizer |
Field Summary | |
---|---|
protected static Tokenizer.PatternRunner |
infoboxKiller
A pattern runner that eliminates all infoboxes |
protected static Tokenizer.PatternRunner |
infoboxSaver
A pattern runner that eliminates everything except infoboxes |
protected static java.util.List<Tokenizer.PatternRunner> |
wikiPatternRunners
Wikipedia-specific patterns |
Fields inherited from class sofie.parsing.Tokenizer |
---|
standardPatternRunners |
Constructor Summary | |
---|---|
Tokenizer.WikiTokenizer()
|
Method Summary | |
---|---|
java.lang.CharSequence |
load(java.io.File f)
Loads the file, decodes UTF8 |
protected java.util.List<Tokenizer.PatternRunner> |
patternRunners()
Pattern runners that are to be run for this specific type of tokenizer. |
protected java.lang.String |
prepare(java.lang.CharSequence s)
Decodes Ampersand |
java.util.List<Token> |
tokenizeNoInfobox(java.lang.CharSequence s)
Tokenizes the Wikipedia article, but removes the infobox |
java.util.List<Token> |
tokenizeNoInfobox(java.io.File f)
Tokenizes the Wikipedia article, but removes the infobox |
java.util.List<Token> |
tokenizeOnlyInfobox(java.lang.CharSequence s)
Tokenizes the Wikipedia article, but removes everything outside the infobox |
java.util.List<Token> |
tokenizeOnlyInfobox(java.io.File f)
Tokenizes the Wikipedia article, but only the infobox |
Methods inherited from class sofie.parsing.Tokenizer |
---|
autodetectTokenize, canonicalizeKnownWord, forFile, main, tokenize, tokenize, tokenizeHTML, tokenizeString, tokenizeText, tokenizeWikipedia, tokenizeWikipediaNoInfobox, tokenizeWikipediaOnlyInfobox |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected static final java.util.List<Tokenizer.PatternRunner> wikiPatternRunners
protected static final Tokenizer.PatternRunner infoboxKiller
protected static final Tokenizer.PatternRunner infoboxSaver
Constructor Detail |
---|
public Tokenizer.WikiTokenizer()
Method Detail |
---|
protected java.util.List<Tokenizer.PatternRunner> patternRunners()
Tokenizer
patternRunners
in class Tokenizer
protected java.lang.String prepare(java.lang.CharSequence s)
prepare
in class Tokenizer
public java.lang.CharSequence load(java.io.File f) throws java.io.IOException
load
in class Tokenizer
java.io.IOException
public java.util.List<Token> tokenizeNoInfobox(java.lang.CharSequence s)
public java.util.List<Token> tokenizeOnlyInfobox(java.lang.CharSequence s)
public java.util.List<Token> tokenizeNoInfobox(java.io.File f) throws java.io.IOException
java.io.IOException
public java.util.List<Token> tokenizeOnlyInfobox(java.io.File f) throws java.io.IOException
java.io.IOException
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |