Why am I running out of memory tagging a lot of data (using the 2008-10-26 version)?
You’re probably using the tagString() method. Unfortunately, it does use increasing memory in this version. That method may well not be what you want anyway. It assumes that the input is correctly tokenized according to the conventions of the tagger training corpus. For the English models we use derived from the Penn Treebank, this means things like separating off contractions of “be” and “n’t”, rendering parentheses as -LRB-, -RRB-, etc. If you don’t do this correctly, then accuracy will suffer. (For no very good reason) in the 2008-09-28 distribution, the tagSentence method is set up to do tagging by using a beam search, whereas the main method of MaxentTagger and the tagSentence(Sentence) method called in TaggerDemo.java call a different Viterbi search routine to do the part-of-speech tagging. There seem to be problems with the former, and so you should use tagSentence().
Related Questions
- Can I export data (pages, content areas, applications) from a database running under one version of Oracle9iAS Portal to a database running under another version (for example, from 3.0.6 to 3.0.7)?
- When processing large data sets, KNIME seems to be running out of memory. What can I do?
- Why am I running out of memory tagging a lot of data (using the 2008-10-26 version)?
