Package weka.core.converters

Class DictionarySaver

  • All Implemented Interfaces:
    java.io.Serializable, CapabilitiesHandler, CapabilitiesIgnorer, BatchConverter, FileSourcedConverter, IncrementalConverter, Saver, EnvironmentHandler, OptionHandler, RevisionHandler
    public class DictionarySaver
extends AbstractFileSaver
implements BatchConverter, IncrementalConverter
    Writes a dictionary constructed from string attributes in incoming instances to a destination.

    Valid options are: 

     -binary-dict
  Save as a binary serialized dictionary
 
     -R <range>
  Specify range of attributes to act on. This is a comma separated list of attribute
  indices, with "first" and "last" valid values.
 
     -V
  Set attributes selection mode. If false, only selected attributes in the range will
  be worked on. If true, only non-selected attributes will be processed
 
     -L
  Convert all tokens to lowercase when matching against dictionary entries.
 
     -stemmer <spec>
  The stemming algorithm (classname plus parameters) to use.
 
     -stopwords-handler <spec>
  The stopwords handler to use (default = Null)
 
     -tokenizer <spec>
  The tokenizing algorithm (classname plus parameters) to use.
  (default: weka.core.tokenizers.WordTokenizer)
 
     -P <integer>
  Prune the dictionary every x instances
  (default = 0 - i.e. no periodic pruning)
 
     -W <integer>
  The number of words (per class if there is a class attribute assigned) to attempt to keep.
 
     -M <integer>
  The minimum term frequency to use when pruning the dictionary
  (default = 1).
 
     -O
  If this is set, the maximum number of words and the
  minimum term frequency is not enforced on a per-class
  basis but based on the documents in all the classes
  (even if a class attribute is set).
 
     -sort
  Sort the dictionary alphabetically
 
     -i <the input file>
  The input file
 
     -o <the output file>
  The output file
    Version:
    $Revision: 12690 $
    Author:
    Mark Hall (mhall{[at]}pentaho{[dot]}com)
    See Also:
    Serialized Form

    • Constructor Detail

      • DictionarySaver

        public DictionarySaver()

    • Method Detail

      • globalInfo

        public java.lang.String globalInfo()
        Returns a string describing this Saver.
        Returns:
        a description of the Saver suitable for displaying in the explorer/experimenter gui

      • getSaveBinaryDictionary

        public boolean getSaveBinaryDictionary()
        Get whether to save the dictionary as a binary serialized dictionary, rather than a plain text one
        Returns:
        true if the dictionary is to be saved as binary rather than plain text

      • getAttributeIndices

        public java.lang.String getAttributeIndices()
        Gets the current range selection.
        Returns:
        a string containing a comma separated list of ranges

      • setAttributeIndices

        @OptionMetadata(displayName="Range of attributes to operate on",
                description="Specify range of attributes to act on. This is a comma separated list of attribute\nindices, with \"first\" and \"last\" valid values.",
                commandLineParamName="R",
                commandLineParamSynopsis="-R <range>",
                displayOrder=4)
public void setAttributeIndices​(java.lang.String rangeList)
        Sets which attributes are to be worked on.
        Parameters:
        rangeList - a string representing the list of attributes. Since the string will typically come from a user, attributes are indexed from 1.
        eg: first-3,5,6-last
        Throws:
        java.lang.IllegalArgumentException - if an invalid range list is supplied

      • getInvertSelection

        public boolean getInvertSelection()
        Gets whether the supplied columns are to be processed or skipped.
        Returns:
        true if the supplied columns will be kept

      • getLowerCaseTokens

        public boolean getLowerCaseTokens()
        Gets whether if the tokens are to be downcased or not.
        Returns:
        true if the tokens are to be downcased.

      • getStemmer

        public Stemmer getStemmer()
        Returns the current stemming algorithm, null if none is used.
        Returns:
        the current stemming algorithm, null if none set

      • getStopwordsHandler

        public StopwordsHandler getStopwordsHandler()
        Gets the stopwords handler.
        Returns:
        the stopwords handler

      • getTokenizer

        public Tokenizer getTokenizer()
        Returns the current tokenizer algorithm.
        Returns:
        the current tokenizer algorithm

      • getPeriodicPruning

        public long getPeriodicPruning()
        Gets the rate at which the dictionary is periodically pruned, as a percentage of the dataset size.
        Returns:
        the rate at which the dictionary is periodically pruned

      • setPeriodicPruning

        @OptionMetadata(displayName="Periodic pruning rate",
                description="Prune the dictionary every x instances\n(default = 0 - i.e. no periodic pruning)",
                commandLineParamName="P",
                commandLineParamSynopsis="-P <integer>",
                displayOrder=14)
public void setPeriodicPruning​(long newPeriodicPruning)
        Sets the rate at which the dictionary is periodically pruned, as a percentage of the dataset size.
        Parameters:
        newPeriodicPruning - the rate at which the dictionary is periodically pruned

      • getWordsToKeep

        public int getWordsToKeep()
        Gets the number of words (per class if there is a class attribute assigned) to attempt to keep.
        Returns:
        the target number of words in the output vector (per class if assigned).

      • setWordsToKeep

        @OptionMetadata(displayName="Number of words to attempt to keep",
                description="The number of words (per class if there is a class attribute assigned) to attempt to keep.",
                commandLineParamName="W",
                commandLineParamSynopsis="-W <integer>",
                displayOrder=15)
public void setWordsToKeep​(int newWordsToKeep)
        Sets the number of words (per class if there is a class attribute assigned) to attempt to keep.
        Parameters:
        newWordsToKeep - the target number of words in the output vector (per class if assigned).

      • getMinTermFreq

        public int getMinTermFreq()
        Get the MinTermFreq value.
        Returns:
        the MinTermFreq value.

      • getDoNotOperateOnPerClassBasis

        public boolean getDoNotOperateOnPerClassBasis()
        Get the DoNotOperateOnPerClassBasis value.
        Returns:
        the DoNotOperateOnPerClassBasis value.

      • setDoNotOperateOnPerClassBasis

        @OptionMetadata(displayName="Do not operate on a per-class basis",
                description="If this is set, the maximum number of words and the\nminimum term frequency is not enforced on a per-class\nbasis but based on the documents in all the classes\n(even if a class attribute is set).",
                commandLineParamName="O",
                commandLineParamSynopsis="-O",
                commandLineParamIsFlag=true,
                displayOrder=17)
public void setDoNotOperateOnPerClassBasis​(boolean newDoNotOperateOnPerClassBasis)
        Set the DoNotOperateOnPerClassBasis value.
        Parameters:
        newDoNotOperateOnPerClassBasis - The new DoNotOperateOnPerClassBasis value.

      • getKeepDictionarySorted

        public boolean getKeepDictionarySorted()
        Get whether to keep the dictionary sorted alphabetically or not
        Returns:
        true to keep the dictionary sorted

      • writeIncremental

        public void writeIncremental​(Instance inst)
                      throws java.io.IOException
        Description copied from class: AbstractSaver
        Method for incremental saving. Standard behaviour: no incremental saving is possible, therefore throw an IOException. An incremental saving process is stopped by calling this method with null.
        Specified by:
        writeIncremental in interface Saver
        Overrides:
        writeIncremental in class AbstractSaver
        Parameters:
        inst - the instance to be saved
        Throws:
        java.io.IOException - IOEXception if the instance acnnot be written to the specified destination

      • writeBatch

        public void writeBatch()
                throws java.io.IOException
        Description copied from class: AbstractSaver
        Writes to a file in batch mode To be overridden.
        Specified by:
        writeBatch in interface Saver
        Specified by:
        writeBatch in class AbstractSaver
        Throws:
        java.io.IOException - exception if writting is not possible

      • setDestination

        public void setDestination​(java.io.OutputStream output)
                    throws java.io.IOException
        Description copied from class: AbstractFileSaver
        Sets the destination output stream.
        Specified by:
        setDestination in interface Saver
        Overrides:
        setDestination in class AbstractFileSaver
        Parameters:
        output - the output stream.
        Throws:
        java.io.IOException - throws an IOException if destination cannot be set

      • getRevision

        public java.lang.String getRevision()
        Description copied from interface: RevisionHandler
        Returns the revision string.
        Specified by:
        getRevision in interface RevisionHandler
        Returns:
        the revision

      • main

        public static void main​(java.lang.String[] args)