Package weka.filters.unsupervised.attribute

Class FixedDictionaryStringToWordVector

  • All Implemented Interfaces:
    java.io.Serializable, CapabilitiesHandler, CapabilitiesIgnorer, CommandlineRunnable, EnvironmentHandler, OptionHandler, RevisionHandler, WeightedInstancesHandler, StreamableFilter, UnsupervisedFilter
    public class FixedDictionaryStringToWordVector
extends SimpleStreamFilter
implements UnsupervisedFilter, EnvironmentHandler, WeightedInstancesHandler
    Converts String attributes into a set of attributes representing word occurrence (depending on the tokenizer) information from the text contained in the strings. The set of words (attributes) is taken from a user-supplied dictionary, either in plain text form or as a serialized java object.

    Valid options are: 

      -dictionary <path to dictionary file>
  The path to the dictionary to use
 
      -binary-dict
  Dictionary file contains a binary serialized dictionary
 
      -C
  Output word counts rather than boolean 0 or 1 (indicating presence or absence of a word
 
      -R <range>
  Specify range of attributes to act on. This is a comma separated list of attribute
  indices, with "first" and "last" valid values.
 
      -V
  Set attributes selection mode. If false, only selected attributes in the range will
  be worked on. If true, only non-selected attributes will be processed
 
      -P <attribute name prefix>
  Specify a prefix for the created attribute names (default: "")
 
      -T
  Set whether the word frequencies should be transformed into
  log(1+fij), where fij is the frequency of word i in document (instance) j.
 
      -I
  Set whether the word frequencies in a document should be transformed into
  fij*log(num of Docs/num of docs with word i), where fij is the frequency
  of word i in document (instance) j.
 
      -N
  Whether to normalize to average length of documents seen during dictionary construction
 
      -L
  Convert all tokens to lowercase when matching against dictionary entries.
 
      -stemmer <spec>
  The stemming algorithm (classname plus parameters) to use.
 
      -stopwords-handler <spec>
  The stopwords handler to use (default = Null)
 
      -tokenizer <spec>
  The tokenizing algorithm (classname plus parameters) to use.
  (default: weka.core.tokenizers.WordTokenizer)
 
      -output-debug-info
  If set, filter is run in debug mode and
  may output additional info to the console
 
      -do-not-check-capabilities
  If set, filter capabilities are not checked before filter is built
  (use with caution).
    Version:
    $Revision: 14534 $
    Author:
    Mark Hall (mhall{[at]}pentaho{[dot]}com)
    See Also:
    Serialized Form

    • Constructor Detail

      • FixedDictionaryStringToWordVector

        public FixedDictionaryStringToWordVector()

    • Method Detail

      • getDictionaryHandler

        public DictionaryBuilder getDictionaryHandler()
        Get the dictionary builder used to manage the dictionary and perform the actual vectorization
        Returns:
        the DictionaryBuilder in use

      • setDictionarySource

        public void setDictionarySource​(java.io.InputStream source)
        Set an input stream to load a binary serialized dictionary from, rather than source it from a file
        Parameters:
        source - the input stream to read the dictionary from

      • setDictionarySource

        public void setDictionarySource​(java.io.Reader source)
        Set an input reader to load a textual dictionary from, rather than source it from a file
        Parameters:
        source - the input reader to read the dictionary from

      • getDictionaryFile

        public java.io.File getDictionaryFile()
        Get the dictionary file to read from
        Returns:
        the dictionary file to read from

      • getDictionaryIsBinary

        public boolean getDictionaryIsBinary()

      • getOutputWordCounts

        public boolean getOutputWordCounts()
        Gets whether output instances contain 0 or 1 indicating word presence, or word counts.
        Returns:
        true if word counts should be output.

      • getAttributeIndices

        public java.lang.String getAttributeIndices()
        Gets the current range selection.
        Returns:
        a string containing a comma separated list of ranges

      • setAttributeIndices

        @OptionMetadata(displayName="Range of attributes to operate on",
                description="Specify range of attributes to act on. This is a comma separated list of attribute\nindices, with \"first\" and \"last\" valid values.",
                commandLineParamName="R",
                commandLineParamSynopsis="-R <range>",
                displayOrder=4)
public void setAttributeIndices​(java.lang.String rangeList)
        Sets which attributes are to be worked on.
        Parameters:
        rangeList - a string representing the list of attributes. Since the string will typically come from a user, attributes are indexed from 1.
        eg: first-3,5,6-last
        Throws:
        java.lang.IllegalArgumentException - if an invalid range list is supplied

      • getInvertSelection

        public boolean getInvertSelection()
        Gets whether the supplied columns are to be processed or skipped.
        Returns:
        true if the supplied columns will be kept

      • getAttributeNamePrefix

        public java.lang.String getAttributeNamePrefix()
        Get the attribute name prefix.
        Returns:
        The current attribute name prefix.

      • getTFTransform

        public boolean getTFTransform()
        Gets whether if the word frequencies should be transformed into log(1+fij) where fij is the frequency of word i in document(instance) j.
        Returns:
        true if word frequencies are to be transformed.

      • setTFTransform

        @OptionMetadata(displayName="TFT transform",
                description="Set whether the word frequencies should be transformed into\nlog(1+fij), where fij is the frequency of word i in document (instance) j.",
                commandLineParamName="T",
                commandLineParamSynopsis="-T",
                displayOrder=7)
public void setTFTransform​(boolean TFTransform)
        Sets whether if the word frequencies should be transformed into log(1+fij) where fij is the frequency of word i in document(instance) j.
        Parameters:
        TFTransform - true if word frequencies are to be transformed.

      • getIDFTransform

        public boolean getIDFTransform()
        Sets whether if the word frequencies in a document should be transformed into:
        fij*log(num of Docs/num of Docs with word i)
        where fij is the frequency of word i in document(instance) j.
        Returns:
        true if the word frequencies are to be transformed.

      • setIDFTransform

        @OptionMetadata(displayName="IDF transform",
                description="Set whether the word frequencies in a document should be transformed into\nfij*log(num of Docs/num of docs with word i), where fij is the frequency\nof word i in document (instance) j.",
                commandLineParamName="I",
                commandLineParamSynopsis="-I",
                displayOrder=8)
public void setIDFTransform​(boolean IDFTransform)
        Sets whether if the word frequencies in a document should be transformed into:
        fij*log(num of Docs/num of Docs with word i)
        where fij is the frequency of word i in document(instance) j.
        Parameters:
        IDFTransform - true if the word frequecies are to be transformed

      • getNormalizeDocLength

        public boolean getNormalizeDocLength()
        Gets whether if the word frequencies for a document (instance) should be normalized or not.
        Returns:
        true if word frequencies are to be normalized.

      • getLowerCaseTokens

        public boolean getLowerCaseTokens()
        Gets whether if the tokens are to be downcased or not.
        Returns:
        true if the tokens are to be downcased.

      • getStemmer

        public Stemmer getStemmer()
        Returns the current stemming algorithm, null if none is used.
        Returns:
        the current stemming algorithm, null if none set

      • getStopwordsHandler

        public StopwordsHandler getStopwordsHandler()
        Gets the stopwords handler.
        Returns:
        the stopwords handler

      • getTokenizer

        public Tokenizer getTokenizer()
        Returns the current tokenizer algorithm.
        Returns:
        the current tokenizer algorithm

      • globalInfo

        public java.lang.String globalInfo()
        Description copied from class: SimpleFilter
        Returns a string describing this filter.
        Specified by:
        globalInfo in class SimpleFilter
        Returns:
        a description of the filter suitable for displaying in the explorer/experimenter gui

      • main

        public static void main​(java.lang.String[] args)