Package weka.core.tokenizers

Class CharacterNGramTokenizer

  • All Implemented Interfaces:
    java.io.Serializable, java.util.Enumeration<java.lang.String>, OptionHandler, RevisionHandler
    public class CharacterNGramTokenizer
extends Tokenizer
    Splits a string into an n-gram with min and max grams.

    Valid options are: 

     -max <int>
  The max size of the Ngram (default = 3).
 
     -min <int>
  The min size of the Ngram (default = 1).
    Version:
    $Revision: 10971 $
    Author:
    Sebastian Germesin (sebastian.germesin@dfki.de), Eibe Frank (eibe@cs.waikato.ac.nz)
    See Also:
    Serialized Form

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      int getNGramMaxSize()
      Gets the max N of the NGram.
      int getNGramMinSize()
      Gets the min N of the NGram.
      java.lang.String[] getOptions()
      Gets the current option settings for the OptionHandler.
      java.lang.String getRevision()
      Returns the revision string.
      java.lang.String globalInfo()
      Returns a string describing the tokenizer
      boolean hasMoreElements()
      returns true if there's more elements available
      java.util.Enumeration<Option> listOptions()
      Returns an enumeration of all the available options..
      static void main​(java.lang.String[] args)
      Runs the tokenizer with the given options and strings to tokenize.
      java.lang.String nextElement()
      Returns N-grams and also (N-1)-grams and ....
      java.lang.String NGramMaxSizeTipText()
      Returns the tip text for this property.
      java.lang.String NGramMinSizeTipText()
      Returns the tip text for this property.
      void setNGramMaxSize​(int value)
      Sets the max size of the Ngram.
      void setNGramMinSize​(int value)
      Sets the min size of the Ngram.
      void setOptions​(java.lang.String[] options)
      Parses a given list of options.
      void tokenize​(java.lang.String s)
      Sets the string to tokenize.

      • Methods inherited from class java.lang.Object

        equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface java.util.Enumeration

        asIterator

    • Constructor Detail

      • CharacterNGramTokenizer

        public CharacterNGramTokenizer()

    • Method Detail

      • globalInfo

        public java.lang.String globalInfo()
        Returns a string describing the tokenizer
        Specified by:
        globalInfo in class Tokenizer
        Returns:
        a description suitable for displaying in the explorer/experimenter GUI

      • listOptions

        public java.util.Enumeration<Option> listOptions()
        Returns an enumeration of all the available options..
        Specified by:
        listOptions in interface OptionHandler
        Overrides:
        listOptions in class Tokenizer
        Returns:
        an enumeration of all available options.

      • getOptions

        public java.lang.String[] getOptions()
        Gets the current option settings for the OptionHandler.
        Specified by:
        getOptions in interface OptionHandler
        Overrides:
        getOptions in class Tokenizer
        Returns:
        the list of current option settings as an array of strings

      • setOptions

        public void setOptions​(java.lang.String[] options)
                throws java.lang.Exception
        Parses a given list of options.

        Valid options are: 

         -max <int>
  The max size of the Ngram (default = 3).
 
         -min <int>
  The min size of the Ngram (default = 1).
        Specified by:
        setOptions in interface OptionHandler
        Overrides:
        setOptions in class Tokenizer
        Parameters:
        options - the list of options as an array of strings
        Throws:
        java.lang.Exception - if an option is not supported

      • getNGramMaxSize

        public int getNGramMaxSize()
        Gets the max N of the NGram.
        Returns:
        the size (N) of the NGram.

      • setNGramMaxSize

        public void setNGramMaxSize​(int value)
        Sets the max size of the Ngram.
        Parameters:
        value - the size of the NGram.

      • NGramMaxSizeTipText

        public java.lang.String NGramMaxSizeTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

      • setNGramMinSize

        public void setNGramMinSize​(int value)
        Sets the min size of the Ngram.
        Parameters:
        value - the size of the NGram.

      • getNGramMinSize

        public int getNGramMinSize()
        Gets the min N of the NGram.
        Returns:
        the size (N) of the NGram.

      • NGramMinSizeTipText

        public java.lang.String NGramMinSizeTipText()
        Returns the tip text for this property.
        Returns:
        tip text for this property suitable for displaying in the explorer/experimenter gui

      • hasMoreElements

        public boolean hasMoreElements()
        returns true if there's more elements available
        Specified by:
        hasMoreElements in interface java.util.Enumeration<java.lang.String>
        Specified by:
        hasMoreElements in class Tokenizer
        Returns:
        true if there are more elements available

      • nextElement

        public java.lang.String nextElement()
        Returns N-grams and also (N-1)-grams and ....
        Specified by:
        nextElement in interface java.util.Enumeration<java.lang.String>
        Specified by:
        nextElement in class Tokenizer
        Returns:
        the next element

      • tokenize

        public void tokenize​(java.lang.String s)
        Sets the string to tokenize.
        Specified by:
        tokenize in class Tokenizer
        Parameters:
        s - the string to tokenize

      • getRevision

        public java.lang.String getRevision()
        Returns the revision string.
        Returns:
        the revision

      • main

        public static void main​(java.lang.String[] args)
        Runs the tokenizer with the given options and strings to tokenize. The tokens are printed to stdout.
        Parameters:
        args - the commandline options and strings to tokenize