libstring-tokenizer-perl - simple string tokenizer

Distribution: Debian 8 (Jessie)
Repository: Debian Main amd64
Package name: libstring-tokenizer-perl
Package version: 0.05
Package release: 1
Package architecture: all
Package type: deb
Installed size: 80 B
Download size: 14.97 KB
Official Mirror:
String::Tokenizer is a simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways. This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple split / /, $string and the other options that involve much larger and complex modules. Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.




    Source package: libstring-tokenizer-perl

    Install Howto

    1. Update the package index:
      # sudo apt-get update
    2. Install libstring-tokenizer-perl deb package:
      # sudo apt-get install libstring-tokenizer-perl


    • /usr/share/doc/libstring-tokenizer-perl/changelog.Debian.gz
    • /usr/share/doc/libstring-tokenizer-perl/changelog.gz
    • /usr/share/doc/libstring-tokenizer-perl/copyright
    • /usr/share/man/man3/String::Tokenizer.3pm.gz
    • /usr/share/perl5/String/


    2011-08-02 - Ben Webb <> libstring-tokenizer-perl (0.05-1) unstable; urgency=low * Initial Release. (closes: #636288)