Tokenizer

xowiki
- Documentation
  - Miscellaneous
    - thread-mallocs
    - naviserver-connthreadqueue
  - Next Scripting Language (NX)

Assumes Tcl 8.6 (couroutine support)

if {[catch {package req Tcl 8.6}]} return

Rosetta example: Tokenize a string with escaping

Write a class which allows for splitting a string at each non-escaped occurrence of a separator character.

See https://rosettacode.org/wiki/Tokenize_a_string_with_escaping

package req nx

nx::Class create Tokenizer {
    :property s:required
    :method init {} {
        :require namespace
        set coro [coroutine [current]::nextCoro [current] iter ${:s}]
        :public object forward next $coro
    }
    :public method iter {s} {
        yield [info coroutine]
        for {set i 0} {$i < [string length $s]} {incr i} {
            yield [string index $s $i]
        }
        return -code break
    }
    :public object method tokenize {{-sep |} {-escape ^} s} {
        set t [[current] new -s $s]
        set part ""
        set parts [list]
        while {1} {
            set c [$t next]
            if {$c eq $escape} {
                append part [$t next]
            } elseif {$c eq $sep} {
                lappend parts $part
                set part ""
            } else {
                append part $c
            }
        }
        lappend parts $part
        return $parts
    }
}

Run some tests incl. the escape character:

% Tokenizer tokenize -sep | -escape ^ ^|
|
% Tokenizer tokenize -sep | -escape ^ ^|^|
||
% Tokenizer tokenize -sep | -escape ^ ^^^|
^|
% Tokenizer tokenize -sep | -escape ^ |
{} {}

Test for the output required by the Rosetta example:

% Tokenizer tokenize -sep | -escape ^ one^|uno||three^^^^|four^^^|^cuatro|
one|uno {} three^^ four^|cuatro {}

Tokenizer

Listing of doc/example-scripts/rosetta-tokenizer.tcl

Rosetta example: Tokenize a string with escaping