TcLex documentation - About tcLex
Author: Frédéric BONNET
Description: TcLex is a lexer (lexical analyzer) generator extension to Tcl
Keywords: text processing, lexical analyzer, parser, pattern matching, regular expression

## About tcLex

### Introduction

TcLex is a lexer (lexical analyzer) generator extension to Tcl. It is inspired by Unix and GNU lex and flex, which are "tools for generating programs that perform pattern-matching on text". TcLex is very similar to these programs, except it uses Tcl philosophy and syntax, whereas the others use their own syntax and are used in conjunction with the C language. People used to lex or flex should then feel familiar with tcLex.

TcLex is a small extension (the Windows compiled version is about 20kb, and the source is about 150kb), because it extensively uses the Tcl library. However, the current doesn't use Tcl's regexp code anymore but a patched version is now included in tcLex, which makes it slightly bigger (by a few KB).

TcLex should work with Tcl 8.0 and later. TcLex will NEVER work with earlier versions, because it uses Tcl 8.0's "object" system for performance.

The most interesting features are:

- cross-platform support, thanks to Tcl. Though it has been developped on Windows and tested on Windows and Unix only, it should work on other platforms as long as Tcl exists on these platforms. Supported Tcl platforms are Windows 95/NT, Unix (Linux, Solaris...) and Macintosh. Other platforms are VMS, OS/2, NeXTStep, Amiga...
- unlike lex and flex, which only generate static lexers written in C and intended to be compiled, tcLex dynamically generates Tcl commands that can be used like other C commands or Tcl procedures from within Tcl scripts or C programs.
- it uses Tcl regular expressions. That means you don't have to learn another regexp language.
- it works with Tcl namespaces
- the generated lexer commands can be used in one pass or incrementally, because they maintain state information. That way, several instances of the same lexer (eg a HTML parser) can run at the same time in distinct call frames and maintain distinct states (local variables...). Lexer need not be specially designed in order to be used incrementally, the same lexer can transparently be used in one pass or incrementally. This feature is especially useful when processing text from a file or an Internet socket (Web pages for example), when data is not necessarily available at the beginning of the processing.

### Comparison with flex

Flex is a tool for generating lexical analyzers using the C language. It uses its own syntax to specify rules, start conditions, regular expressions... Rules codes are written in C. Flex is a command-line tool that takes lexer definition files as input and outputs C source files.

TcLex resembles flex on many points, but its philosophy differs slightly. tcLex is intended to be used from within Tcl rather than as an external command-line tool. Rather than a specific syntax, it uses Tcl syntax to specify lexers constructs using regular Tcl strings, lists, and substitution rules. As a regular Tcl command, it can be used to generate lexers on the fly rather than statically generate Tcl sources. Lexers then become new Tcl control structures like procs and switches that can be used from any Tcl script.

While flex is most suited for performance-tuned lexers writing, tcLex is perfect for interactively exploring the power of lexers using higher level concepts. TcLex compares to flex the same way Tcl compares to C, and goes beyond by turning lexers into a new control structure, rather than keeping them a tool for string processing. Thanks to the many concepts shared with flex, tcLex can also be regarded as a tool for rapid prototyping of lexers intended to be converted to flex, the same way Tcl can be used for rapid prototyping of applications which in turn can be partly rewritten in C for performance issues.

### Philosophy

Contrary to flex, tcLex allows programmers to write lexers with the ease of use allowed by the Tcl scripting language: dynamically, interactively, programmatically, rather than statically. Such written lexers are immediately available as regular Tcl commands, and thus can be tested interactively from within a Tcl shell. The rather high-level nature of lexers perfectly suits Tcl philosophy.

TcLex was designed as a powerful mixture of Tcl commands regexp, switch and proc. It borrows many of its syntactical constructs to the switch command, extending it to introduce many concepts taken from flex. Lexers are made available to programmers as new Tcl commands that can be transparently used in a wide variety of situations, the same way proc creates new command from Tcl scripts.