TWILL(1)

NAME
twill - twill 3.1

TWILL OVERVIEW

twill is a simple language that allows users to browse the Web from a command-line interface. With twill, you can navigate through web sites that use forms, cookies, and most standard Web features.

twill supports automated web testing and has a simple Python interface. Check out the twill Examples!

twill is open source and written in Python.

  Downloading twill

The latest release of twill is twill version 3.0. It is available for download from the Python Package Index, and you can use Python's pip tool to install or upgrade twill. Please see the ChangeLog for what's new in this version.

twill 3.0 works with Python 3.6 and newer.

To start using twill, install it and then type twill. At the prompt, type:

    go https://www.slashdot.org/
    show
    showforms
    showhistory

  Documentation

The documentation for the latest release is published online at https://twill-tools.github.io/twill/ and https://twill.readthedocs.io/.

Documentation is available for the following topics:

•   twill Examples -- some short examples.

•   Web browsing with twill -- General introduction to twill.

•   twill language reference -- the twill scripting language.

•   Testing Web sites with twill -- how to use twill to test Web sites.

•   twill Extension Modules -- extension modules that come with twill.

•   twill's Python API -- for programmers interested in using twill from Python.

•   Developing twill -- for developers interested in extending or fixing twill.

•   Propaganda, tools and packages -- projects relevant to, or based upon, twill.

  Contributing

Issues including bug reports, fixes or extensions and pull requests can be submitted on the GitHub project page.

When reporting bugs, please be sure to use the -f -l debug options with twill so that we can see the full traceback.

  Authors and License

The original author of twill is C. Titus Brown, titus@idyll.org. A number of people have contributed bug reports and code since the first release; they are acknowledged below.

The original twill source code is Copyright (C) 2005, 2006, 2007 C. Titus Brown. twill is available under the MIT license.

  Acknowledgements

In May 2004, Cory Dodt had a great idea with "Python Browser Poseur" (PBP), his insights helped C. Titus Brown. creating twill. Ian Bicking had the idea of reimplementing PBP on top of IPython (since abandoned in favor of cmd), and suggested the "in-process" hack. Grig Gheorghiu was strangely enthusiastic about a simple demo shown by Titus and promoted twill since then. John J. Lee has promptly and enthusiastically checked in various patches to mechanize created by Titus. Michele Simionato is an early adopter who has also helped quite a bit.

Bug reports have come in from the following fine people: Chris Miles, Matsuno Tokuhiro, Elvelind Grandin, Mike Rovner, sureshvv, Terry Peppers, Kieran Holland, Alexander Shvedunov, Norman Khine, Leonardo Santagada, Sebastien Pierre, Herve Cauwelier, aledain, Uy Do, David Hancock, and Tomi Hautakoski.

Patches have been submitted by Joeri van Ruth, Paul McGuire, Ed Rahn, Nic Ferrier, Robert Leftwich, James Cameron, William Volkman, Tommi Virtanen, Simon Buenzli, sureshvv, Jeff Martin, Stephen Thorne, and Bob Halley.

Features were proposed by Ben Bangert, and Tristan De Buysscher.

In April 2014, Ben Talyor created version 1.8 using requests and lxml instead of mechanize.

In July 2016, Christoph Zwerschke created version 2.0 which also integrates ideas and code from Flunc which was created by Luke Tucker and Robert Marianski in 2006-2007, and from ReTwill which was created in April 2012 as a fork from twill 0.9 by Adam Victor Brandizzi.

The official relaunch of twill on its current GitHub project page and the release of version 2.0 happened in April 2020 with the approval of its original author C. Titus Brown.

Thanks, all!

INSTALLING TWILL

The recommended way to install twill is to use pip:

    pip install twill

You can also install it directly from the distribution tar.gz file by unpacking the .tar.gz file and running:

    python setup.py install

  Troubleshooting your installation

The first and only thing you should do before creating an issue on the GitHub project page is to see if you have the twill package installed correctly:

    $ python
    >>> import twill.shell
    >>> twill.shell.main()

(This should drop you into the twill shell irrespective of whether 'twill' is on your path.) If this works, twill is installed correctly and you just need to adjust your path (see above for examples). If you still need help, or the above code doesn't work, please copy and paste the results of entering the above code into your e-mail to the list -- thanks!

  Upgrading twill

If you don't want to download a new tar.gz file, you can use pip to upgrade twill. To get the latest release, use:

    pip install -U twill

To download the latest development release (which is usually pretty stable) use:

    pip install -U https://github.com/twill-tools/twill/archive/master.zip

CHANGELOG

  3.1 (released 2022-10-30)

•   The submit command now takes an additional parameter to specify a form that can be used in rare cases when there are no form fields (#7).

•   Most commands do not return values any more, they are just commands. If you are using twill from Python, you should check browser properties like 'forms' or 'url' instead of using the return values of commands like 'show_forms' or 'back' (see #13, #14).

•   Two-word commands now consistently have underscores in their names, (e.g. 'form_action', 'get_input', 'show_links'). However, for convenience and backward compatibility, you can still use the names without underscores (e.g. 'formaction', 'getinput', 'showlinks'), and the old two-letter abbreviations (e.g. 'fa' for 'form_action') (#13, #14).

•   Instead of 'showforms' or 'show_forms' you can now also write 'show forms', and similarly for 'cookies', 'links', 'history' and 'html'. The command 'show html' does the same as 'show' without any arguments.

•   Renamed shortcuts for user agent strings, and added some more existing ones.

•   Added type hints (#15).

•   Support Python 3.11.

•   Many minor fixes and improvements.

  3.0.3 (released 2022-10-12)

•   Form numbers are now printed correctly with 'showforms' (#12).

  3.0.2 (released 2022-04-10)

•   Save HTML file with browser encoding or as UTF-8 (#9).

•   Do not modify root logger any more (#10).

  3.0.1 (released 2021-12-04)

•   This version now also supports Python 3.10.

•   The twill language now allows 8-bit letters to appear in strings unquoted.

  3.0 (released 2021-02-25)

•   In this version we require Python 3.6 to 3.9. If you still need support for Python 2 or Python 3.5, then please use the latest version from the 2.x branch.

•   The code has been optimized for Python 3 now.

•   Some minor fixes.

  2.0.3 (released 2021-02-25)

•   Backported the fixes in version 3.0.

  2.0.2 (released 2021-02-13)

•   This version now also supports Python 3.9.

•   'tidy_should_exist' has been renamed ot 'require_tidy'.

•   Support for setting options to be used with HTML Tidy.

•   Cleanup scripts are now also read as UTF-8 in Python 3.

  2.0.1 (released 2020-07-12)

•   Fixes an issue with encoding declarations (#5).

  2.0 (released 2020-04-04)

This version is based on twill 1.8, which was a refactoring of version 0.9 that used requests and lxml instead of mechanize, done by Ben Taylor in April 2014. It also integrates ideas and code from Flunc which was created by Luke Tucker and Robert Marianski in 2006-2007, and from ReTwill which was created in April 2012 as a fork from twill 0.9 by Adam Victor Brandizzi. The following improvements and changes were made in this version:

•   Larger refactoring, clean-up and modernization efforts to support Python 2.7, 3.5 and higher.

•   The console script has been renamed from 'twill-sh' to just 'twill'.

•   We assume the default file extension '.twill' for twill scripts now.

•   Uses lxml and requests instead of mechanize (like in version 1.8), but doesn't need cssselect and Beautiful Soup any more (unlike 1.8).

•   Removed bundled packages which have become unnecessary (mechanize) or are available in newer versions on PyPI (pyparsing, wsgi_intercept) or in the standard library (subprocess).

•   Removed parsing options (use_tidy, use_BeautifulSoup, allow_parse_errors) which have become insignificant due to the use of lxml.html.

•   We use pytest instead of nose for testing twill now.

•   A tox configuration file for running tests with different Python versions has been added.

•   Optimized the order of the URLs that are tried out by the twill browser.

•   Added an option '-d' to dump the last HTML to a file or standard output and an option '-w' to show the HTML directly in the web browser (this feature was taken over from Flunc).

•   Added alias 'rf' for 'runfiles' and made runfiles run directories of scripts as well. This helps writing test suites for twill scripts.

•   Added command 'add_cleanup' to unconditionally run cleanup scripts after the current script finished. This allows resetting the state of the tested server, so that tests will always re-run on a clean state. Together with a small init.twill script, this creates a test fixture. (This idea was taken from Flunc, which supports cleanup scripts for test suites, although in a somewhat different way.)

•   Non string values are now accepted in variable substitution (this feature has been backported from ReTwill).

•   XPath expressions are now supported in find/notfind commands (this feature has been backported from ReTwill).

•   Made output better controllable by using log levels (this feature has been backported from ReTwill). See options '-l' and '-o'.

•   Updated the map of predefined user agent strings.

•   Basic authentication with realm is now supported again (the 'with_default_realm' option, which was broken in version 1.8, has been switched off).

•   Server certificates are not verified by default any more, since they are usually not valid on test and staging servers.

•   Improved handling of meta refresh. Circular redirects are detected and 'debug equiv-refresh' is functional again. A limit for the refresh time interval can be set with the 'equiv_refresh_interval' option. By default this is set to 2, so refresh intervals of 2 or more seconds are ignored.

•   Moved the examples and additional stuff into an 'extras' directory.

•   The documentation in the 'docs' directory has been updated and is now created with Sphinx.

•   Made sure everything (except twill-forks) also works on Windows.

•   Fixed a lot of smaller and larger bugs and problems.

TWILL EXAMPLES

  Example: logging into slashdot

This script logs you into slashdot (assuming you have an account!):

    setlocal username <your username>
    setlocal password <your password>
    go http://www.slashdot.org/
    formvalue 1 unickname $username
    formvalue 1 upasswd $password
    submit
    code 200     # make sure form submission is correct!

  Example: searching Google and going to the first hit

Please be aware that automated searching of Google violates their Terms of Service. This is for example purposes only!:

    setlocal query "twill Python"
    go https://www.google.com/
    fv 1 q $query
    submit btnI     # use the "I'm feeling lucky" button
    show

WEB BROWSING WITH TWILL

twill strives to be a complete implementation of a web browser, omitting only JavaScript support. It includes support for cookies, basic authentication, and trickery such as "meta refresh" redirects.

twill implements a variety of commands. With the built-in language, you can do things like go to a specific URL; follow links; fill out forms and submit them; save, load, and delete cookies; and change the user agent string. You can also easily extend twill with new and specialized commands using Python.

  Using twill interactively

The twill command line script lets you interactively browse the Web. It features built-in help, e.g. "help go" will describe the command 'go' to you; command-line completion with the TAB key; and history browsing with the UP/DOWN arrow keys.

  Proxy servers

twill understands the http_proxy environment variable generically used to set proxy server information. To use a proxy in UNIX or Windows, just set the http_proxy environment variable, e.g.:

    % export http_proxy="http://www.someproxy.com:3128"

or:

    % setenv http_proxy="http://www.someotherproxy.com:3148"

  Recording scripts

Writing twill scripts is boring. One simple way to get at least a rough initial script is to use the MaxQ recorder to generate a twill script. MaxQ acts as an HTTP proxy and records all HTTP traffic; I have written a simple twill script generator for it. The script generator and installation docs are included in the twill distribution under the directory extras/maxq.

  Miscellaneous implementation details

•   twill ignores robots.txt.

•   http-equiv=refresh headers are handled immediately, independent of the 'pause' component of the 'content' attribute.

•   twill ignores JavaScript.

  Using HTML Tidy to check for broken HTML

The HTML Tidy tool can be used to check and clean up broken HTML pages. You can use it with the 'tidy_ok' command to assert that it reports no warnings or errors. In order to use this command in twill, you need to install the PyTidyLib package. If it is not installed, twill will silently ignore it. It may be desirable to require a functioning PyTidyLib installation; so, to fail when it isn't installed, set config require_tidy 1.

TWILL LANGUAGE REFERENCE

The following commands are built into twill. Note that all text after a '#' is ignored as a comment, unless it's in a quoted string.

  Browsing

go <url> -- visit the given URL.

back -- return to the previous URL.

reload -- reload the current URL.

follow <link name> -- follow the given link.

Note: When used from Python, you get the new URL after executing these commands (after possible redirects) with browser.url.

  Assertions

code <code> -- assert that the last page loaded had this HTTP status, e.g. code 200 asserts that the page loaded fine.

find <pattern> [<flags>] -- assert that the page contains the specified regular expression pattern. The variable __match__ is set to the first matching subgroup (or the entire matching string, if no subgroups are specified). When called from Python, the matching string is also returned.

Fhe command also accepts some flags: i, for case-insensitive matching, m for multi-line mode and s for dotall matching. The flag x uses XPath expressions instead of regular expressions (see below).

find <xpath> x -- assert that the page contains the element matched by the XPath expression. The variable __match__ is set to the first matching element's text content. When called from Python, the matching text is also returned.

not_find <pattern> [<flags>] -- assert that the page does not contain the specified regular expression pattern. This command accepts the same flags as find.

Alternative spelling: "notfind"

not_find <xpath> x -- assert that the page does not contain the element pointed by the XPath expression.

url <pattern> -- assert that the current URL matches the given regex pattern. The variable __match__ is set to the first matching subgroup (or the entire matching string, if no subgroups are specified). When called from Python, the matching string is returned.

title <pattern> -- assert that the title of this page matches this regular expression. The variable __match__ is set to the first matching subgroup (or the entire matching string, if no subgroups are specified). When called from Python, the matching string is returned.

  Display

echo <string> -- echo the string to the screen.

info -- display information about the current page.

redirect_output <filename> -- append all twill output to the given file.

reset_output -- display all output to the screen.

save_html [<filename>] -- save the current page's HTML into a file. If no filename is given, derive the filename from the URL.

show -- show the current page's HTML.

Alternative spelling: "showhtml" or "show html"

show_links -- show all of the links on the current page.

Alternative spellings: "showlinks" or "show links"

show_forms -- show all of the forms on the current page.

Alternative spellings: "showforms" or "show forms"

show_history -- show the browser history.

Alternative spellings: "showhistory" or "show history"

show_cookies -- show the current cookies.

Alternative spellings: "showcookies" or "show cookies"

Note: When used from Python, you get these objects using the properties of the browser with the same names, e.g. browser.html, browser.links, browser.forms, browser.history and browser.cookies.

  Forms

submit [<button_name>] -- click the submit button with the given name. Instead of the name of the button you can also specify the number of the form element that shall be used as the submit button. If you do not specify a button name or number, the form is submitted via the last submission button clicked; if nothing had been clicked, use the first submit button on the form. See details on form handling for more information. In rare cases you may need to specify the form explicitly, you can do so by adding the form name or number as an additional parameter to the button name or number.

form_value <form_name> <field_name> <value> --- set the given field in the given form to the given value. For read-only form controls, the click may be recorded for use by submit, but the value is not changed unless the 'config' command has changed the default behavior. See 'config' and details on form handling for more information on the 'form_value' command.

For list controls, you can use 'form_value <form_name> <field_name> +value' or 'form_value <form_name> <field_name> -value' to select or deselect a particular value.

Alternative spellings: "formvalue" or "fv"

fv -- abbreviation for 'form_value'.

form_action <form_name> <action> -- change the form action URL to the given URL.

Alternative spellings: "formaction" or "fa"

fa -- abbreviation for 'form_action'.

form_clear -- clear all values in the form.

Alternative spelling: "formclear"

form_file <form_name> <field_name> <filename> [ <content_type> ] -- attach a file to a file upload button by filename.

Alternative spelling: 'formfile'.

show_forms -- show all of the forms on the current page.

Alternative spellings: "showforms" or "show forms"

  Cookies

save_cookies <filename> -- save current cookie jar into a file.

load_cookies <filename> -- replace current cookie jar with file contents.

clear_cookies -- clear all of the current cookies.

show_cookies -- show all of the current cookies.

Alternative spellings: "showcookies" or "show cookies"

  Debugging

debug <what> <level> -- turn on or off debugging/tracing for various functions. The first argument is either 'http' to show HTTP headers, 'equiv-refresh' to show "meta refresh" redirects, or 'commands' to show twill commands. The second argument is '0' for off, '1' for on.

  Variable handling

set_global <name> <value> -- set variable <name> to value <value> in global dictionary. The value can be retrieved with '$value'.

Alternative spelling: "setglobal"

set_local <name> <value> -- set variable <name> to value <value> in local dictionary. The value can be retrieved with '$value'.

Alternative spelling: "setlocal"

The local dictionary is file-specific, while the global module is general to all the commands. Local variables will override global variables if they exist.

Note that you can do variable interpolation in strings with ${var}, e.g.

    setglobal a 1
    setglobal b 2
    fv thisform thatfield "${a}${b}"

  Other commands

tidy_ok -- check to see if HTML Tidy runs on this page without any errors or warnings. This check is very stringent, but you can relax the default configuration by setting tidy_* configuration options.

exit [<code>] -- exit with the given integer code, if specified. 'code' defaults to 0.

run <command> -- execute the given Python command.

run_file <file1> [ <file2> ... ] -- execute the given file(s).

Alternative spellings: "runfile" or "rf"

rf -- abbreviation for 'run_file'.

add_cleanup <file1> [ <file2> ... ] -- add the given cleanup file(s). These will be run after the current file has executed (successfully or not).

agent -- set the browser's "User-agent" string.

sleep [<seconds>] -- sleep the given number of seconds. Defaults to 1 second.

reset_browser -- reset the browser.

extend_with <module> -- import commands from Python module. This acts like from <module> import * does in Python, so e.g. a function fn in extmodule would be available as fn. See extras/examples/extend_example.py for an example.

get_input <prompt> -- get keyboard input and store it in __input__. When called from Python, this function returns the input value.

Alternative spelling: "getinput"

get_password <prompt> -- get silent keyboard input and store it in __password__. When called from Python, this function returns the input value.

Alternative spelling: "getpassword"

add_auth <realm> <uri> <user> <password> -- add HTTP Basic Authentication information for the given realm/URI combination. For example,

    add_auth IdyllStuff http://www.idyll.org/ titus test

tells twill that a request from the authentication realm "IdyllStuff" under http://www.idyll.org/ should be answered with username 'titus', password 'test'. If the 'with_default_realm' option is set to True, ignore 'realm'.

config [<key> [<value>]] -- show/set configuration options.

Configuration options starting with tidy_ will be used when checking documents with HTML Tidy. For example,

    config tidy_drop_empty_elements no

add_extra_headers <name> <value> -- add an extra HTTP header to each HTTP request.

show_extra_headers -- show the headers being added to each HTTP request.

clear_extra_headers -- clear the headers being added to each HTTP request.

  Special variables

__input__ -- result of last getinput.

__match__ -- matched text from last find, title, or url.

__password__ -- result of last getpassword.

__url__ -- current URL.

  Details on form handling

Both the form_value (or fv) and submit commands rely on a certain amount of implicit cleverness to do their work. In odd situations, it can be annoying to determine exactly what form field form_value is going to pick based on your field name, or what form and field submit is going to "click" on.

Here is the pseudocode for how form_value and submit figure out what form to use (function twill.browser.form):

    for each form on page:
        if supplied regex pattern matches the form name, select
    if no form name, try converting to an integer N & using N-1 as
    an index into the list or forms on the page (i.e. form 1 is the
    first form on the page).

Here is the pseudocode for how form_value and submit figure out what form field to use (function twill.browser.form_field):

    search current form for control name with exact match to field_name;
    if single (unique) match, select.
    if no match, convert field_name into a number and use as an index, if
    possible.
    if no match, search current form for control name with regex match to
    field_name; if single (unique) match, select.
    if *still* no match, look for exact matches to submit-button values.
    if single (unique) match, select.

Here is the pseudocode for submit:

    if a form was _not_ previously selected by form_value:
       if there's only one form on the page, select it.
       otherwise, fail.
    if a field is not explicitly named:
       if a submit button was "clicked" with form_value, use it.
       otherwise, use the first submit button on the form, if any.
    otherwise:
       find the field using the same rules as form_value
    finally, if a button has been picked, submit using it;
    otherwise, submit without using a button

TESTING WEB SITES WITH TWILL

twill was initially designed for testing web sites, although since then people have also figured out that it's good for browsing unsuspecting web sites.

  Using the twill command

The simplest way to test Web sites is to write one or more twill scripts and then simply run

    twill [ -u initial_url ] script(s)

either from the command-line (for development purposes), via a cron job (to check to see if sites are up and responding), or from your functional or unit tests (see below).

twill will try to run each script given to it on the command line once, and will report the number of scripts that failed. The exit value of the script will be 0 if there are no failures, so you can use it in a shell script easily enough.

twill will gather scripts from directories, so you can create a whole directory hierarchy containing your scripts and they will all be gathered and run, in standard lexical order.

twill scripts are assumed to have the '.twill' extensions; you don't need to specify it explicitly.

The -u flag can be used to give twill an initial URL; this is equivalent to placing a "go (initial_url)" command at the top of the script, but is particularly handy for test frameworks where the URL might change depending on the developer.

  Stress testing

You can use the twill-fork script to do some stress testing. The syntax is

    twill-fork -n <number to execute> -p <number of processes> script [ scripts... ]

For example,

    twill-fork -n 500 -p 10 test-script

will fork 10 times and run test-script 50 times in each process. twill-fork will record the time it takes to run all of the scripts specified on the command and print a summary at the end.

The time recorded is not the CPU time used. (This would lead to an inaccurate estimate because the client code uses blocking calls to retrieve Web pages.) Rather, the time recorded is the clock time measured between the start and end of script execution.

Try twill-fork -h to get a list of other command line arguments.

Note that twill-fork runs only under Unix and still needs a lot of work...

  Unit testing

twill can be used in unit testing, and it contains some Python support infrastructure for this purpose.

As an example, here's the code from twill's own unit test, testing the unit-test support code:

    import os
    import twill
    from quixote.server.simple_server import run as quixote_run
    PORT=8090  # port to run the server on
    def run_server():
        """Function to run the server"""
        quixote_run(twill.tests.server.create_publisher, port=PORT)
    def test():
        """The unit test"""
        test_dir = twill.tests.utils.testdir
        script = os.path.join(test_dir, 'test_unit_support.twill')
        # create test_info object
        test_info = twill.unit.TestInfo(script, run_server, PORT)
        # run tests!
        twill.unit.run_test(test_info)

Here, I'm unit testing the Quixote application twill.tests.server, which is run by quixote_run (a.k.a. quixote.server.simple_server.run) on port PORT, using the twill script test_unit_support.twill. That script contains this code:

    # starting URL is provided to it by the unit test support framework.
    go ./multisubmitform
    code 200

A few things to note:

•   the initial URL is set based on the URL reported by TestInfo, which calculates it based on the PORT argument. (This can be overridden by subclasses.)

•   TestInfo contains code to (a) run the server function in a new process, and (b) run the twill script against that server. It then kills the server after script completion.

•   You can also pass a 'sleep' argument to the TestInfo constructor that specifies how many seconds to wait for the server to start before executing the script.

  Testing WSGI applications "in-process"

You can use wsgi_intercept for testing WSGI applications.

It provides two functions, add_wsgi_intercept and remove_wsgi_intercept, that allow Python applications to redirect HTTP calls into a WSGI application "in-process", without going via an external Internet call. This is particularly useful for unit tests, where setting up an externally available Web server can be inconvenient.

For example, the following code redirects all localhost:80 calls to the given WSGI app:

    def create_app():
        return wsgi_app
    import wsgi_intercept
    wsgi_intercept.requests_intercept.install()
    wsgi_intercept.add_wsgi_intercept('localhost', 80, create_app)
    # your twill tests go here...
    wsgi_intercept.remove_wsgi_intercept('localhost', 80)
    wsgi_intercept.requests_intercept.uninstall()

See the tests/test_wsgi_intercept.py unit test for more examples.

TWILL EXTENSION MODULES

  Contents

•   twill Extension Modules

•   check_links -- a simple link checker

•   match_parse -- extension for regular expressions with multiple matches

•   require -- assert that specific conditions hold after each page is loaded

•   mailman_sf -- discard spam messages from your SourceForge mailing lists

•   argparse -- pass arguments into twill scripts via sys.argv

•   dns_check -- make assertions about domain name service records

•   csv_iterate -- iterate over lists of values

•   dirstack -- manipulate the current working directory (cwd)

•   formfill -- convenience functions for filling out forms

    Several different extension modules are distributed with twill, under 'twill.extensions'.

  check_links -- a simple link checker

A simple link checker is included with twill. To use it, do

    go <page>
    extend_with check_links
    check_links

'check_links' will visit each link on the current page and verify that the HTTP status code is 200 (success). A list of failing links will be printed out at the end of the function.

The 'check_links' function takes an optional argument, a regex to filter for links that should be checked. By default the filter matches everything; if you only wanted to check links on e.g. idyll.org, you could do

    go http://idyll.org/
    check_links .*\.idyll\.org

A few notes:

•   'follow' is used to visit each link, so the referrer URL should be set correctly.

•   HTTP basic authentication and cookies work as you'd expect, so you can set up twill with all of the permissions necessary to visit restricted links & only then run check_links.

  match_parse -- extension for regular expressions with multiple matches

The 'match_parse' extension module contains a number of functions for dealing with multiple matches, etc. Here's some example code:

    extend_with match_parse
    go http://www.idyll.org/
    split "org"
    echo __matchlist__
    findall "t."
    echo __matchlist__
    split "org"
    popmatch 0
    getmatch test 'm[0].split()'
    showvar test
    split "org"
    setmatch "m.split()[0]"
    popmatch 0
    echo __matchlist__
    echo __match__

  require -- assert that specific conditions hold after each page is loaded

The 'require' extension module contains four functions that let you assert specific conditions. For example,

    extend_with require
    require success

will assert that after each page load, the HTTP code is 200 ("success"). 'require' does nothing on a 'back' call (which doesn't actually reload a page).

Currently there are only two assertions available, 'success' and 'links_ok'. 'links_ok' automatically runs 'check_links' (see above) after each page is loaded. 'links_ok' will not check links twice unless you call 'flush_visited' (see below).

The other functions in this module are:

skip_require -- don't check the requirements for the next action.

no_require -- turn off requirements checking & reset conditions.

flush_visited -- flush the list of visited links.

  mailman_sf -- discard spam messages from your SourceForge mailing lists

The 'mailman_sf' extension module contains two functions, 'discard_all_messages' and 'exit_if_empty'. Here's some example code:

    # load in the mailman_sf extensions module
    extend_with mailman_sf
    # unfortunately we have to hard-code in the mailing list name for
    # the moment. not sure how to do substitutions here.
    go https://lists.sourceforge.net/lists/admindb/pywx-announce
    # fill out the page with the list password.
    getpassword "Enter list password: "
    formvalue 1 adminpw __password__
    submit 0
    code 200
    # if there aren't any messages on the page, exit.
    exit_if_empty
    # if not empty, discard all messages & submit
    discard_all_messages
    submit 0

  argparse -- pass arguments into twill scripts via sys.argv

The 'argparse' extension module contains one function, 'get_args'. 'get_args' loads all of the post-scriptfile command-line arguments into variables, e.g.

    $ twill script1 script2 script3 -- val1 val2 val3
    ...
    >> extend_with argparse
    >> get_args
    >> echo "args are ${arg1} ${args} ${arg3}"
    args are val1 val2 val3

  dns_check -- make assertions about domain name service records

dns_check implements functions to test domain name service (DNS) resolution. You'll need dnspython to use it.

Functions:

•   dns_resolves -- assert that a host resolves to a specific IP address.

•   dns_a -- assert that a host directly resolves to a specific IP address

•   dns_cname -- assert that a host is an alias for another hostname.

•   dnx_mx -- assert that a given host is a mail exchanger for the given name.

•   dns_ns -- assert that a given hostname is a name server for the given name.

  csv_iterate -- iterate over lists of values

csv_iterate provides a single function, 'csv_iterate', that lets you run a script once for each row in a file of comma-separated values. For example, if you had a file 'test.csv' containing

    value1, value2
    value3, value4

and a script 'test' containing

    echo "first value is ${col1}, second is ${col2}"

then

    >> csv_iterate test.csv test

would produce

    first value is value1, second is value2
    first value is value3, second is value4

This is designed for building test suites containing many different combinations of values for specific forms.

  dirstack -- manipulate the current working directory (cwd)

dirstack provides two functions, 'chdir' and 'popd', which change the directory and recover the original directory, respectively. The cwd is kept in the global variable '__dir__'.

For example

    >> chdir /tmp
    >> echo __dir__
    >> popd
    >> echo __dir__

will change to the '/tmp' directory, print out '/tmp', and then change back to the original directory.

  formfill -- convenience functions for filling out forms

formfill provides three functions, 'fv_match', 'fv_multi', and 'fv_multi_sub'.

'fv_match' allows you to set many form fields all at once, based on regex matching to the form field name, e.g.

    fv_match <form> test-(\d+) 'hello world!'

will set all fields with the name 'test-#' (# is any number) to 'hello world!'

'fv_multi' allows you to set many form fields at once, using a more compact notation:

    fv_multi <form> field1=value1 field2=value2 ...

'fv_multi' uses the same field-matching convention that 'formvalue' uses: it simply iterates over all arguments, splits at the first '=', and runs 'formvalue' directly.

'fv_multi_sub' does the same thing as 'fv_multi' and then executes 'submit'.

(Thanks to Ben Bangert for the idea!)

TWILL'S PYTHON API

twill is essentially a web browsing and testing tool based on the requests and lxml packages. All twill commands are implemented in the commands.py file, and pyparsing does the work of parsing the input and converting it into Python commands (see parse.py). Interactive shell work and readline support is implemented via the cmd module (from the standard Python library).

  Using twill from Python

There are two fairly simple ways to use twill from Python. (They are compatible with each other, so you don't need to choose between them; just use whichever is appropriate.)

The first is to simply import all of the commands in commands.py and use them directly from Python. For example,

    from twill.commands import *
    go("http://www.python.org/")
    showforms()

This has the advantage of being very simple, as well as being tied directly to the documented set of commands in the commands reference.

However, the functions in commands.py are too simple for some situations. In particular, most of them do not have any return values, so in order to e.g. get the HTML for a particular page, you will need to ask the actual "Web browser" object that twill uses.

To make use of to the Web browser directly, import the browser object:

    from twill import browser
    browser.go("https://www.python.org/")
    assert 'Python' in browser.html
    browser.showforms()

This is the second way to use twill from Python, and it is much more flexible. All of the functions in commands.py use this same browser object, so you can mix and match:

    from twill import browser
    from twill.commands import *
    go("https://www.python.org/")
    assert 'Python' in browser.html
    find("Documentation")
    browser.showforms()

The basic difference is that functions available through the browser object are intended for use from Python, while commands.py functions define the twill language. In fact, all of the functions in commands.py are small snippets of code wrapped around the browser object.

Most importantly, the browser object also provides several properties that can be used to introspect the current state of the browser and evaluated programmatically in Python, such as url, code, html, title, links, forms, cookies, response_headers or history.

For more information on the functions exposed by the browser object, see the code of the TwillBrowser class in twill.browser.

  Extending twill

Right now twill is very easy to extend: just build a Python module that exports the functions you want to call, place it in the PYTHONPATH, and run extend_with <modulename>. See extensions/mailman_sf.py for an extension that helps deal with mailman lists on SourceForge; this extension is used by examples/discard-sf-mailman-msgs.

Notes:

•   If your extension raises SystemExit, twill will stop processing the script. This is a useful way to build in conditionals, e.g. see the discard-sf-mailman-msgs example script.

  Passing variables from Python into twill

Suppose you write an extension function, and you want to return some values from that extension function for later use in filling out forms. How do you do this?

The simplest way is to simply insert them into the global or local dictionary, like so:

    from twill.namespaces import get_twill_glocals
    global_dict, local_dict = get_twill_glocals()
    global_dict['varname'] = value

You can then use the usual variable access methods to substitute for varname, e.g.

    formvalue 1 field $varname

  Using twill in other Python programs

All of the commands available in twill are implemented as top-level functions in the twill.commands module. For example, to use twill functionality from another Python program, you can do:

    from twill.commands import go, showforms, formclear, fv, submit
    go('http://localhost:8080/')
    go('./widgets')
    showforms()
    formclear('1')
    fv('1', 'name', 'test')
    fv('1', 'password', 'testpass')
    fv('1', 'confirm', 'yes')
    showforms()
    submit('0')

Note that all arguments need to be strings, at least for the moment.

You can capture command output by passing any write-enabled file handle to twill.set_output, e.g.

    twill.set_output(StringIO())

will send all non-error output into a StringIO() object.

DEVELOPING TWILL

twill is entirely written in Python. You will need Python 3.6 or newer to use and develop it.

  Package tests

twill comes with several unit tests. They depend on pytest, wsgi_intercept and Quixote. To run them, type 'pytest' in the top level directory. To run an individual test, you can use the command 'pytest tests/test_something.py'.

  Licensing

twill 0.8 and above are licensed under the MIT license. All code taken from twill 0.x is Copyright (C) 2005-2007 C. Titus Brown <titus@idyll.org>.

In plain English: C. Titus Brown owns the original code, but you're welcome to use it, update it, subsume it into other projects, and distribute it freely. However, you must retain copyright attribution.

Newer versions 1.x, 2.x and 3.x are also Copyright (C) 2007-2022 Ben R. Taylor , Adam V. Brandizzi, Christoph Zwerschke et al.

The newer versions are released under the same MIT license.

  Developer releases

"Developer releases" incorporating all recent significant changes are made available at https://github.com/twill-tools/twill.

PROPAGANDA, TOOLS AND PACKAGES

  Other opinions

Grig Gheorgiu has written a blog entry on Web app testing using twill. Michele Simionato wrote a nice long article on Testing Web Apps, and Nitesh Djanjani tried it out as well.

  Offshoots from the twill project

C. Titus Brown wrote a simple wsgi_intercept standalone package based on the WSGI interception stuff for in-process testing of WSGI apps by twill. This is for people who want to talk directly to their Web apps without going through a network connection.

  Other Python-based Web testing and browsing tools

For those who want to use Python-based toolkits to test their Web apps, there are many great options. Here are some of them:

PBP was very similar to twill -- twill was initially based on it -- but the project seems to be defunct now. Like PBP, twill was based on mechanize in the past, unfortunately development of that project also stalled. The current version is based on requests and lxml instead.

funkload is a nifty looking tool that does functional load testing. It is built on webunit.

webtest is an extension to unittest for testing WSGI applications, without starting up an HTTP server.

zope.testbrowser provides an easy-to-use programmable web browser with special focus on testing.

mechanoid is a fork of mechanize that claims many bug fixes and a different programming style. It's primarily used for scripting Web sites, not for testing, but it can easily be used for testing.

PAMIE and PyXPCOM provide a Python interface for interacting with IE and Mozilla-based browsers, respectively.

Finally, Selenium is an in-browser testing system that several people have given rave reviews. Note that it's not written in Python...

You might also consider checking out Ian Bicking's proto-implementation of twill in Javascript.

As twill was based upon mechanize, so is Perl's WWW::Mechanize::Shell based upon WWW::Mechanize. There's even an HTTP::Recorder. WebTst also looks interesting.

INDICES AND TABLES

•   Index

•   Module Index

•   Search Page

AUTHOR
C. Titus Brown, Ben R. Taylor, Christoph Zwerschke et al

COPYRIGHT
2022, C. Titus Brown, Ben R. Taylor, Christoph Zwerschke et al