# Server Operation

## Recommended Operational Practices

Operation of a SIP server is not always easy task. Server administrators are challenged by broken or misconfigured user agents, network and host failures, hostile attacks and other stress-makers. All such situations may lead to an operational failure. It is sometimes very difficult to figure out the root reason of a failure, particularly in a distributed environment with many SIP components involved. In this section, we share some of our practices and refer to tools which have proven to make life of administrators easier

Keeping track of messages is good

Frequently, operational errors are discovered or reported with a delay. Users frustrated by an error frequently approach administrators and scream "even though my SIP requests were absolutely ok yesterday, they were mistakenly denied by your server". If administrators do not record all SIP traffic at their site, they will be no more able to identify the problem reason. We thus recommend that site operators record all messages passing their site and keep them stored for some period of time. They may use utilities such as ngrep or tcpdump . There is also a utility scripts/harv_ser.sh in ser distribution for post-processing of captured messages. It summarizes messages captured by reply status and user-agent header field.

Real-time Traffic Watching

Looking at SIP messages in real-time may help to gain understanding of problems. Though there are commercial tools available, using a simple, text-oriented tool such as ngrep makes the job very well thanks to SIP's textual nature.

### Using ngrep

In this example, all messages at port 5060 which include the string "bkraegelin" are captured and displayed

    [jiri@fox s]$ ngrep bkraegelin@ port 5060
    interface: eth0 (195.37.77.96/255.255.255.240)
    filter: ip and ( port 5060 )
    match: bkraegelin@
    #
    U +0.000000 153.96.14.162:50240 -> 195.37.77.101:5060
     REGISTER sip:iptel.org SIP/2.0.
     Via: SIP/2.0/UDP 153.96.14.162:5060.
     From: sip:bkraegelin@iptel.org.
     To: sip:bkraegelin@iptel.org.
     Call-ID: 0009b7aa-1249b554-6407d246-72d2450a@153.96.14.162.
     Date: Thu, 26 Sep 2002 22:03:55 GMT.
     CSeq: 101 REGISTER.
     Expires: 10.
     Content-Length: 0.
     .

    #
    U +0.000406 195.37.77.101:5060 -> 153.96.14.162:5060
     SIP/2.0 401 Unauthorized.
     Via: SIP/2.0/UDP 153.96.14.162:5060.
     From: sip:bkraegelin@iptel.org.
     To: sip:bkraegelin@iptel.org.
     Call-ID: 0009b7aa-1249b554-6407d246-72d2450a@153.96.14.162.
     CSeq: 101 REGISTER.
     WWW-Authenticate: Digest realm="iptel.org", nonce="3d9385170000000043acbf6ba9c9741790e0c57adee73812", algorithm=MD5.
     Server: Sip EXpress router(0.8.8 (i386/linux)).
     Content-Length: 0.
     Warning: 392 127.0.0.1:5060 "Noisy feedback tells: pid=31604 req_src_ip=153.96.14.162 in_uri=sip:iptel.org out_uri=sip:iptel.org via_cnt==1".

Tracing Errors in Server Chains

A request may pass any number of proxy servers on its path to its destination. If an error occurs in the chain, it is difficult for upstream troubleshooters and/or users complaining to administrators to learn more about error circumstances. ser does its best and displays extensive diagnostics information in SIP replies. It allows troubleshooters and/or users who report to troubleshooters to gain additional knowledge about request processing status. This extended debugging information is part of the warning header field. See usingngrep for an illustration of a reply that includes such a warning header field. The header field contains the following pieces of information:

- Server's IP Address -- good to identify from which server in a chain the reply came.

- Incoming and outgoing URIs -- good to learn for which URI the reply was generated, as it may be rewritten many times in the path. Particularly useful for debugging of numbering plans.

- Number of Via header fields in replied request -- that helps in assessment of request path length. Upstream clients would not know otherwise, how far away in terms of SIP hops their requests were replied.

- Server's process id. That is useful for debugging to discover situations when multiple servers listen at the same address.

- IP address of previous SIP hop as seen by the SIP server.

If server administrator is not comfortable with disclosing all this information, he can turn them off using the `sip_warning` configuration option.

A nice utility for debugging server chains is sipsak, SIP Swiss Army Knife, traceroute-like tool for SIP developed at iptel.org. It allows you to send OPTIONS request with low, increasing Max-Forwards header-fields and follow how it propagates in SIP network. See its webpage at http://sipsak.berlios.de/ .

### Use of SIPSak for Learning SIP Path

    [jiri@bat sipsak]$ ./sipsak -T -s sip:7271@iptel.org
    warning: IP extract from warning activated to be more informational
    0: 127.0.0.1 (0.456 ms) SIP/2.0 483 Too Many Hops
    1: ?? (31.657 ms) SIP/2.0 200 OK
       without Contact header

Note that in this example, the second hop server does not issue any warning header fields in replies and it is thus impossible to display its IP address in SIPsak's output.

Watching Server Health

Watching Server's operation status in real-time may also be a great aid for trouble-shooting. ser has an excellent facility, a FIFO server, which allows UNIX tools to access server's internals. (It is similar to how Linux tool access Linux kernel via the proc file system.) The FIFO server accepts commands via a FIFO (named pipe) and returns data asked for. Administrators do not need to learn details of the FIFO communication and can serve themselves using a front-end utility serctl. Of particular interest for monitoring server's operation are serctl commands `ps` and `moni`. The former displays running ser processes, whereas the latter shows statistics.

### serctl ps command

This example shows 10 processes running at a host. The process 0, "attendant" watches child processes and terminates all of them if a failure occurs in any of them. Processes 1-4 listen at local interface and processes 5-8 listen at Ethernet interface at port number 5060. Process number 9 runs FIFO server, and process number 10 processes all server timeouts.

    [jiri@fox jiri]$ serctl ps
    0	31590	attendant
    1	31592	receiver child=0 sock=0 @ 127.0.0.1::5060
    2	31595	receiver child=1 sock=0 @ 127.0.0.1::5060
    3	31596	receiver child=2 sock=0 @ 127.0.0.1::5060
    4	31597	receiver child=3 sock=0 @ 127.0.0.1::5060
    5	31604	receiver child=0 sock=1 @ 195.37.77.101::5060
    6	31605	receiver child=1 sock=1 @ 195.37.77.101::5060
    7	31606	receiver child=2 sock=1 @ 195.37.77.101::5060
    8	31610	receiver child=3 sock=1 @ 195.37.77.101::5060
    9	31611	fifo server
    10	31627	timer

Is Server Alive

It is essential for solid operation to know continuously that server is alive. We've been using two tools for this purpose. sipsak does a great job of "pinging" a server, which may be used for alerting on unresponsive servers.

monit is a server watching utility which alerts when a server dies.

Dealing with DNS

SIP standard leverages DNS. Administrators of ser should be aware of impact of DNS on server's operation. Server's attempt to resolve an unresolvable address may block a server process in terms of seconds. To be safer that the server doesn't stop responding due to being blocked by DNS resolving, we recommend the following practices:

- Start a sufficient number of children processes. If one is blocked, the other children will keep serving.

- Use DNS caching. For example, in Linux, there is an nscd daemon available for this purpose.

- Process transactions statefully if memory allows. That helps to absorb retransmissions without having to resolve DNS for each of them.

Logging

ser by default logs to syslog facility. It is very useful to watch log messages for abnormal behavior. Log messages, subject to syslog configuration may be stored at different files, or even at remote systems. A typical location of the log file is `/var/log/messages`.

One can also use other syslogd implementation. metalog ( http://metalog.sourceforge.net/ ) features regular expression matching that enables to filter and group log messages.

For the purpose of debugging configuration scripts, one may want to redirect log messages to console not to pollute syslog files. To do so configure ser in the following way:

- Attach ser to console by setting `fork=no`.

- Set explicitly at which address ser should be listening, e.g., `listen=192.168.2.16`.

- Redirect log messages to standard error by setting `log_stderror=yes`

- Set appropriately high log level. (Be sure that you redirected logging to standard output. Flooding system logs with many detailed messages would make the logs difficult to read and use.) You can set the global logging threshold value with the option `debug=nr`, where the higher `nr` the more detailed output. If you wish to set log level only for some script events, include the desired log level as the first parameter of the `log` action in your script. The messages will be then printed if `log`'s level is lower than the global threshold, i.e., the lower the more noisy output you get.

  ### Logging Script

  The following SIP message causes then logging output as shown below.

      REGISTER sip:192.168.2.16 SIP/2.0
      Via: SIP/2.0/UDP 192.168.2.33:5060
      From: sip:113311@192.168.2.16
      To: sip:113311@192.168.2.16
      Call-ID: 00036bb9-0fd305e2-7daec266-212e5ec9@192.168.2.33
      Date: Thu, 27 Feb 2003 15:10:52 GMT
      CSeq: 101 REGISTER
      User-Agent: CSCO/4
      Contact: sip:113311@192.168.2.33:5060
      Content-Length: 0
      Expires: 600

      [jiri@cat sip_router]$ ./ser -f examples/logging.cfg
      Listening on
      	192.168.2.16 [192.168.2.16]::5060
      	Aliases: cat.iptel.org:5060 cat:5060
      WARNING: no fork mode
      0(0) INFO: udp_init: SO_RCVBUF is initially 65535
      0(0) INFO: udp_init: SO_RCVBUF is finally 131070
      0(17379) REGISTER received
      0(17379) request for other domain received

Labeling Outbound Requests

Without knowing, which pieces of script code a relayed request visited, trouble-shooting would be difficult. Scripts typically apply different processing to different routes such as to IP phones and PSTN gateways. We thus recommend to label outgoing requests with a label describing the type of processing applied to the request.

Attaching "routing-history" hints to relayed requests is as easy as using the `append_hf` action exported by textops module. The following example shows how different labels are attached to requests to which different routing logic was applied.

### "Routing-history" labels

    # is the request for our domain?
    # if so, process it using UsrLoc and label it so.
    if (uri=~[@:\.]domain.foo") {
        if (!lookup("location")) {
            sl_send_reply("404", "Not Found");
            break;
        };
        # user found -- forward to him and label the request
        append_hf("P-hint: USRLOC\r\n");
    } else {
        # it is an outbound request to some other domain --
        # indicate it in the routing-history label
        append_hf("P-hint: OUTBOUND\r\n");
    };
    t_relay();

This is how such a labeled requests looks like. The last header field includes a label indicating the script processed the request as outbound.

    #
    U 2002/09/26 02:03:09.807288 195.37.77.101:5060 -> 203.122.14.122:5060
     SUBSCRIBE sip:rajesh@203.122.14.122 SIP/2.0.
     Max-Forwards: 10.
     Via: SIP/2.0/UDP 195.37.77.101;branch=53.b44e9693.0.
     Via: SIP/2.0/UDP 203.122.14.115:16819.
     From: sip:rajeshacl@iptel.org;tag=5c7cecb3-cfa2-491d-a0eb-72195d4054c4.
     To: sip:rajesh@203.122.14.122.
     Call-ID: bd6c45b7-2777-4e7a-b1ae-11c9ac2c6a58@203.122.14.115.
     CSeq: 2 SUBSCRIBE.
     Contact: sip:203.122.14.115:16819.
     User-Agent: Windows RTC/1.0.
     Proxy-Authorization: Digest username="rajeshacl", realm="iptel.org", algorithm="MD5", uri="sip:rajesh@203.122.14.122", nonce="3d924fe900000000fd6227db9e565b73c465225d94b2a938", response="a855233f61d409a791f077cbe184d3e3".
     Expires: 1800.
     Content-Length: 0.
     P-hint: OUTBOUND.

## HOWTOs

This section is a "cookbook" for dealing with common tasks, such as user management or controlling access to PSTN gateways.

### User Management

There are two tasks related to management of SIP users: maintaining user accounts and maintaining user contacts. Both these jobs can be done using the serctl command-line tool. Also, the complimentary web interface, serweb, can be used for this purpose as well.

If user authentication is turned on, which is a highly advisable practice, user account must be created before a user can log in. To create a new user account, call the `serctl add` utility with username, password and email as parameters. It is important that the environment `SIP_DOMAIN` is set to your realm and matches realm values used in your script. The realm value is used for calculation of credentials stored in subscriber database, which are bound permanently to this value.

    [jiri@cat gen_ha1]$ export SIP_DOMAIN=foo.bar
    [jiri@cat gen_ha1]$ serctl add newuser secret newuser@foo.bar
    MySql Password:
    new user added

serctl can also change user's password or remove existing accounts from system permanently.

    [jiri@cat gen_ha1]$ serctl passwd newuser newpassword
    MySql Password:
    password change succeeded
    [jiri@cat gen_ha1]$ serctl rm newuser
    MySql Password:
    user removed

User contacts are typically automatically uploaded by SIP phones to server during registration process and administrators do not need to worry about them. However, users may wish to append permanent contacts to PSTN gateways or to locations in other administrative domains. To manipulate the contacts in such cases, use serctl ul tool. Note that this is the only correct way to update contacts -- direct changes to back-end MySql database do not affect server's memory. Also note, that if persistence is turned off (usrloc "db_mode" parameter set to "0"), all contacts are gone on server reboot. Make sure that persistence is enabled if you add permanent contacts.

To add a new permanent contact for a user, call serctl ul add <username> <contact>. To delete all user's contacts, call serctl ul rm <username>. serctl ul show <username> prints all current user's contacts.

    [jiri@cat gen_ha1]$ serctl ul add newuser sip:666@gateway.foo.bar
    sip:666@gateway.foo.bar
    200 Added to table
    ('newuser','sip:666@gateway.foo.bar') to 'location'
    [jiri@cat gen_ha1]$ serctl ul show newuser
    <sip:666@gateway.foo.bar>;q=1.00;expires=1073741812
    [jiri@cat gen_ha1]$ serctl ul rm newuser
    200 user (location, newuser) deleted
    [jiri@cat gen_ha1]$ serctl ul show newuser
    404 Username newuser in table location not found

### User Aliases

Frequently, it is desirable for a user to have multiple addresses in a domain. For example, a user with username "john.doe" wants to be reachable at a shorter address "john" or at a numerical address "12335", so that PSTN callers with digits-only key-pad can reach him too.

With ser, you can maintain a special user-location table and translate existing aliases to canonical usernames using the `lookup` action from usrloc module. The following script fragment demonstrates use of `lookup` for this purpose.

#### Configuration of Use of Aliases

    if (!uri==myself) { # request not for our domain...
        route(1); # go somewhere else, where outbound requests are processed
        break;
    };
    # the request is for our domain -- process registrations first
    if (method=="REGISTER") { route(3); break; };

    # look now, if there is an alias in the "aliases" table; don't care
    # about return value: whether there is some or not, move ahead then
    lookup("aliases");

    # there may be aliases which translate to other domain and for which
    # local processing is not appropriate; check again, if after the
    # alias translation, the request is still for us
    if (!uri==myself) { route(1); break; };

    # continue with processing for our domain...
    ...

The table with aliases is updated using the serctl tool. serctl alias add <alias> <uri> adds a new alias, serctl alias show <user> prints an existing alias, and serctl alias rm <user> removes it.

    [jiri@cat sip_router]$ serctl alias add 1234 sip:john.doe@foo.bar
    sip:john.doe@foo.bar
    200 Added to table
    ('1234','sip:john.doe@foo.bar') to 'aliases'
    [jiri@cat sip_router]$ serctl alias add john sip:john.doe@foo.bar
    sip:john.doe@foo.bar
    200 Added to table
    ('john','sip:john.doe@foo.bar') to 'aliases'
    [jiri@cat sip_router]$ serctl alias show john
    <sip:john.doe@foo.bar>;q=1.00;expires=1073741811
    [jiri@cat sip_router]$ serctl alias rm john
    200 user (aliases, john) deleted

Note that persistence needs to be turned on in usrloc module. All changes to aliases will be otherwise lost on server reboot. To enable persistence, set the db_mode usrloc parameter to a non-zero value.

    # ....load module ...
    loadmodule "modules/usrloc/usrloc.so"
    # ... turn on persistence -- all changes to user tables are immediately
    # flushed to mysql
    modparam("usrloc", "db_mode",   1)
    # the SQL address:
    modparam("usrloc", "db_url","mysql://ser:secret@dbhost/ser")

### Access Control (PSTN Gateway)

It is sometimes important to exercise some sort of access control. A typical use case is when ser is used to guard a PSTN gateway. If a gateway was not well guarded, unauthorized users would be able to use it to terminate calls in PSTN, and cause high charges to its operator.

There are few issues you need to understand when configuring ser for this purpose. First, if a gateway is built or configured to accept calls from anywhere, callers may easily bypass your access control server and communicate with the gateway directly. You then need to enforce at transport layer that signaling is only accepted if coming via ser and deny SIP packets coming from other hosts and port numbers. Your network must be configured not to allow forged IP addresses. Also, you need to turn on record-routing to assure that all session requests will travel via ser. Otherwise, caller's devices would send subsequent SIP requests directly to your gateway, which would fail because of transport filtering.

Authorization (i.e., the process of determining who may call where) is facilitated in ser using group membership concept. Scripts make decisions on whether a caller is authorized to make a call to a specific destination based on user's membership in a group. For example a policy may be set up to allow calls to international destinations only to users, who are members of an "int" group. Before user's group membership is checked, his identity must be verified first. Without cryptographic verification of user's identity, it would be impossible to assert that a caller really is who he claims to be.

The following script demonstrates, how to configure ser as an access control server for a PSTN gateway. The script verifies user identity using digest authentication, checks user's privileges, and forces all requests to visit the server.

#### Script for Gateway Access Control

Use the serctl tool to maintain group membership. serctl acl grant <username> <group> makes a user member of a group, serctl acl show <username> shows groups of which a user is member, and serctl acl revoke <username> [<group>] revokes user's membership in one or all groups.

    [jiri@cat sip_router]$ serctl acl grant john int
    MySql Password:
    +------+-----+---------------------+
    | user | grp | last_modified       |
    +------+-----+---------------------+
    | john | int | 2002-12-08 02:09:20 |
    +------+-----+---------------------+

### Accounting

In some scenarios, like termination of calls in PSTN, SIP administrators may wish to keep track of placed calls. ser can be configured to report on completed transactions. Reports are sent by default to syslog facility. Support for RADIUS and mysql accounting exists as well.

Note that ser is no way call-stateful. It reports on completed transactions, i.e., after a successful call set up is reported, it drops any call-related state. When a call is terminated, transactional state for BYE request is created and forgotten again after the transaction completes. This is a feature and not a bug -- keeping only transactional state allows for significantly higher scalability. It is then up to the accounting application to correlate call initiation and termination events.

To enable call accounting, tm and acc modules need to be loaded, requests need to be processed statefully and labeled for accounting. That means, if you want a transaction to be reported, the initial request must have taken the path "`setflag(X)`, `t_relay`" in ser script. X must have the value configured in `acc_flag` configuration option.

Also note, that by default only transactions that initiate a SIP dialog (typically INVITE) visit a proxy server. Subsequent transactions are exchanged directly between end-devices, do not visit proxy server and cannot be reported. To be able to report on subsequent transactions, you need to force them visit proxy server by turning record-routing on.

#### Configuration with Enabled Accounting

### Reliability

It is essential to guarantee continuous service operation even under erroneous conditions, such as host or network failure. The major issue in such situations is transfer of operation to a backup infrastructure and making clients use it.

The SIP standard's use of DNS SRV records has been explicitly constructed to handle with server failures. There may be multiple servers responsible for a domain and referred to by DNS. If it is impossible to communicate with a primary server, a client can proceed to another one. Backup servers may be located in a different geographic area to minimize risk caused by areal operational disasters: lack of power, flooding, earthquake, etc.

Unless there are redundant DNS servers, fail-over capability cannot be guaranteed.

Unfortunately, at the moment of writing this documentation (end of December 2002) only very few SIP products actually implement the DNS fail-over mechanism. Unless networks with SIP devices supporting this mechanism are built, alternative mechanisms must be used to force clients to use backup servers. Such a mechanism is disconnecting primary server and replacing it with a backup server locally. It unfortunately precludes geographic dispersion and requires network multihoming to avoid dependency on single IP access. Another method is to update DNS when failure of the primary server is detected. The primary drawback of this method is its latency: it may take long time until all clients learn to use the new server.

The easier part of the redundancy story is replication of ser data. ser relies on replication capabilities of its back-end database. This works with one exception: user location database. User location database is a frequently accessed table, which is thus cached in server's memory to improve performance. Back-end replication does not affect in-memory tables, unless server reboots. To facilitate replication of user location database, server's SIP replication feature must be enabled in parallel with back-end replication.

The design idea of replication of user location database is easy: Replicate any successful REGISTER requests to a peer server. To assure that digest credentials can be properly verified, both servers need to use the same digest generation secret and maintain synchronized time. A known limitation of this method is it does not replicate user contacts entered in another way, for example using web interface through FIFO server. The following script example shows configuration of a server that replicates all REGISTERs.

#### Script for Replication of User Contacts

### Stateful versus Stateless Forwarding

ser allows both stateless and stateful request processing. This memo explains what are pros and cons of using each method. The rule of thumb is "stateless for scalability, stateful for services". If you are unsure which you need, stateful is a safer choice which supports more usage scenarios.

Stateless forwarding with the `forward(uri:host, uri:port)` action guarantees high scalability. It withstands high load and does not run out of memory. A perfect use of stateless forwarding is load distribution.

Stateful forwarding using the `t_relay()` action is known to scale worse. It can quickly run out of memory and consumes more CPU time. Nevertheless, there are scenarios which are not implementable without stateful processing. In particular:

- Accounting requires stateful processing to be able to collect transaction status and issue a single report when a transaction completes.

- Forking only works with stateful forwarding. Stateless forwarding only forwards to the default URI out of the whole destination set.

- DNS resolution. DNS resolution may be better served with stateful processing. If a request is forwarded to a destination whose address takes long time to resolve, a server process is blocked and unresponsive. Subsequent request retransmissions from client will cause other processes to block too if requests are processed statelessly. As a result, ser will quickly run out of available processes. With stateful forwarding, retransmissions are absorbed and do not cause blocking of another process.

- Forwarding Services. All sort of services with the "forward_on_event" logic, which rely on `t_on_failure` tm action must be processed statefully.

- Fail-over. If you wish to try out another destination, after a primary destination failed you need to use stateful processing. With stateless processing you never know with what status a forwarded request completed downstream because you immediately release all processing information after the request is sent out.

  Positive return value of stateless `forward` action only indicates that a request was successfully sent out, and does not gain any knowledge about whether it was successfully received or replied. Neither does the return value of the stateful `t_relay` action family gain you this knowledge. However, these actions store transactional context with which includes original request and allows you to take an action when a negative reply comes back or a timer strikes. See replyprocessingsection for an example script which launches another branch if the first try fails.

### Serving Multiple Domains

ser can be configured to serve multiple domains. To do so, you need to take the following steps:

1. Create separate subscriber and location database table for each domain served and name them uniquely.

2. Configure your script to distinguish between multiple served domains. Use regular expressions for domain matching as described in redomainmatching.

3. Update table names in usrloc and auth actions to reflect names you created in createtable.

The latest SER release includes automated multidomain management which greatly automates maintenance of multiple domains. Ask our technical support for more help.

### Reporting Missed Calls

ser can report missed calls via syslog facility or to mysql. Mysql reporting can be utilized by ser's complementary web-interface, serweb. (See more in serweb).

Reporting on missed calls is enabled by acc module. There are two cases, on which you want to report. The first case is when a callee is off-line. The other case is when a user is on-line, but call establishment fails. There may be many failure reasons (call cancellation, inactive phone, busy phone, server timer, etc.), all of them leading to a negative (>=300) reply sent to caller. The acc module can be configured to issue a missed-call report whenever a transaction completes with a negative status. Two following script fragment deals with both cases.

First, it reports on calls missed due to off-line callee status using the `acc_request` action. The action is wrapped in transactional processing (`t_newtran`) to guarantee that reports are not duplicated on receipt of retransmissions.

Secondly, transactions to on-line users are marked to be reported on failure. That is what the `setflag(3)` action is responsible for, along with the configuration option "log_missed_flag". This option configures ser to report on all transactions, which were marked with flag 3.

    loadmodule("modules/tm/tm.so");
    loadmodule("modules/acc/acc.so");
    ....
    # if a call is labeled using setflag(3) and is missed, it will
    # be reported
    ...
    modparam("acc", "log_missed_flag", 3 );
    if (!lookup("location")) {
        # call invitations to off-line users are reported using the
        # acc_request action; to avoid duplicate reports on request
        # retransmissions, request is processed statefully (t_newtran,
        # t_reply)
        if ((method=="INVITE" || method=="ACK") && t_newtran() ) {
             t_reply("404", "Not Found");
             acc_request("404 Not Found");
             break;
        };
        # all other requests to off-line users are simply replied
        # statelessly and no reports are issued
        sl_send_reply("404", "Not Found");
        break;
    } else {
        # user on-line; report on failed transactions; mark the
        # transaction for reporting using the same number as
        # configured above; if the call is really missed, a report
        # will be issued
        setflag(3);
        # forward to user's current destination
        t_relay();
        break;
    };

### NAT Traversal

NATs are worst things that ever happened to SIP. These devices are very popular because they help to conserve IP address space and save money charged for IP addresses. Unfortunately, they translate addresses in a way which is not compatible with SIP. SIP advertises receiver addresses in its payload. The advertised addresses are invalid out of NATed networks. As a result, SIP communication does not work across NATs without extra effort.

There are few methods that may be deployed to traverse NATs. How proper their use is depends on the deployment scenario. Unfortunately, all the methods have some limitations and there is no straight-forward solution addressing all scenarios. Note that none of these methods takes explicit support in ser.

The first issue is whether SIP users are in control of their NATs. If not (NATs are either operated by ISP or they are sealed to prevent users setting them up), the only method is use of a STUN-enabled phone. STUN is a very simple protocol used to fool NAT in such a way, they permit SIP sessions. Currently, we are aware of one softphone (kphone) and one hardphone (snom) with STUN support, other vendors are working on STUN support too. Unfortunately, STUN gives no NAT traversal guarantee -- there are types of NATs, so called symmetric NATs, over which STUN fails to work.

There is actually yet another method to address SIP-unaware, user-uncontrolled NATs. It is based on a proxy server, which relays all signaling and media and mangles packets to make them more NAT-friendly. The very serious problem with this method is it does not scale.

If users are in control of their own NAT, as typically residential users are, they can still use STUN. However, they may use other alternatives too. One of them is to replace their NAT with a SIP-aware NAT. Such NATs have built-in SIP awareness, that patches problems caused by address translations. Prices of such devices are getting low and there are available implementations (Intertex, Cisco/PIX). No special support in phones is needed.

Other emerging option is UPnP. UPnP is a protocol that allows phones to negotiate with NAT boxes. You need UPnP support in both, NAT and phones. As UPnP NATs are quite affordable, costs are not an obstacle. Currently, we are aware of one SIP phone (SNOM) with UPnP support.

Geeks not wishing to upgrade their firewall to a SIP-aware or UPnP-enabled one may try to configure static address translation. That takes phones with configuration ability to use fixed port numbers and advertise outside address in signaling. Cisco phones have this capability, for example. The NAT devices need to be configured to translate outside port ranges to the ranges configured in phones.

### Using Only Latest User's Contact for Forwarding

In some scenarios, it may be beneficial only to use only one registered contact per user. If that is the case, setting registrar module's parameter `append_branches` to 1 will eliminate forking and forward all requests only to a single contact. If there are multiple contacts, a contact with highest priority is chosen. This can be changed to the "freshest" contact by setting module parameter's `desc_time_order` to 1.

### Authentication Policy: Prevention of Unauthorized Domain Name Use in From and More

Malicious users can claim a name of domain, to which they do not administratively belong, in From header field. This behavior cannot be generally prevented. The reason is that requests with such a faked header field do not need to visit servers of the domain in question. However, if they do so, it is desirable to assure that users claiming membership in a domain are actually associated with it. Otherwise the faked requests would be relayed and appear as coming from the domain, which would increase credibility of the faked address and decrease credibility of the proxy server.

Preventing unauthorized domain name use in relayed requests is not difficult. One needs to authenticate each request with name of the served domain in From header field. To do so, one can search for such a header field using `search` action (textops module) and force authentication if the search succeeds.

A straight-forward solution might be to authenticate ALL requests. However, that only works in closed networks in which all users have an account in the server domain. In open networks, it is desirable to permit incoming calls from callers from other domains without any authentication. For example, a company may wish to accept calls from unknown callers who are new prospective customers.

    # does the user claim our domain "foo.bar" in From?
    if (search("^(f|From):.*foo.bar")) {
        # if so, verify credential
        if (!proxy_authorize("foo.bar", "subscriber")) {
             # don't proceed if credentials broken; challenge
             proxy_challenge("foo.bar", "0");
             break;
        };
    };

In general, the authentication policy may be very rich. You may not forget each request deserves its own security and you need to decide whether it shall be authenticated or not. As mentioned above, in closed networks, you may want to authenticate absolutely every request. That however prohibits traffic from users from other domains. A pseudo-example of a reasonable policy is attached: it looks whether a request is registration, it claims to originate from our domain in From header field, or is a local request to another domain.

    # (example provided by Michael Graff on [serusers] mailing list
    if (to me):
        if register
         www_authorize or fail if not a valid register
         done
       if claiming to be "From" one of the domains I accept registrations for
       proxy_authorize
       done
       if not to me (I'm relaying for a local phone to an external address)
        proxy_authorize
       done

You also may want to apply additional restriction to how digest username relates to usernames claimed in From and To header fields. For example, the `check_to` action enforces the digest id to be equal to username in To header fields. That is good in preventing someone with valid credentials to register as someone else (e.g., sending a REGISTER with valid credentials of "joe" and To belonging to "alice"). Similarly, `check_from` is used to enforce username in from to equal to digest id.

There may be a need for a more complex relationship between From/To username and digest id. For example, providers with an established user/password database may wish to keep using it, whereas permitting users to claim some telephone numbers in From. To address such needs generally, there needs to be a 1:N mapping between digest id and all usernames that are acceptable for it. This is being addressed in a newly contributed module "domain", which also addresses more generally issues of domain matching for multidomain scenarios.

Other operational aspect affecting the authentication policy is guarding PSTN gateways (see acl). There may be destinations that are given away for free whereas other destinations may require access control using group membership, to which authentication is a prerequisite.

### Connecting to PBX Voicemail Using a Cisco Gateway

In some networks, administrators may wish to utilize their PBX voicemail systems behind PSTN gateways. There is a practical problem in many network settings: it is not clear for whom a call to voicemail is. If voicemail is identified by a single number, which is then put in INVITE's URI, there is no easy way to learn for whom a message should be recorded. PBX voicemails utilize that PSTN protocols signal the number of originally called party. If you wish to make the PBX voicemail work, you need to convey the number in SIP and translate it in PSTN gateways to its PSTN counterpart.

There may be many different ways to achieve this scenario. Here we describe the proprietary mechanism Cisco gateways use and how to configure ser to make the gateways happy. Cisco gateways expect the number of originally called party to be located in proprietary `CC-Diversion` header field. When a SIP INVITE sent via a PSTN gateway to PBX voicemail has number of originally called party in the header field, the voicemail system knows for whom the incoming message is. That is at least true for AS5300/2600 with Cisco IOS 12.2.(2)XB connected to Nortel pbxs via PRI. (On the other hand, 12.2.(7b) is known not to work in this scenario.)

ser needs then to be configured to append the `CC-Diversion` header field name for INVITEs sent to PBX voicemail. The following script shows that: when initial forwarding fails (nobody replies, busy is received, etc.), a new branch is initiated to the pbx's phone number. `append_urihf` is used to append the `CC-Diversion` header field. It takes two parameters: prefix, which includes header name, and suffix which takes header field separator. `append_urihf` inserts original URI between those two.

#### Forwarding to PBX/Voicemail via Cisco Gateways

## Troubleshooting

This section gathers practices how to deal with errors known to occur frequently. To understand how to watch SIP messages, server logs, and in general how to troubleshoot, read also operationalpractices.

SIP requests are replied by ser with "483 Too Many Hops" or "513 Message Too Large"

In both cases, the reason is probably an error in request routing script which caused an infinite loop. You can easily verify whether this happens by watching SIP traffic on loopback interface. A typical reason for misrouting is a failure to match local domain correctly. If a server fails to recognize a request for itself, it will try to forward it to current URI in believe it would forward them to a foreign domain. Alas, it forwards the request to itself again. This continues to happen until value of max_forwards header field reaches zero or the request grows too big. Solutions is easy: make sure that domain matching is correctly configured. See domainmatching for more information how to get it right.

Windows Messenger authentication fails.

The most likely reason for this problem is a bug in Windows Messenger. WM only authenticates if server name in request URI equals authentication realm. After a challenge is sent by SIP server, WM does not resubmit the challenged request at all and pops up authentication window again. If you want to authenticate WM, you need to set up your realm value to equal server name. If your server has no name, IP address can be used as realm too. The realm value is configured in scripts as the first parameter of all `{www|proxy}_{authorize|challenge}` actions.

On a multihomed host, forwarded messages carry other interface in Via than used for sending, or messages are not sent and an error log is issued "invalid sendtoparameters one possible reason is the server is bound to localhost".

Set the configuration option `mhomed` to "1". ser will then attempt to calculate the correct interface. It's not done by default as it degrades performance on single-homed hosts or multi-homed hosts that are not set-up as routers.

I receive "ERROR: t_newtran: transaction already in process" in my logs.

That looks like an erroneous use of tm module in script. tm can handle only one transaction per request. If you attempt to instantiate a transaction multiple times, ser will complain. Anytime any of `t_newtran`, `t_relay` or `t_relay_to_udp` actions is encountered, tm attempts to instantiate a transaction. Doing so twice fails. Make sure that any of this commands is called only once during script execution.

I try to add an alias but `serctl` complains that table does not exist.

You need to run ser and use the command `lookup("aliases")` in its routing script. That's because the table of aliases is stored in cache memory for high speed. The cache memory is only set up when the ser is running and configured to use it. If that is not the case, serctl is not able to manipulate the aliases table.

I started ser with `children=4` but many more processes were started. What is wrong?

That's ok. The `children` parameter defines how many children should process each transport protocol in parallel. Typically, the server listens to multiple protocols and starts other supporting processes like timer or FIFO server too. Call serctl ps to watch running processes.

I decided to use a compiled version of ser but it does not start any more.

You probably kept the same configuration file, which tries to load modules from the binary distribution you used previously. Make sure that modules paths are valid and point to where you compiled ser. Also, watch logs for error messages "ERROR: load_module: could not open module".