# &adminguide;

## Overview

This module provides an interactive config file debugger. It can print a trace of config script execution for a SIP message to log and set breakpoints on every script action, allowing step-by-step execution of the routing and response scripts. Moreover, this module allows setting static and dynamic module specific debug settings.

Debugging can be done from local or remote host via RPC interface (e.g., XMLRPC, &sercmd;, siremis).

The framework to set breakpoints on specific actions and config lines is not exported to RPC. Each action can be accompanied by a breakpoint or you can use dbg_breakpoint() function to set a breakpoint at certain line. Global breakpoints can be enabled/disabled at runtime. The script running trace can also be enabled/disabled at runtime.

When the SIP router process is stopped at a breakpoint, you can investigate the values of any pseudo-variables. Note that some of pseudo-variables may produce memory leaks; a fix is planned in the future (here fall pseudo-variables with dynamic name such as htable, sqlops). References to SIP message, avps, headers, script and shared variables are safe.

## Dependencies

### &kamailio; Modules

The following modules must be loaded before this module:

- none.

NOTE: Due to the debugger module child_init() function, one should load the module first in the module sequence in order to initialize _dbg_pid_list. Otherwise, another module (i.e. p_usrloc) forking a process with rank != PROC_INIT will fail.

### External Libraries or Applications

The following libraries or applications must be installed before running &kamailio; with this module loaded:

- None.

## Parameters

### `cfgtrace` (int)

Control whether the config script trace is enabled or disabled at startup. You can change the value at runtime without restart, globally or per process.

Default value is 0 (disabled).

#### Set `cfgtrace` parameter

    ...
    modparam("debugger", "cfgtrace", 1)
    ...

### `cfgtrace_format` (int)

Control what is printer in the cfgtrace log message. If it is set to 1, then the configuration file path is not printed, making the log messages shorter when not including other configuration files in the main one.

Default value is 0.

#### Set `cfgtrace_format` parameter

    ...
    modparam("debugger", "cfgtrace_format", 1)
    ...

### `breakpoint` (int)

Control whether every line (global) breakpoint is enabled or disabled at startup.

Default value is 0 (disabled).

#### Set `breakpoint` parameter

    ...
    modparam("debugger", "breakpoint", 1)
    ...

### `log_level` (int)

What log level is to be used to print module-specific messages.

Default value is -1 (L_ERR).

#### Set `log_level` parameter

    ...
    modparam("debugger", "log_level", 1)
    ...

### `log_level_name` (str)

What log level name is to be used to print cfg trace messages.

Default value is NULL (use default log names).

#### Set `log_level_name` parameter

    ...
    modparam("debugger", "log_level_name", "exec")
    ...

### `log_facility` (str)

Which log facility is to be used to print module-specific messages. By using this setting, you can configure syslog to send debug messages to a separate log channel, like a specific kamailio-debug log file.

Default value is NULL (default from core).

#### Set `log_facility` parameter

    ...
    modparam("debugger", "log_facility", "LOG_DAEMON")
    ...

### `log_prefix` (str)

String to print before any module-specific messages.

Default value is *** cfgtrace:.

#### Set `log_prefix` parameter

    ...
    modparam("debugger", "log_prefix", "from-debugger-with-love:")
    ...

### `step_usleep` (int)

Microseconds to sleep before checking for new commands when waiting at a breakpoint.

Default value is 100000 (that is 0.1 sec).

#### Set `step_usleep` parameter

    ...
    modparam("debugger", "step_usleep", 500000)
    ...

### `step_loops` (int)

How many sleeps of 'step_usleep' the RPC process performs when waiting for a reply from a worker process before responding to RPC. This avoids blocking RPC process forever in case the worker process 'forgets' to write back a reply.

Default value is 200.

#### Set `step_loops` parameter

    ...
    modparam("debugger", "step_loops", 100)
    ...

### `mod_hash_size` (int)

Used to compute power of two as size of internal hash table to store levels per module (e.g., if it is set to 4, internal hash table has 16 slots). One must set its value grater than 0 such that memory to be allocated to save the module specific debug levels or facility configured by `mod_level` or `mod_facility`. This parameter is accessible readonly via the Kamailio config framework.

Default value is 0 - feature disabled.

#### Set `mod_hash_size` parameter

    ...
    modparam("debugger", "mod_hash_size", 5)
    ...

### `mod_level_mode` (int)

Enable or disable per module log level (0 - disabled, 1 - enabled). This parameter is tunable via the Kamailio config framework. To use per module log level you also have to set `mod_hash_size`.

Default value is 0.

#### Set `mod_level_mode` parameter

    ...
    modparam("debugger", "mod_level_mode", 1)
    ...

### `mod_level` (str)

Specify module log level - the value must be in the format: modulename=level. The parameter can be set many times. For core log level, use module name 'core'. You also must enable `mod_level_mode` and `mod_hash_size`.

#### Set `mod_level` parameter

    ...
    modparam("debugger", "mod_level", "core=3")
    modparam("debugger", "mod_level", "tm=3")
    ...

### `mod_facility_mode` (int)

Enable or disable per module log facility (0 - disabled, 1 - enabled). This parameter is tunable via the Kamailio config framework. To use per module log facility you also have to set `mod_hash_size`.

Default value is 0.

#### Set `mod_facility_mode` parameter

    ...
    modparam("debugger", "mod_facility_mode", 1)
    ...

### `mod_facility` (str)

Specify module log facility - the value must be in the format: modulename=facility. The parameter can be set many times. For core log facility, use module name 'core'. You also must enable `mod_facility_mode` and `mod_hash_size`.

NOTE: See the syslog() library call for facility names (http://linux.die.net/man/3/syslog). The most used facilities are LOG_LOCAL[0-7].

#### Set `mod_facility` parameter

    ...
    modparam("debugger", "mod_facility", "core=LOG_LOCAL0")
    modparam("debugger", "mod_facility", "debugger=LOG_LOCAL1")
    ...

### `log_assign` (int)

Enable or disable log assign actions on config (0 - disabled, 1 - enabled).

Default value is 0.

#### Set `log_assign` parameter

    ...
    modparam("debugger", "log_assign", 1)
    ...

### `cfgpkgcheck` (int)

If set, before each config action is done pkg memory check, useful to detect buffer overflows.

Default value is 0 (disabled).

#### Set `cfgpkgcheck` parameter

    ...
    modparam("debugger", "cfgpkgcheck", 1)
    ...

### `reset_msgid` (int)

Used to enable or disable the ability to reset the msgid ($mi) through the dbg.reset_msgid RPC command. (0 - disabled, 1 - enabled).

Default value is 0 - feature disabled.

#### Set `reset_msgid` parameter

    ...
    modparam("debugger", "reset_msgid", 1)
    ...

### `cfgtest` (int)

Control whether the cfgt module is enabled or disabled at startup. Module cfgt needs to be loaded before.

Default value is 0 (disabled).

#### Set `cfgtest` parameter

    ...
    loadmodule "cfgt.so"
    modparam("debugger", "cfgtest", 1)
    ...

## Functions

### `dbg_breakpoint(mode)`

Anchor a breakpoint at the current line of the config (the one on which this function is called). The 'mode' specifies whether the breakpoint is enabled (1) or disabled (0) at startup.

Note that this version of the module does not export this anchors to RPC for interactive debugging (temporarily disabled).

#### `dbg_breakpoint` usage

    ...
    if($si=="10.0.0.10")
    	dbg_breakpoint("1");
    ...

### `dbg_pv_dump([mask] [, level])`

Prints the content of pv_cache on json format. Defaults are mask=31 and level = "L_DBG"

mask - Controls the content to dump:

- 1 - dump null values

- 2 - dump avp vars

- 4 - dump script vars

- 8 - dump xavp vars

- 16 - dump DP_OTHER vars

level - The level that will be used in LOG function. It can be:

- L_ALERT - log level -5

- L_BUG - log level -4

- L_CRIT - log level -3

- L_ERR - log level -1

- L_WARN - log level 0

- L_NOTICE - log level 1

- L_INFO - log level 2

- L_DBG - log level 3

#### `dbg_pv_dump` usage

    ...
    $var(temp) = 1;
    $avp(s:more_avp) = 2;
    $avp(s:more_avp) = 3;
    $xavp(x=>more) = "bye";
    $xavp(x[0]=>more) = "hi";
    $xavp(x[0]=>other) = 1;
    $xavp(x[0]=>other) = 2;
    $xavp(x=>different) = "foo";
    $var(empty) = $null;

    dbg_pv_dump(30, "L_DBG");
    ...

Output

    ...
     4(30943) DEBUG: debugger [debugger_api.c:1613]: dbg_dump_json(): {"$sp":37597,"$var(rc)":0,"$var(temp)":1,"$avp(more_avp)":[3,2],"$si":"127.0.0.1","$rc":0,"$xavp(x)":[{"different":["foo"]},{"other":[2,1],"more":["hi","bye"]}],"$T_branch_idx":0,"$var(empty)":0}
     ...

### `dbg_sip_msg([log_level], [facility])`

Prints how the sip message would look like if it would be sent out at that point in the config(i.e. if the current lump lists would have been applied at that point in the config). It also prints a diff list for both header and body of sip msg which contain the lump lists content. The lumps deleted are printed with "-" sign whereas the lumps added have no sign. The config line where the function has been called is also printed.

NOTE that dbg_sip_msg function does not modify the initially received SIP message. Just displays how it WOULD look like if it were to send it at that point.

NOTE that the lump lists are usually applied only once, just before sending, to spare message reparse processing. All the changes present in lump list are applied on the initially received SIP message. One can force the lump application using msg_apply_changes() function from textopsx module.

#### `dbg_sip_msg` usage

    ...
        dbg_sip_msg();
        dbg_sip_msg("L_ERR");
        dbg_sip_msg("L_ERR", "LOG_LOCAL0");
    ...

Output when dbg_sip_msg("L_ERR") is called after append_hf("P-Hint: My hint\r\n"); remove_hf("Contact");

    ERROR: debugger [debugger_mod.c:467]: w_dbg_sip_msg(): CONFIG LINE 338
    ------------------------- START OF SIP message debug --------------------------
    OPTIONS sip:nobody@127.0.0.1 SIP/2.0
    Via: SIP/2.0/UDP 127.0.1.1:56872;branch=z9hG4bK.6d7c487a;rport;alias
    From: sip:sipsak@127.0.1.1:56872;tag=188b7433
    To: sip:nobody@127.0.0.1
    Call-ID: 411792435@127.0.1.1
    CSeq: 1 OPTIONS
    Content-Length: 0
    Max-Forwards: 70
    User-Agent: sipsak 0.9.6
    Accept: text/plain
    P-Hint: My hintt

    ------------------------------ SIP header diffs -------------------------------
    - Contact: sip:sipsak@127.0.1.1:56872
    P-Hint: My hint
    ------------------------------- SIP body diffs --------------------------------
    -------------------------- END OF SIP message debug ---------------------------

## RPC Commands

### `dbg.ls`

List &kamailio; processes with info related to interactive debugging.

Name: dbg.list

Parameters:

- _pid_ : pid for which to list the details. If 'pid' is omitted then will print for all processes.

Examples of use with &sercmd;:

    		dbg.ls
    		dbg.ls 1234

### `dbg.trace`

Control config script running trace.

Name: dbg.trace

Parameters:

- _cmd_ : command can be 'on' or 'off' to enable or disable tracing for one or all processes.

- _pid_ : pid for which to list the details. If 'pid' is omitted, then details for all processes will be printed.

Examples for using with &sercmd;:

    		dbg.trace on
    		dbg.trace off
    		dbg.trace on 1234

### `dbg.bp`

Control breakpoints and config execution.

Name: dbg.bp

Parameters:

- _cmd_ : command, see next section for the list of available values.

- _pid_ : process id for which to apply the command. If 'pid' is omitted, then the inner command will be applied to all processes.

- _params_ : extra params specific for each command.

Commands:

- on - turn on breakpoints. Pid parameter is optional.

- off - turn off breakpoints. Pid parameter is optional.

- keep - keep breakpoints only for pid given as parameter

- release - disable breakpoints for processes that are not waiting at a breakpoint. Pid parameter is optional.

- next - run the action under breakpoint and stop at next one (step by step execution). Pid parameter is mandatory.

- move - run the action under breakpoint and remove the rest of breakpoints (continue execution without stopping again at next actions). Pid parameter is mandatory.

- show - print details about the current breakpoint for pid. Pid parameter is mandatory.

- eval - evaluate a pseudo-variable and print the result in RPC Pid parameter is mandatory.

- log - evaluate a pseudo-variable and print the result in SIP router logs. Pid parameter is mandatory.

Examples for using with &sercmd;:

    		dbg.bp off
    		dbg.bp on
    		dbg.bp release
    		dbg.bp on 1234
    		dbg.bp eval 1234 $fu
    		dbg.bp move 1234

### `dbg.mod_level`

Specify module log level.

Name: dbg.mod_level

Parameters:

- _module_ : For core log level, use module name 'core'

- _level_ : integer

Examples of use with &sercmd;:

    		dbg.mod_level core 3
    		dbg.mod_level tm 3

### `dbg.reset_msgid`

Resets the message sequence ($mi). Internally there is no real change. This can be useful for unit test cases in order to be able to replicate exactly the same kamailio output. You need to set the debugger parameter reset_msgid to 1 to activate this functionality.

Name: dbg.reset_msgid

Examples of use with &sercmd;:

    		dbg.reset_msgid

### `dbg.set_mod_level`

Set the module log level. If module does not exist in kamailio, the entry in the level hashtable is still added for the bogus module.

Name: dbg.set_mod_level

Examples of use with &sercmd;:

    			dbg.set_mod_level core 1

### `dbg.set_mod_facility`

Set the module log facility. If module does not exist in kamailio, the entry in the facility hashtable is still added for the bogus module.

Name: dbg.set_mod_facility

Examples of use with &sercmd;:

    			dbg.set_mod_facility core LOG_LOCAL1

### `dbg.get_mod_level`

Get the module log level. If mod_name does not exist in the level hashtable, returns the config file value.

Name: dbg.get_mod_level

Examples of use with &sercmd;:

    			dbg.get_mod_level core

### `dbg.get_mod_facility`

Get the module log facility. If mod_name does not exist in the facility hashtable, returns the config file value.

Name: dbg.get_mod_facility

Examples of use with &sercmd;:

    			dbg.get_mod_facility core

## Usage

A common usage is to investigate the execution path for a specific SIP message. Just enable cfg running trace, send the message and watch the logs.

Another typical usage is to do interactive debugging and run each line of the route blocks of the configuration file step-by-step for a particular SIP message.

You need to connect using &sercmd; (or other RPC client) to &kamailio;. Then you can enable cfg breakpoints and send the SIP message. One process will be in waiting state ('state' field different than 0 when you do dbg.ls). Calling dbg.release will set the other &kamailio; processes in no-breakpoint mode so they can process other SIP messages without need to interact with them.

A process blocked at a breakpoint is waiting for a command. Use 'dbg.bp next pid' to execute the current action and stop at the next one. 'dbg.bp eval pid PV' can be used to retrieve the value of PV. Once you are done and want to continue the execution of the config without interaction use 'dbg.bp move pid'.

Example of a session:

    ...
    &sercmd;> dbg.ls
    {
    	entry: 0
    	pid: 6393
    	set: 3
    	state: 0
    	in.pid: 0
    	in.cmd: 0
    }
    {
    	entry: 1
    	pid: 6394
    	set: 3
    	state: 0
    	in.pid: 0
    	in.cmd: 0
    }
    ...
    {
    	entry: 9
    	pid: 6402
    	set: 3
    	state: 1
    	in.pid: 0
    	in.cmd: 0
    }

    &sercmd;> dbg.bp show 6402
    at bkp [/etc/kamailio/debugger.cfg:369] a=6 n=route

    &sercmd;> dbg.bp next 6402
    exec [/etc/kamailio/debugger.cfg:369] a=6 n=route

    &sercmd;> dbg.bp next 6402
    exec [/etc/kamailio/debugger.cfg:462] a=17 n=if

    &sercmd;> dbg.bp eval 6402 $fu
    $fu : t=str v=sip:test@kamailio.org

    &sercmd;> dbg.bp move 6402
    200 ok
    ...

Running the config trace looks like:

    ...
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=368 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=461 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=456 a=26 n=mf_process_maxfwd_header
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=466 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=461 a=27 n=sanity_check
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=371 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=659 a=3 n=return
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=374 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=501 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=470 a=25 n=has_totag
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=386 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=379 a=26 n=is_method
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=386 a=25 n=t_check_trans
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=389 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=643 a=3 n=return
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=393 a=26 n=remove_hf
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=398 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=394 a=26 n=is_method
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=404 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=398 a=26 n=is_method
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=404 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=682 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=409 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=575 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=550 a=26 n=is_method
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=551 a=3 n=return
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=412 a=6 n=route
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=518 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=505 a=26 n=is_method
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=513 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=507 a=42 n=isflagset
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=516 a=17 n=if
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=513 a=26 n=save
     9(6285) ERROR: *** cfgtrace: c=[/etc/kamailio/debugger.cfg] l=516 a=3 n=exit
    ...

The above example is for a registration with default config for version 3.1.0, without authentication. Listed fields are: 'c' - config file; 'l' - line; 'a' - internal action id; 'n' - name of executed action. 'ERROR' prefix is printed because these messages were sent to the L_ERR log level.