Protocol
=========

The language server expects to communicate with a client using standard input and standard
output. Because of this, it is the client's responsibility to start the language server as it
desires. To allow regular use of standard input and standard output for debugging purposes, all data
in the protocol starts with the null-byte. Everything else is considered UTF-8-encoded text which is
to be shown to the user.

The protocol used in the language server is based on s-expressions in LISP. Therefore, it can
represent four basic data types: integers, strings, symbols and cons-cells. A cons-cell is an object
containing two members, usually referred to as `car` and `cdr` or `first` and `rest`. These members
may refer to any instance of the data types in the protocol, including other cons-cells. They can
therefore be used to form more complex data structures, such as singly linked lists. A singly linked
list is implemented by creating one cons-cell for each element in the list. The list content is
stored in the `car` or `first` member of each cons cell. `cdr` or `rest` is used to point to the
next cons-cell in the list. The last element contains the symbol `nil`, which is the LISP equivalent
of `null` in C++ or Java.

S-expressions use the same textual representations as in many LISP dialects, such as Emacs
LISP. *Strings* and *numbers* are represented in the same way string- and numeric literals are
represented in most programming languages. *Symbols* are similar to identifiers in programming
languages, they are a sequence of non-whitespace characters that do not start with digits or a dash
(since that is used for unary negation). *Cons-cells* are represented by two primitives enclosed in
parenthesis and separated by a dot. `(10 . 11)` is a cons-cell containing the numbers 10 and 11. In
this notation, a linked list containing two elements is represented as follows: `(10 . (11 . nil))`.
This notation quickly becomes impractical for large lists. To remedy this, linked lists can also be
written as a sequence of s-expressions separated by whitespace enclosed in parenthesis. The list
above can then be written as `(10 11)`, which is equivalent to `(10 . (11. nil))`.

The protocol is encoded in binary and based around messages. A message is a complete s-expression,
usually a list of some sort. As mentioned earlier, messages always start with a null-byte. The
null-byte is followed by the length of the message, in bytes, encoded as a 32-bit (= 4-bytes)
integer in big-endian format. The *length* bytes following the size constitutes the message body,
which contains a single s-expression. S-expressions always start with a single byte indicating the
type of the stored element, followed by the actual data. The supported types are as follows:

* `0x00`: the symbol `nil`. No additional data is required. `nil` is a commonly used symbol,
  and as such it has a special encoding.
* `0x01`: a cons-cell. Following this byte are two additional s-expressions. The first one
  is the contents of the `car` or `first` member, the second is the contents of the `cdr`
  or `rest` member.
* `0x02`: a number. The following four bytes contain the actual value of a 32-bit number stored
  in big-endian format, just like the size.
* `0x03`: a string. The first four bytes contain the length of the string (in bytes) as a 32-bit
  number in big-endian format. Following the size is the bytes constituting the string. These are
  encoded in UTF-8.
* `0x04`: a symbol. This encoding is used for a symbol that has not previously been sent. The first
  4 bytes is an integer containing an id to use for this symbol in future communication. Following
  the id is a string (without the type-mark) containing the name of the symbol.
* `0x05`: a symbol. This encoding is used for symbols that have previously been sent encoded as `0x04`.
  This representation only contains 4 bytes containing the identifier of the symbol. This identifier
  should be the same as a symbol previously sent using `0x04` in either direction.

For example, the message `(a 10 a "b")`, or equivalently `(a . (10 . (a . ("b" . nil))))`
would be encoded as follows:

```
0x00                     //       // Start of message.
0x00 0x00 0x00 0x1F      //       // Length of the message, 31 bytes.
0x01                     // (     // Start of a cons-cell.
 0x04                    // a     // A new symbol...
  0x00 0x00 0x00 0x01    //       //  with index 1...
  0x00 0x00 0x00 0x01    //       //  identifier length 1...
  0x61                   //       //  identifier content "a"
 0x01                    // . (   // Start of a cons-cell.
  0x02                   // 10    // A number...
   0x00 0x00 0x00 0x0A   //       // value 10
  0x01                   // . (   // Start of a cons-cell.
   0x05                  // a     // A previously seen symbol...
    0x00 0x00 0x00 0x01  //       //  with index 1
   0x01                  // . (   // Start of a cons-cell.
    0x03                 // "b"   // A string...
     0x00 0x00 0x00 0x01 //       //  with length 1...
     0x62                //       //  and content "b".
   0x00                  // . nil // The symbol nil.
                         // ))))  //
```

Messages
---------

Using the protocol described above, it is possible to send s-expressions as messages from the
language server to a text editor and vice versa. Both the language server and the editor may send
messages at any time, but communication is usually initiated by the client. In the language server,
messages are always lists where the first element is a symbol describing the message type.

The language server associates each open file with an integer identifier. This is decided by the
client when a new file is opened and used to refer to that file in all further communication. If an
identifier is reused, the previous file is closed. Each open file contains the following state: the
complete contents of the file, the index of the last edit operation and an approximate location of
the user's cursor. The index of the last edit operation is passed along in `color` messages, so that
the client can determine which version of the file the language server is referring to. The edit
number is initially set to zero, and is changed every time the client sends an `edit` message. The
cursor location is maintained so that the language server is able to prioritize sending syntax
coloring information for text visible to the user (i.e. close to the cursor).

The following messages can be sent from the text editor to the language server:

* `(quit)`: Asks the language server to terminate.
* `(supported type)`: Ask the language server if the file extension *type* is supported. *type* is
  a string containing the file extension without a dot. The language server sends a `(supported type result)`
  message containing the result.
* `(open id path content pos)`: Open a new file, *path* (a string), and give it the identifier *id*
  (a number). The current contents of the file are sent as a string in *content*. *pos* is
  optional. If present, it is the current position of the cursor within this file.
* `(close id)`: Closes the file with identifier *id* (a number).
* `(edit id edit from to text)`: An edit has been made to the file with identifier *id* (a
  number). The text between *from* and *to* (numbers, zero based character indexes) has been
  replaced with *text* (a string). *edit* is the edit number of this edit. Initially opening a file
  is considered to be edit number zero.
* `(point id pos)`: The user has moved their cursor in the file with identifier *id* to *pos* (a
  number, zero based character index). This message does not need to be sent every time the user
  moves the cursor. As long as the language server's knowledge of the cursor is close enough, no
  updates need to be sent.
* `(indent id pos)`: Request indentation information for character *pos* (a number) in the file with
  identifier *id* (a number). The language server replies to this message with an `(indent file value)` message.
* `(color id)`: Send all coloring information for file *id* once more.
* `(complete-name string context)`: Reply with a list of names matching *string*. *string* is parsed using the
  default syntax for names in Storm. Names are resolved using a scope that looks for names inside `core`
  and optionally in the package corresponding *context*, if provided.
* `(documentation name context)`: Reply with the documentation for *name*. *name* is parsed just like
  in the `complete-name` message.

The following messages can be sent from the language server to the text editor:

* `(supported type result)`: Sent as a reply to `(supported type)`. Indicates whether the previously
  queried file extension *type* is supported or not. *result* is either `t` or `nil`, indicating
  *supported* and *not supported* respectively.
* `(color id edit start colors...)`: Color a part of the file with identifier *id* as denoted by
  colors. The character indexes in this message refers the text as it was after edit number *edit*.
  If additional edit operations have been performed since, indexes may need to be adjusted by the
  editor. In the Emacs plugin, this is done by remembering the last 20 or so edit
  operations. *start* (a number) is the zero based index of the first character to highlight. The
  remaining elements in the message (labeled *colors...*) describe the colors. The first element in
  *colors* contain the number of characters to color. They shall be colored in the color indicated
  by the next element in *colors*. If there are more elements in *colors*, they are interpreted in
  the same way. For example: `(color 1 1 5 10 comment 5 keyword)` means: start at character number
  5, color the next 10 characters as a comment. The 5 characters following those of the comment
  shall be colored as a keyword.
* `(indent id type value)`: Sent as a reply to `(indent id pos)`. Tells the indentation for the last
  `indentation` request sent for file *id*. *type* is either `level` or `as`, and describes the
  meaning of the number *value*. If *type* is `level`, the line shall be indented *value* levels (a
  level is defined by the client). If *type* is `as`, the line shall be indented as the line
  containing the character at offset *value* (zero based index).
* `(complete-name name...)`: Sent as a reply to `(complete-name string context)` to provide a list
  of possible matches. The matches provided are generally only one level deep, meaning that
  the language server will not traverse the entire name tree to provide all possible completions since
  that would be very expensive and most likely not what is actually desired.
* `(documentation name data)`: Sent as a reply to `(documentation name context)`. If *name* did not exist,
  then *data* is `nil`. Otherwise, *data* is a list with elements of the form described below.

The following colors are available to the language server:

* `comment`: Used for comments. Emacs uses `font-lock-comment-face`.
* `delimiter`: Used for delimiters. Emacs uses `font-lock-delimiter-face`.
* `string`: Used for string literals. Emacs uses `font-lock-string-face`.
* `constant`: Used for constants, such as integer literals. Emacs uses `font-lock-constant-face`.
* `keyword`: Used for keywords in a language. Emacs uses `font-lock-keyword-face`.
* `fn-name`: Used for names of functions in a language. Emacs uses `font-lock-function-face`.
* `var-name`: Used for names of variables in a language. Emacs uses `font-lock-variable-face`.
* `type-name`: Used for names of types in a language. Emacs uses `font-lock-type-face`.

The data inside the `documentation` is a list containing the following data:

* `name`: A string containing the name of the entity.
* `params`: A list of parameters to this entity. Each of the parameters are a list, `(name type-name type-ref ref)`,
  where *name* is the name of the parameter, *type-name* is a string that contains the name of the type
  shown to the user, *type-ref* is a string containing the fully-qualified name of the type, or `nil` if
  it refers to `void`. *ref* is `t` if the parameter is a reference, otherwise `nil`.
* `notes`: A list of notes for this entity. Takes the same form as `params` above, except that *type-name* does
  not necessarily refer to a type (and thus has a *type-ref* that is `nil`). Any entity is valid as a
  note. The `type` and `ref` may be left out for notes, which means that no type shall be shown for this
  note at all. Note: The return value for functions is usually returned as a note.
* `visibility`: Visibility of this entity as a cons cell `(type . title)`, or `nil` if none was provided.
  *title* is a string suitable to show the user and *type* is the type implementing this visibility.
* `body`: The body text of the documentation as a string.
* `pos`: The location in source code of this element, as a list `(filename start-offset end-offset)`, where
  `filename` is the filename, and `start-offset` and `end-offset` is the start and end character offsets the
  file, or `nil` if no information is available.
* `refs`: A list of references to other things in the system. Each element has the form `(name . contents)`,
  where *name* is the name of this category and *contents* is a list with the actual contents. Each element
  there has the form `(name . title)` where *name* is the full name of the member and *title* is a short
  summary that is suitable to show the user.
