docs: Add Protocol.md with information on host / firmware communication

Signed-off-by: Kevin O'Connor <kevin@koconnor.net>

docs/Overview.md

@@ -4,6 +4,9 @@ installing, and running Klipper.
 See [code overview](Code_Overview.md) for developer information on the
 structure and layout of the Klipper code.
 
+See [protocol](Protocol.md) for developer information on the messaging
+protocol between host and firmware.
+
 See [debugging](Debugging.md) for developer information on how to test
 and debug Klipper.
 

docs/Protocol.md

@@ -0,0 +1,296 @@
+Klipper uses a binary protocol for communication between host and
+firmware. This page provides a high-level description of that
+protocol.
+
+The goal of the protocol is to enable an error-free communication
+channel between the host and firmware that is low-latency,
+low-bandwidth, and low-complexity for the firmware.
+
+The protocol acts as a
+[RPC](https://en.wikipedia.org/wiki/Remote_procedure_call) mechanism
+between firmware and host. The firmware declares the commands that the
+host may invoke along with the response messages that it can
+generate. The host uses that information to command the firmware to
+perform actions and to interpret the results.
+
+Firmware Interface
+==================
+
+Declaring commands
+------------------
+
+The firmware declares a "command" by using the DECL_COMMAND() macro in
+the C code. For example:
+
+```
+DECL_COMMAND(command_set_digital_out, "set_digital_out pin=%u value=%c");
+```
+
+The above declares a command named "set_digital_out". This allows the
+host to "invoke" this command which would cause the
+command_set_digital_out() C function will be executed in the
+firmware. The above also indicates that the command takes two integer
+parameters. When the command_set_digital_out() C code is executed, it
+will be passed an array containing these two integers - the first
+corresponding to the 'pin' and the second corresponding to the
+'value'.
+
+In general, the parameters are described with printf() style
+parameters (eg, "%u"). In the above example, "value=" is a parameter
+name and "%c" indicates the parameter is an integer. The parameter
+name is used as documentation. In this example, the "%c" is used as
+documentation to indicate the expected integer is 1 byte in size (the
+declared integer size does not impact the parsing or encoding).
+
+At firmware compile time, the build will collect all commands declared
+with DECL_COMMAND(), determine their parameters, and arrange for them
+to be callable.
+
+Declaring responses
+-------------------
+
+To send information from the firmware to the host a "response" is
+generated. These are both declared and transmitted using the sendf() C
+macro. For example:
+
+```
+sendf("status clock=%u status=%c", sched_read_time(), sched_is_shutdown());
+```
+
+The above transmits a "status" response message that contains two
+integer parameters ("clock" and "status"). At firmware compile time
+the build automatically finds all sendf() calls and generates encoders
+for them. The first parameter of the sendf() function describes the
+response and it is in the same format as command declarations.
+
+The host can arrange to register a callback function for each
+response. So, in effect, commands allow the host to invoke C functions
+in the firmware and responses allow the firmware to invoke code in the
+host.
+
+The firmware should only invoke sendf() from command or task handlers,
+and it should not be invoked from interrupts or timers. The firmware
+does not need to issue a sendf() from a command, it is not limited in
+the number of times sendf() may be invoked, and it may invoke sendf()
+at any time from a task handler.
+
+### Output responses
+
+To simplify debugging, the firmware also has an output() C
+function. For example:
+
+```
+output("The value of %u is %s with size %u.", x, buf, buf_len);
+```
+
+The output() function is similar in usage to printf() - it is intended
+to generate and format arbitrary messages for human consumption. It is
+a wrapper around sendf() and as with sendf() it should not be called
+from interrupts or timers.
+
+Low-level message encoding
+==========================
+
+To accomplish the above RPC mechanism, each command and response is
+encoded into a binary format for transmission. This section describes
+the transmission system.
+
+Message Blocks
+--------------
+
+All data sent from host to firmware and vice-versa are contained in
+"message blocks". A message block has a two byte header and a three
+byte trailer.  The format of a message block is:
+
+```
+<1 byte length><1 byte sequence><n-byte content><2 byte crc><1 byte sync>
+```
+
+The length byte contains the number of bytes in the message block
+including the header and trailer bytes (thus the minimum message
+length is 5 bytes). The maximum message block length is currently 64
+bytes. The sequence byte contains a 4 bit sequence number in the
+low-order bits and the high-order bits always contain 0x10 (the
+high-order bits are reserved for future use). The content bytes
+contain arbitrary data and its format is described in the following
+section. The crc bytes contain a 16bit CCITT
+[CRC](https://en.wikipedia.org/wiki/Cyclic_redundancy_check) of the
+message block including the header bytes but excluding the trailer
+bytes. The sync byte is 0x7e.
+
+The format of the message block is inspired by
+[HDLC](https://en.wikipedia.org/wiki/High-Level_Data_Link_Control)
+message frames. Like in HDLC, the message block may optionally contain
+an additional sync character at the start of the block. Unlike in
+HDLC, a sync character is not exclusive to the framing and may be
+present in the message block content.
+
+Message Block Contents
+----------------------
+
+Each message block sent from host to firmware contains a series of
+zero or more message commands in its contents. Each command starts
+with a Variable Length Quantity (VLQ) encoded command-id followed by
+zero or more VLQ parameters for the given command. So, the contents of
+an example message block might look like:
+
+```
+<id_cmd_a><param1><id_cmd_b><param1><param2><id_cmd_c><id_cmd_d>
+```
+
+In order to encode and parse the message contents, both the host and
+firmware must agree on a "data dictionary". The data dictionary
+associates high-level commands with specific integer command-ids along
+with the number of parameters that the command takes. When processing
+the data, the parser will know to expect a specific number of VLQ
+encoded parameters following a given command. So, in the above
+example, both the host and firmware would know that "id_cmd_a" is
+always followed by exactly one parameter, "id_cmd_b" two parameters,
+and "id_cmd_c" / "id_cmd_d" zero parameters.
+
+The message contents for blocks sent from firmware to host follow the
+same format. The identifiers in these messages are "response ids", but
+they serve the same purpose and follow the same encoding rules. In
+practice, message blocks sent from the firmware to the host never
+contain more than one response in the message block contents.
+
+### Variable Length Quantities
+
+See the [wikipedia article](https://en.wikipedia.org/wiki/Variable-length_quantity)
+for more information on the general format of VLQ encoded
+integers. Klipper uses an encoding scheme that supports both positive
+and negative integers. Integers close to zero use less bytes to encode
+and positive integers typically encode using less bytes than negative
+integers. The following table shows the number of bytes each integer
+takes to encode:
+
+| Integer                   | Encoded size |
+|---------------------------|--------------|
+| -32 .. 95                 | 1            |
+| -4096 .. 12287            | 2            |
+| -524288 .. 1572863        | 3            |
+| -67108864 .. 201326591    | 4            |
+| -2147483648 .. 4294967295 | 5            |
+
+### Variable length strings
+
+As an exception to the above encoding rules, if a parameter to a
+command or response is a dynamic string then the parameter is not
+encoded as a simple VLQ integer. Instead it is encoded by transmitting
+the length as a VLQ encoded integer followed by the contents itself:
+
+```
+<VLQ encoded length><n-byte contents>
+```
+
+The data dictionary includes information on each parameter so that
+both the host and firmware know which command parameters use simple
+VLQ encoding and which parameters use string encoding.
+
+Data Dictionary
+===============
+
+In order for meaningful communications to be established between
+firmware and host, both sides must agree on a "data dictionary". This
+data dictionary contains the integer identifiers for commands and
+responses along with the number of parameters and the types of
+parameters for each command/response.
+
+At compile time the firmware build uses the contents of DECL_COMMAND()
+and sendf() macros to generate the data dictionary. The build
+automatically assigns unique identifiers to each command and
+response. The data dictionary maps the high-level names (eg,
+"get_status") to the integer command ids. This allows the host and
+firmware to use descriptive ASCII names while still using minimal
+bandwidth.
+
+The host queries the data dictionary when it first connects to the
+firmware. Once the host downloads the data dictionary from the
+firmware, it uses that data dictionary to encode all commands and to
+parse all responses from the firmware. The host must therefore handle
+a dynamic data dictionary. However, to keep the firmware simple, the
+firmware always uses its static (compiled in) data dictionary.
+
+The data dictionary is queried by sending "identify" commands to the
+firmware. The firmware will respond to each identify command with an
+"identify_response" message. Since these two commands are needed prior
+to obtaining the data dictionary, their integer ids and parameter
+types are hard-coded in both the firmware and the host. The
+"identify_response" response id is 0, the "identify" command id
+is 1. Other than having hard-coded ids the identify command and its
+response are declared and transmitted the same way as other commands
+and responses. No other command or response is hard-coded.
+
+The format of the transmitted data dictionary itself is a zlib
+compressed JSON string. The firmware compile process generates the
+string, compresses it, and stores it in the text section of the
+firmware. The data dictionary can be much larger than the maximum
+message block size - the host downloads it by sending multiple
+identify commands requesting progressive chunks of the data
+dictionary. Once all chunks are obtained the host will assemble the
+chunks, uncompress the data, and parse the contents.
+
+In addition to information on the communication protocol, the data
+dictionary also contains firmware version, configuration, and other
+useful information.
+
+Static Strings
+--------------
+
+To reduce bandwidth the data dictionary also contains a set of static
+strings known to the firmware. This is useful when sending messages
+from firmware to host. For example, if the firmware were to run:
+
+```
+shutdown("Unable to handle command");
+```
+
+The error message can be encoded and sent using a single VLQ. The host
+uses the data dictionary to resolve the VLQ encoded static string id
+to the associated message in the data dictionary.
+
+Message flow
+============
+
+Message commands sent from host to firmware are intended to be
+error-free. The firmware will check the CRC and sequence numbers in
+each message block to ensure the commands are accurate and
+in-order. The firmware always processes message blocks in-order -
+should it receive a block out-of-order it will discard it and any
+other out-of-order blocks until it receives blocks with the correct
+sequencing.
+
+The low-level host code implements an automatic retransmission system
+for lost and corrupt message blocks sent to the firmware. To
+facilitate this, the firmware transmits an "ack message block" after
+each successfully received message block. The host schedules a timeout
+after sending each block and it will retransmit should the timeout
+expire without receiving a corresponding "ack". In addition, if the
+firmware detects a corrupt or out-of-order block it may transmit a
+"nak message block" to facilitate fast retransmission.
+
+An "ack" is a message block with empty content (ie, a 5 byte message
+block) and a sequence number greater than the last received host
+sequence number. A "nak" is a message block with empty content and a
+sequence number less than the last received host sequence number.
+
+The protocol facilitates a "window" transmission system so that the
+host can have many outstanding message blocks in-flight at a
+time. (This is in addition to the many commands that may be present in
+a given message block.) This allows maximum bandwidth utilization even
+in the event of transmission latency. The timeout, retransmit,
+windowing, and ack mechanism are inspired by similar mechanisms in
+[TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol).
+
+In the other direction, message blocks sent from firmware to host are
+designed to be error-free, but they do not have assured
+transmission. (Responses should not be corrupt, but they may go
+missing.) This is done to keep the implementation in the firmware
+simple. There is no automatic retransmission system for responses -
+the high-level code is expected to be capable of handling an
+occasional missing response (usually by re-requesting the content or
+setting up a recurring schedule of response transmission). The
+sequence number field in message blocks sent to the host is always one
+greater than the last received sequence number of message blocks
+received from the host. It is not used to track sequences of response
+message blocks.