SHV RPC

SHV RPC is RPC framework build around ChainPack packing schema. RPC call is realized as sending ChainPack encoded Message either via broker or peer-to-peer.

Why ChainPack? I liked XML attributes, but I dislike fact that XML attributes syntax is not XML. I like JSON brevity, but you cannot have attributes, comments, utf8 in JSON. ChainPack is attempt to choose the best from XML and JSON.

ChainPack main features:

• UTF8 string encoding
• every value can have set of named attributes, attributes values are ChainPack again.
• binary ChainPack and text Cpon representation convertible each to other, see cp2cp or ccp2cp utilities
• DateTime native type
• Blob native type

See available implementations for various programming languages and various tools.

SHV RPC concepts

SHV RPC is a protocol that allows remote control, data collection and general supervision on multiple devices. It tries to land middle ground between "feature rich" and "minimal and simple". It archives it by breaking different responsibilities to a different components. The full network is built from these components.

Messages in the network

There are three types of messages used in the Silicon Heaven RPC communication. Together they facilitate two types of communication:

• Remote procedure call (referenced as method call in SHV RPC)
• Signal broadcasting

Remote procedure calls are always confirmed, while signals are never confirmed.

Request - message used to call some method. Such messages are required to exactly identify such method and can optionally carry parameter for it. They also must specify request ID that is later used to pair response message with request.

Response - message sent by side that received some request. This message copies request ID from request message to allow calling peer to identify it as the response for that specific request.

Signal - spontaneous message sent without prior request and thus without request ID. Optionally it can carry a parameter.

SHV RPC network design

Silicon Heaven RPC is point to point protocol. There are always only two sides in the communication. Broadcast is not directly utilized (do not confuse this with signal broadcasting that is performed by SHV RPC Broker and described in next chapter).

Messages are transmitted between two sides. Both sides can send both requests as well as signals, and they should respond to requests with responses.

The bigger networks are constructed using brokers. Broker supports multiple client connections in parallel and provides exchange of messages between them.

It is a good practice to always connect sides through broker and never directly. That is because broker is the only one that needs to manage multiple connections in parallel in such case. Clients are only connected to the broker.

Clients can be simple dummy clients that only call some methods and listen for responses and signals, but there are also clients that expose methods and nodes. These are mountable clients. Mountable clients have tree of nodes where every node can have multiple methods associated with it. This tree can be discovered through method ls that needs to be implemented on every node. Methods associated with some node can be discovered with dir method that also has to be implemented on every node.

Mountable client's tree needs to be accessible to other clients through SHV broker and this is achieved by attaching it somewhere in the broker's tree. This operation is called mounting, and you need to specify mount point (node in the SHV broker's tree) when connecting client to the SHV broker. Thanks to this clients can communicate with multiple mounted clients in parallel as well as mounted client can be used by multiple clients.

SHV RPC Broker

Broker is an element in the network that allows exchange of the messages between multiple clients. To connected clients it behaves like a mountable client with exception that some SHV paths are not handled directly by it but rather propagated to some other client. The message propagation depends on its type.

Request - Broker looks up the correct mounted client it should forward message to based on the SHV path. It handles request itself if there is no such client. If mounted client is located, broker removes client's mount point from request's SHV path, it appends client ID of caller to CallerIds, and limits access level based on its configuration. The message is then sent to the located client. Broker doesn't remember this request because all the request state is contained in request meta-data.

Response is returned to the correct client based on the CallerIds contained in request. Note, that client must copy RequestId and CallerIds (with its client ID removed) from request to response meta-data, if it should be delivered back to caller.

Signal - Signals are propagated based on the subscriptions clients made beforehand. All clients are checked for the subscription and if message matches some and client has hight enough access level, then it is propagated to that client. The SHV path of the message is prefixed with mount point of the client the signal was received from. This way other clients see signals as being delivered from the correct place in the SHV tree. Signals received from not-mounted clients are simply dropped.

The API client's have available on Broker is documented in Broker API.

Brokers must maintain their own subscriptions in mounted clients that are broker so they receive signals and can propagate them further.

Access control

User can be limited from accessing some methods. The right of access is controlled by the client that handles request not by the intermediate brokers. At the same time clients don't and should not know about user accounts and thus the complete access control is in reality split to two steps. Client sends request to the broker, and it limits access level in the message based on its configuration. The message is in the end delivered to the mounted client that checks this level and either performs the method or raises error based on it. Brokers can choose to handle request with error if not even minimal browse access is given to the client.

The predefined access levels are the following:

Levels are sorted from the lowest to the highest and are understood to include all lover level rights.

User ID

Sometimes devices want to record in their logs who triggered some of their methods. It can be, for example, when some error state is cleared. The information about user, who called that method, is normally known by the brokers on the way but not by the final device, because it gets only user's access level. And thus SHV RPC provides an optional way to propagate this info to it. To enable it the client sending request must add additional ClientId field to the RPC message's meta table with empty string value. Brokers on the way extend this value by their user's identification (appended after comma). The device thus gets full range of all users used to access it. Devices can signal the need for this field with UserIDRequired error.

RPC Value

SHV RPC messages are encoded in JSON like meta-data format. There are two defined representations: human-readable CPON and binary ChainPack.

• ChainPack
• CPON

Types

A single RPC Value has one of the following types while some of them are containers and thus more complex representations can be constructed.

Conversion of all data types between CPON and ChainPack is mostly without any information lost with exception of exact encoding that is not possible to cary over (such as String and CString in ChainPack converging to just String in CPON, or CPON's HexBlob and Blob converting to ChainPack's Blob). hexadecimal numbers

Some examples in Cpon

Utf8 string with meta-information about its format.

<"format":"Date">"2023-01-02"	

ID Int.

<"type":"ID">123	

Addressbook entry

<1:"AdressBookEntry", "format":"cpon">
{
  "name": "John",
  "birth": <"format": "ISODate">"2000-12-11",
}

ChainPack

ChainPack is binary data format that caries type information with data. It is designed to be read and written in streams.

Note that different implementations can have different limitations on data representation compared to the full range ChainPack supports. Always consult your implementation's documentation to identify them.

Every value is prefixed with a single byte that specifies format of the following data, referenced as packing schema in this document. The following formats are defined:

The values from 0 to 127 are used as UInt and Int values and are more discussed in their paragraphs.

Null

Represented just with its packing schema, no additional bytes are expected.

+------+
| 0x80 |
+------+

Bool

Represented with either TRUE or FALSE packing schema. There are no additional bytes expected after it.

True:

+------+
| 0xfd |
+------+

False:

+------+
| 0xfe |
+------+

Int

Signed integers can be packed directly in packing schema for values from 0 to 63 or with packing schema Int followed by bytes with integer value encoded in a specific way.

The values from 0 to 63 can be packed directly in packing schema by adding 64 to them. The value 42 thus would be packed as 0x6a.

+--------------+
| 0x40 + Value |
+--------------+
Value ∈ <0, 63>

+------+---------+
| 0x82 | Data ...
+------+---------+

Signed integers are stored in their absolute value with sign bit. Number of bytes integer spans can be decoded from the first byte in the way described in the following visualization.

Bytes in stream must be from left to right and unsigned integer itself is included with its most significant bit in first byte that should carry it (big-endian).

 0 ...  6 bits  1  byte  |0|s|x|x|x|x|x|x|<-- LSB
 7 ... 13 bits  2  bytes |1|0|s|x|x|x|x|x| |x|x|x|x|x|x|x|x|<-- LSB
14 ... 20 bits  3  bytes |1|1|0|s|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x|<-- LSB
21 ... 27 bits  4  bytes |1|1|1|0|s|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x|<-- LSB
28+       bits  5+ bytes |1|1|1|1|n|n|n|n| |s|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x| ... <-- LSB
                    n ==  0 ->  4 bytes number (32 bit number)
                    n ==  1 ->  5 bytes number
                    n == 13 -> 17 bytes number
                    n == 14 -> RESERVED
                    n == 15 -> not used because it is TERM type

s is sign bit and x is big-endian unsigned integer bits.

The representation must strictly be sent in lowest possible bytes length.

Examples:

             4            0x4 ... len:  1  dump:  01000100
            16           0x10 ... len:  1  dump:  01010000
            64           0x40 ... len:  3  dump:  10000010|10000000|01000000
          1024          0x400 ... len:  3  dump:  10000010|10000100|00000000
          4096         0x1000 ... len:  3  dump:  10000010|10010000|00000000
         16384         0x4000 ... len:  4  dump:  10000010|11000000|01000000|00000000
        262144        0x40000 ... len:  4  dump:  10000010|11000100|00000000|00000000
       1048576       0x100000 ... len:  5  dump:  10000010|11100000|00010000|00000000|00000000
       4194304       0x400000 ... len:  5  dump:  10000010|11100000|01000000|00000000|00000000
      67108864      0x4000000 ... len:  5  dump:  10000010|11100100|00000000|00000000|00000000
     268435456     0x10000000 ... len:  6  dump:  10000010|11110000|00010000|00000000|00000000|00000000
    1073741824     0x40000000 ... len:  6  dump:  10000010|11110000|01000000|00000000|00000000|00000000
   17179869184    0x400000000 ... len:  7  dump:  10000010|11110001|00000100|00000000|00000000|00000000|00000000
   68719476736   0x1000000000 ... len:  7  dump:  10000010|11110001|00010000|00000000|00000000|00000000|00000000
  274877906944   0x4000000000 ... len:  7  dump:  10000010|11110001|01000000|00000000|00000000|00000000|00000000
 4398046511104  0x40000000000 ... len:  8  dump:  10000010|11110010|00000100|00000000|00000000|00000000|00000000|00000000
17592186044416 0x100000000000 ... len:  8  dump:  10000010|11110010|00010000|00000000|00000000|00000000|00000000|00000000
70368744177664 0x400000000000 ... len:  8  dump:  10000010|11110010|01000000|00000000|00000000|00000000|00000000|00000000
     -4 ... len:  2  dump:  10000010|01000100
    -16 ... len:  2  dump:  10000010|01010000
    -64 ... len:  3  dump:  10000010|10100000|01000000
  -1024 ... len:  3  dump:  10000010|10100100|00000000
  -4096 ... len:  3  dump:  10000010|10110000|00000000
 -16384 ... len:  4  dump:  10000010|11010000|01000000|00000000
-262144 ... len:  4  dump:  10000010|11010100|00000000|00000000

UInt

Unsigned integers can be packed directly in packing schema for values from 0 to 63 or with packing schema UInt followed by bytes with integer value.

Values packed directly to packing schema are packed as they are. The value 42 thus would be packed as 0x2a.

+--------------+
| 0x00 + Value |
+--------------+
Value ∈ <0, 63>

+------+---------+
| 0x81 | Data ...
+------+---------+

Number of bytes integer spans can be decoded from the first byte in the way described in the following visualization. Bytes in stream must be from left to right and integer itself is included with its most significant bit in first byte that should carry it (big-endian).

 0 ...  7 bits  1  byte  |0|x|x|x|x|x|x|x|<-- LSB
 8 ... 14 bits  2  bytes |1|0|x|x|x|x|x|x| |x|x|x|x|x|x|x|x|<-- LSB
15 ... 21 bits  3  bytes |1|1|0|x|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x|<-- LSB
22 ... 28 bits  4  bytes |1|1|1|0|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x|<-- LSB
29+       bits  5+ bytes |1|1|1|1|n|n|n|n| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x| |x|x|x|x|x|x|x|x| ... <-- LSB
                    n ==  0 ->  4 bytes number (32 bit number)
                    n ==  1 ->  5 bytes number
                    n == 13 -> 17 bytes number
                    n == 14 -> RESERVED
                    n == 15 -> not used because it is TERM type

s is sign bit and x is big-endian unsigned integer bits.

The representation must strictly be sent in lowest possible bytes length.

Examples:

               2u              0x2 ... len:  1  dump:  00000010
              16u             0x10 ... len:  1  dump:  00010000
             127u             0x7f ... len:  2  dump:  10000001|01111111
             128u             0x80 ... len:  3  dump:  10000001|10000000|10000000
             512u            0x200 ... len:  3  dump:  10000001|10000010|00000000
            4096u           0x1000 ... len:  3  dump:  10000001|10010000|00000000
           32768u           0x8000 ... len:  4  dump:  10000001|11000000|10000000|00000000
         1048576u         0x100000 ... len:  4  dump:  10000001|11010000|00000000|00000000
         8388608u         0x800000 ... len:  5  dump:  10000001|11100000|10000000|00000000|00000000
        33554432u        0x2000000 ... len:  5  dump:  10000001|11100010|00000000|00000000|00000000
       268435456u       0x10000000 ... len:  6  dump:  10000001|11110000|00010000|00000000|00000000|00000000
     68719476736u     0x1000000000 ... len:  7  dump:  10000001|11110001|00010000|00000000|00000000|00000000|00000000
  17592186044416u   0x100000000000 ... len:  8  dump:  10000001|11110010|00010000|00000000|00000000|00000000|00000000|00000000
 140737488355328u   0x800000000000 ... len:  8  dump:  10000001|11110010|10000000|00000000|00000000|00000000|00000000|00000000
4503599627370496u 0x10000000000000 ... len:  9  dump:  10000001|11110011|00010000|00000000|00000000|00000000|00000000|00000000|00000000

Double

Double-precision floating-point as defined in IEEE 754-2008 in little-endian.

+------+-------+-------+-------+-------+-------+-------+-------+-------+
| 0x83 | Byte0 | Byte1 | Byte2 | Byte3 | Byte4 | Byte5 | Byte6 | Byte7 |
+------+-------+-------+-------+-------+-------+-------+-------+-------+

Decimal

Exponential number of base 10 is packed as two Int types (without packing schema) where first is the mantisa and second is exponent (mantisa * 10^exponent). The second Int (exponent) can also be special value 0xff (also use as TERM) to encode special values.

+------+---       ---+---         ---+
| 0x8c | Int mantisa | Int exponent |
+------+---       ---+---         ---+

The special values when exponent is 0xff:

• mantisa == 1 is positive infinity (+INF)
• mantisa == -1 is negative infinity (-INF)
• matinsa == 0 is quiet NaN (qNaN)
• mantisa == 2 is signaling NaN (sNaN)
• Other values of mantisa are reserved

Blob

Blob is sent with UInt (without packing schema) prefixed that specifies number of data bytes. The data should be sent in little-endian.

+------+---       ---+---      ---+
| 0x85 | UInt length | Data bytes |
+------+---       ---+---      ---+

Example:

"fpowf\u0000sapofkpsaokfsa":  10001010|00010100|01100110|01110000|01101111|01110111|01100110|00000000|01110011|01100001|01110000|01101111|01100110|01101011|01110000|01110011|01100001|01101111|01101011|01100110|01110011|01100001

BlobChain

Blob packed in parts. It provides a way to pack binary data that are not of known size upfront. Data are sent in blocks with their UInt (without packing schema) size prefixed and termination is done by packing zero as data length.

+------+---       ---+---      ---+-- --+---+
| 0x8f | UInt length | Data bytes | ... | 0 |
+------+---       ---+---      ---+-- --+---+

String

UTF-8 encoded string. It is sent with UInt (without packing schema) prefixed that specifies number of data bytes (not characters!). The data must be sent in little-endian.

+------+-------------+------------+
| 0x86 | UInt length | UTF-8 data |
+------+-------------+------------+

Example:

"fpowf":  10001010|00000101|01100110|01110000|01101111|01110111|01100110

CString

UTF-8 string stream that is terminated with \0 (thus full size is not immediately known when read from stream). The string itself can't contain \0 byte inside because there is no escaping of it available.

+------+--------------+----+
| 0x8e | UTF-8 data | `\0` |
+------+--------------+----+

Example:

"fpowf":  10001110|00000101|01100110|01110000|01101111|01110111|01100110|00000000

DateTime

Date and time encoded as Int (without packing schema) with these conversion sequence:

• Take integer value msecs since 2018-02-02
• if msec part == 0 val /= 1000
• if UTC offset != 0 val = (val << 7) + (utc_offset_min / 15) -63 <= offset <= 63
• val <<= 2
• set flags for TZ and msec part

bit 0: has TZ flag
bit 1: has not msec part flag

Example:

d"2018-02-02 0:00:00.001"       len:  2  dump:  10001101|00000100
d"2018-02-02 01:00:00.001+01"   len:  3  dump:  10001101|10000010|00010001
d"2018-12-02 0:00:00"           len:  5  dump:  10001101|11100110|00111101|11011010|00000010
d"2018-01-01 0:00:00"           len:  5  dump:  10001101|11101000|10101000|10111111|11111110
d"2019-01-01 0:00:00"           len:  5  dump:  10001101|11100110|11011100|00001110|00000010
d"2020-01-01 0:00:00"           len:  6  dump:  10001101|11110000|00001110|01100000|11011100|00000010
d"2021-01-01 0:00:00"           len:  6  dump:  10001101|11110000|00010101|11101010|11110000|00000010
d"2031-01-01 0:00:00"           len:  6  dump:  10001101|11110000|01100001|00100101|10001000|00000010
d"2041-01-01 0:00:00"           len:  7  dump:  10001101|11110001|00000000|10101100|01100101|01100110|00000010
d"2041-03-04 0:00:00-1015"      len:  7  dump:  10001101|11110001|01010110|11010111|01001101|01001001|01011111
d"2041-03-04 0:00:00.123-1015"  len:  9  dump:  10001101|11110011|00000001|01010011|00111001|00000101|11100010|00110111|01011101
d"1970-01-01 0:00:00"           len:  7  dump:  10001101|11110001|10000001|01101001|11001110|10100111|11111110
d"2017-05-03 5:52:03"           len:  5  dump:  10001101|11101101|10101000|11100111|11110010
d"2017-05-03T15:52:03.923Z"     len:  7  dump:  10001101|11110001|10010110|00010011|00110100|10111110|10110100
d"2017-05-03T15:52:31.123+10"   len:  8  dump:  10001101|11110010|10001011|00001101|11100100|00101100|11011001|01011111
d"2017-05-03T15:52:03Z"         len:  5  dump:  10001101|11101101|10100110|10110101|01110010
d"2017-05-03T15:52:03.000-0130" len:  7  dump:  10001101|11110001|10000010|11010011|00110000|10001000|00010101
d"2017-05-03T15:52:03.923+00"   len:  7  dump:  10001101|11110001|10010110|00010011|00110100|10111110|10110100

List

This is sequence of other RPC values. It starts with packing schema and is terminate with TERM (0xff).

+------+---   ---+-- --+---   ---+------+
| 0x88 | Value 1 | ... | Value n | 0xff |
+------+---   ---+-- --+---   ---+------+

Example:

["a",123,true,[1,2,3],null]
10001100|10001010|00000001|01100001|10000110|01000000|01111011|10000011|10001100|01000001|01000010|01000011|11111111|10000100|11111111

Map

Encoded as sequence of String (with packing schema) key and RPC value pairs. The order of pairs is not guaranteed and should not be relied upon. The last pair must be followed by TERM (0xff) that terminates the Map.

+------+---        ---+---   ---+-- --+---        ---+---   ---+------+
| 0x89 | String key 1 | Value 1 | ... | String key n | Value n | 0xff |
+------+---        ---+---   ---+-- --+---        ---+---   ---+------+

Example:

{"bar":2,"baz":3,"foo":1}
10001101|00000011|01100010|01100001|01110010|01000010|00000011|01100010|01100001|01111010|01000011|00000011|01100110|01101111|01101111|01000001|11111111
{"bar":2,"baz":3,"foo":[11,12,13]}
10001101|00000011|01100010|01100001|01110010|01000010|00000011|01100010|01100001|01111010|01000011|00000011|01100110|01101111|01101111|10001100|01001011|01001100|01001101|11111111|11111111

IMap

Encoded as sequence of Int (with packing schema) key and RPC value pairs. The order of pairs is not guaranteed and should not be relied upon. The last pair must be followed by TERM (0xff) that terminates the IMap.

+------+---     ---+---   ---+-- --+---     ---+---   ---+------+
| 0x8a | Int key 1 | Value 1 | ... | Int key n | Value n | 0xff |
+------+---     ---+---   ---+-- --+---     ---+---   ---+------+

Example:

i{1:"foo",2:"bar",333:15}
10001110|00000001|10001010|00000011|01100110|01101111|01101111|00000010|10001010|00000011|01100010|01100001|01110010|10000001|01001101|01001111|11111111

Notice that keys are of type Int not UInt!

MetaMap

Encoded like Map and IMap but with keys being both Int and String.

+------+---               ---+---   ---+-- --+---               ---+---   ---+------+
| 0x8b | Int or String key 1 | Value 1 | ... | Int or String key n | Value n | 0xff |
+------+---               ---+---   ---+-- --+---               ---+---   ---+------+

CPON - ChainPack Object Notation

CPON is inspired by JSON. It is used for data dumping as well as user input.

CPON text should be encoded in UTF-8.

CPON supports C style comments and thus /* foo */ is just simply ignored and considered as a white space character.

RPC Value types are encoded in the following way:

Null

Represented with plain null (compatible with JSON).

Bool

Represented with either true or false (compatible with JSON).

Int

Represented as number just like in JSON (e.g. 123, -42) with additional support for common hexadecimal (e.g. 0x20) and binary representation (e.g. 0b1001).

Tools generating CPON should prefer plain number representation (thus compatible with JSON).

UInt

Represented as Int with suffix u (e.g. 123u, 0x20u, 0b1001u).

Double

Represented in scientific notation with P (or p) between significant and exponent represented as Int (e.g. 1.25p-2, -0.0625p3, 0b1001p+2).

Tools generating CPON should prefer format [-]0xh.hhhp[+|-]dd, thus significant is normalized hexadecimal number and exponent is decimal.

Decimal

Represented in scientific notation (both E and e can be used) from significant and exponent Int number, or as decimal number with decimal point (.) (e.g. 123.45, 1.2345e2, 12345E-0x2).

Tools generating CPON should prefer scientific notation with e where both significant and exponent is decimal and optionally representation with decimal point if number of inserted zeroes would be small enough.

Blob

Represented as String with b prefix where values from 0x00 to 0x1f and from 0x7f to 0xff are represented as escape sequences \hh (where hh is character's hexadecimal numeric value) with these exceptions:

Example: b"ab\31"

Tools generating CPON should prefer special escape sequences over hexadecimal numeric ones.

HexBlob

This is CPON extension on Blob type. There is no such alternative in ChainPack.

Blob is represented as series of hexadecimal numeric values in String like syntax with x prefix.

Example: x"616231"

Tools generating CPON are discouraged from using this type.

String

Represented as series of characters wrapped in ". It can contain any characters except of the following ones that are mapped to these escape sequences:

Octal, hexadecimal and Unicode escape sequences are not support (this is incompatibility with JSON).

Example: "some\tstring"

DateTime

Represented as String with d prefix where string's value is date, time and optional time zone in ISO-8601 format (e.g. d"2017-05-03T15:52:31.123")

List

Represented as sequence of other RPC values wrapped in []. Th sequence is delimited by comma (,). The additional comma is allowed after last item (JSON incompatible).

Examples: [1 2 3], [1,2,3,]

Map

Represented as pairs of String key and RPC value delimited by colon (:). All pairs are wrapped in {} and delimited by comma (,). Contrary to JSON's objects comma after last pair is allowed.

Examples: {"one": 1, "dec": 1.22,}

IMap

Reprensented as pairs of Int key and RPC value delimited by colon (:). All pairs are wrapped in {} and delimited by comma (,). Contrary to JSON's object comma after past pair is allowed.

Examples: {1: "one", 2: b"foo",}

MetaMap

Represented as pairs of String or Int key and RPC value delimited by color (:). All pairs are wrapped in <> and delimited by comma (,). Comma after past pair is allowed. MetaMap can't be alone, it must be followed by some other RPC value (that is not MetaMap).

Example: <1: "foo", "date": d"2017-05-03T15:52:31.123">42

Syntax highlighting support

Syntax definition for tree sitter can be found at https://github.com/amaanq/tree-sitter-cpon

RPC Message

Message exchanged in SHV is IMap with meta-data attached from RPC Value point of view.

There are three kinds of RPC messages defined:

• Request
• Response
• Signal

RPC message can have meta-data attribute defined.

Second part of RPC message is IMap with following possible keys.

RequestId can be any unique number assigned by side that sends request initially. It is used to pair up requests with their responses. The common approach is to just use request message counter as request ID.

CallerIds are related to the broker and unless you are implementing a broker you need to know only that they are additional identifiers for the message alongside the RequestId to pair requests with their responses and thus should be always included in the response message if they were present in the request.

The ShvPath is used to select exact node of method in the SHV tree.

AccessLevel is the way to specify access level. It is numerical with predefined range (0-63) and brokers on the way can lower this number to even further limit access. Broker are prohibited to increase this number. Access should be used to get level if this field is not present. Admin access level should be considered as the base limit if neither AccessLevel nor Access field is present.

userId is string containing information about the login names and the broker names along the RPC message path through the brokers hierarchy. The format isuser-name1:broker-name1;user-name2:broker-name2;..., for example: john@foo.bar:broker1;broker1-login:broker2. User name and broker name is delimited by :, user:broker pairs are delimited by ';'.

Access is older approach for the access control. It is assigned to request by broker according to user rights. Multiple grants can be specified and separated by comma. This is no longer the primary way and is used only for pre-SHV 3.0 peers or if additional access fields are specified.

The following are all defined AccessLevels and their Access representations:

Request

Message used to invoke remote method.

Methods invoked using this request needs to be idempotent, because RPC transport layers do not ensure deliverability. You might also want to try to send request again when you receive no response because of this.

Attributes

Keys

Examples

RPC call invocation, method switchLeft on path test/pme/849V with request ID 56 and parameter true.

<1:1,8:56,9:"test/pme/849V",10:"switchLeft">i{1:true}

Response

Response to Request

Attributes

Keys

RPC Error

RPC Error is IMap with following keys defined

Error codes

Examples

Successful response to method call from example above

<1:1,8:56>i{2:true}

Exception when unknown method is called

<1:1,8:11>i{3:i{1:8,2:"method: foo path:  what: Method: 'foo' on path 'shv/cze' doesn't exist"}}

Signal

Spontaneous message sent without prior request and thus without RequestId. It is used mainly notify clients that some technological value had changed without need to poll.

Attributes

Keys

Warning: all signal names ending with chng (that includes chng and others such as fchng) are considered as property changes of the method they are associated with, and thus cache can use to update its state them instead of method call. If your method emits signals with different parameter than response's result then do not use signal names ending with chng.

Examples

Spontaneous notification about change of status/motorMoving property to true.

<1:1,9:"shv/test/pme/849V/status/motorMoving",10:"chng",11:"get">i{1:true}

SHV Type Info

TypeInfo =
<"version": 4> {
  "devicePaths": {
    DevicePath: DeviceName*
  }
  "deviceDescriptions": {
    DeviceName: DeviceDescription*
  }
  "types": {
    DefinedTypeName: TypeDescription*
  }
}

DevicePath = String

DeviceName = String

DevicePath = {   "properties": [     PropertyDescription*   ]? }

DeviceDescription = {
  "properties": [
    PropertyDescription*
  ]?
}

PropertyDescription = {
  "name": String
  "typeName": TypeName
  "label": String?
  "description": String?
  "methods": [
    MethodDescription*
  ]?
  "autoload": Bool?
  "monitored": Bool?
}

MethodDescription = {
  "name": String
  "accessGrant": ("bws" | "rd" | "wr" | "cmd" | "cfg" | "svc" | "ssvc" | "dev" | "su")
  "flags": MethodFlags
  "signature": MethodSignature
  "description": String?
  "tags": {
    TagName: Value*
  }?
}

MethodFlags = bitfield
  Signal = 1
  Getter = 2
  Setter = 4
  LargeResultHint = 8

MethodSignature = enum
  VoidVoid = 0
  VoidParam = 1
  RetVoid = 2
  RetParam = 3

TypeName = String
(PrimitiveTypeName | DefinedTypeName)

TypeDescription = {
  "typeName": (TypeName | "Enum" | "BitField")
  "fields": [
    FieldDescription*
  ]? // only for Enum and BitField types
}

FieldDescription = {
  "name": String
  "typeName": TypeName? // default is Bool
  "label": String?
  "description": String?
  "value": (EnumFieldValue | BitFieldFieldIntValue | BitFieldFieldRangeValue)
}

EnumFieldValue = Int

BitFieldFieldIntValue = Int // bit index

BitFieldFieldRangeValue = [
  StartIndex = Int // index of LSB
  EndIndex = Int // index of MSB
]

Common RPC methods

These are well known methods used in various situations. They provide backbone of SHV RPC communication.

• Login sequence
• Discovering SHV paths and methods
• Application API
• Device API
• Broker API
• Property nodes
• File nodes
• Bytes Exchange nodes
• History

Login sequence

The login sequence needs to be performed every time a new client is connected to the broker. It ensures that client gets correct access rights assigned and that devices get to be mounted in the correct location in the broker's nodes tree. Sometimes the authentication can be optional (such as when connecting over local socket or if client certificate is used) but login sequence is required anyway to introduce the client to the broker.

The first method to be sent by the client after connection to the broker is established needs to be hello request. It is called with null SHV path (which means it can be left out in the message's meta) and no parameters.

=> <id:1, method:"hello">i{}

Broker should respond with message containing nonce that can be used to perform login. The nonce needs to be an ASCII string with length from 10 to 32 characters. The nonce must be the same in case, of hello message retransmit during the login phase. Some clients might send more hello message to discover, that shvbroker is started and ready, especially on serial port.

<= <id:1>i{2:{"nonce":"vOLJaIZOVevrDdDq"}}

The next message sent by the client needs to be login request. This message is Map and needs to contain "login" if login is required and can contain "options".

There are two types of the logins you can use. It can be either plain login or sha1. The difference is only the way you send the password in the message. The plain login just sends password as it is in String. The sha1 login hashes the provided user's password, adds the result as suffix to the nonce from hello and hashes the result again. SHA1 hash is represented in HEX format. The complete password deduction is: SHA1(nonce + SHA1(password)). The SHA1 login is highly encouraged and plain login is highly discouraged, even the low end CPUs should be able to calculate SHA1 hash and thus perform the SHA1 login. The "login" map needs to contain these fields:

• "user" with username as String
• "password" with either plain text password or SHA1 password (as described in the paragraph above) as String
• "type" that identifies password format, and thus it can have only one of these values:
  • "PLAIN" for plain text password (discouraged use)
  • "SHA1" for SHA1 login

The "options" needs to be a map of options for the broker. Broker will ignore any unknown options. A different broker implementations can support a different set of options, but minimal needed supported options are these:

• "device" with map containing device specific options. At least these options need to be supported:
  • "deviceId" with String value that identifies the device. This should be used by the broker to automatically assign mount point based on its internal rules.
  • "mountPoint" with String value that specifies path where the device's tree should be mounted in the broker. This should overwrite any mount point assigned by broker based on "deviceId", but it can be disregarded if user doesn't have access rights for the SHV path specified.
• "idleWatchDogTimeOut" specifies number of seconds without message receive before broker will consider the connection to be dead and disconnects it. The default timeout by the broker should be 180 seconds. By increasing this timeout you can reduce the periodic dummy messages sent, but it is suggested to keep it in reasonable range because open but dead connection can consume unnecessary resources on the broker.

=> <id:2, method:"login">i{1:{"login":{"password":"3d613ce0c3b59a36811e4acbad533ee771afa9f3","user":"iot","type":"SHA1"}}, "options":{"device":{"deviceId":"bfsview_test", "mountPoint":"test/bfsview"}, "idleWatchDogTimeOut":180}}}

The broker will respond with Null (some older broker implementation can respond with some value for backward compatibility reasons). From now on you can send any requests and receive any messages you have rights on as logged user from the broker.

<= <id:2>i{}

In case of a login error you can attempt the login again without need to disconnect or sending hello again. Be aware that broker should impose delay of 60 seconds on subsequent login attempts for security reasons.

Note that hello and login methods are not to be reported by dir and no other method can be called before login sequence is completed.

Discovering SHV paths and methods

SHV paths address different nodes in the SHV tree and every node has methods associated with it.

*:dir

This method needs to be implemented for every node (that is every valid SHV path). It provides a way to list all available methods and signals of the node.

Be aware that for existing nodes the result of this method should be constant. There should be no methods added nor removed during the node existence.

This method can be called with or without parameter. The valid parameters are:

• Null or false and in such case all methods are listed with their info.
• true is same as false or Null with exception that result can also contain extra map.
• String with possible method name for which false is provided if there is no such method and some other value (except of null) when there is.

The method info in IMap must contain these fields:

• 1 (name): string containing method's name. This must be unique name for single node. It is not allowed to provide multiple descriptions with same name for the same node.
• 2 (flags): is integer value flag assembled from the following values:
  • 1 (1 << 0) no longer used and reserved for compatibility reasons. In the past it signaled that name is not callable. New implementations should ignore method descriptions with this bit set.
  • 2 (IsGetter, 1 << 1) specifies that method is a getter. This method must be callable without side effects without any parameter.
  • 4 (IsSetter, 1 << 2) specifies that method is a setter. This method must be callable that accepts parameter and provides no value. They are commonly paired with getter.
  • 8 (LargeResult, 1 << 3) specifies that provided value in response is going to be large. This exists to signal that by calling this method you can block the connection for considerable amount of time.
  • 16 (NotIdempotent, 1 << 4) specifies that method is not idempotent. Such method can't be simply called but instead needs to be called first without parameter to get unique submit ID that needs to be used in argument for real call. This unique submit ID prevents from same request being handled multiple times because first execution will invalidate the submit ID and thus prevents re-execution.
  • 32 (UserIDRequired, 1 << 5) specifies that method requires UserID to be called. Calling this method without it should result in UserIDRequired error.
• 3 (paramType): defines parameter type for the requests. Type is a String identifier. It can be missing or have value Null instead of String if method takes no parameter (or only Null).
• 4 (resultType): defines result type for the responses. The is a String identifier. It can be missing or have value Null instead of String if method provides no value (or only Null).
• 5 (accessLevel): specifies minimal access level needed to call this method as Int. The allowed values can be found in table in RpcMessage article.
• 6 (signals): is used for signals associated with this method. Signals have their names and type identifier for value they carry. They are specified as a Map from signal's name to String type identifier. It is allowed to use Null instead of String for type and in such case type is the method's result type (of course field 4 must be defined).
• 63 (extra): extra Map that can contain anything you want. It is provided only if true is passed as argument and can be used to provide additional implementation specific info for this method.

Examples of dir requests:

=> <id:42, method:"dir", path:"">i{}
<= <id:42>i{2:[i{1:"dir", 2:0, 3:"idir", 4:"odir", 5:1},i{1:"ls", 2:0, 3:"ils", 4:"ols", 5:1, 6:{"lsmod":"olsmod"}}]}

=> <id:43, method:"dir", path:"test/path">i{1:false}
<= <id:43>i{2:[i{1:"dir", 2:0, 3:"idir", 4:"odir", 5:1},i{1:"ls", 2:0, 3:"ils", 4:"ols", 5:1, 6:{"lsmod":"olsmod"}},{1:"get", 2:2, 3:"iget", 4:"String", 5:8, 6:{"chng":null}}]}

=> <id:44, method:"dir", path:"test/path">i{1:"get"}
<= <id:44>i{2:true}

=> <id:44, method:"dir", path:"test/path">i{1:"nonexistent"}
<= <id:44>i{2:false}

The previous version (before SHV RPC 3.0) supported both Null and String but provided Map instead of IMap. The String argument also always provided list (with one or no maps). Even older implementations provided list of lists [[name, signature, flags, description],...]. Clients that do want to fully support all existing devices should support both of the old representations as well as the latest one.

The compatibility mapping between IMap keys and historical Map is:

*:ls

This method needs to be implemented for every valid SHV path. It provides a way to list all children nodes of the node.

This method can be called with or without parameter. The valid parameters are:

• Null and in such case all children node names are provided in the list of strings.
• String with possible child name for which false is provided if node has no such child and some other value (except of null) if there is.

Examples of ls requests:

=> <id:42, method:"ls", path:"">i{}
<= <id:42>i{2:["foo", "fee", "faa"]}

=> <id:45, method:"ls", path:"">i{1:"fee"}
<= <id:45>i{2:true}

=> <id:45, method:"ls", path:"">i{1:"nonexistent"}
<= <id:45>i{2:false}

The previous versions (before SHV RPC 0.1) supported Null argument but not String, instead it supported list with String and Int. This variant is discouraged to be used by new clients. The Null variant is fully backward compatible.

Signal lsmod

The signal that has to be sent if there is change in the result of the *:ls method. This includes case when new nodes are added as well as when nodes are removed. The SHV Path must be to the lowest still valid node (valid after change).

The value sent with notification needs to be Map where keys are name of the nodes that were added or removed and value is Bool signaling the node existence. true thus signals added node and false removed one. It is allowed and desirable to specify multiple node changes in one signal but you should not delay notification sending just to combine it with some future changes, it must be sent as soon as possible.

<= <signal:"lsmod", path:"test", source:"ls">i{1:{"device":true}}

Application API

These are methods that are required for every device to be present on its SHV path ".app". Clients do not have to implement these, but their implementation is highly suggested if they are supposed to be connected to the broker for more than just a few requests.

.app:shvVersionMajor

This method provides information about implemented SHV standard. Major version number signal major changes in the standard and thus you are most likely interested just in this number.

=> <id:42, method:"shvVersionMajor", path:".app">i{}
<= <id:42>i{2:0}

.app:shvVersionMinor

This method provides information about implemented SHV standard. Minor version number signals new features added and thus if you wish to check for support of these additions you can use this number.

=> <id:42, method:"shvVersionMinor", path:".app">i{}
<= <id:42>i{2:1}

.app:name

This method must provide the name of the application, or at least the SHV implementation used in the application.

=> <id:42, method:"name", path:".app">i{}
<= <id:42>i{2:"SomeApp"}

.app:version

This method must provide the application version, or at least the SHV implementation used in the application (must be consistent with information in .app:appName).

=> <id:42, method:"version", path:".app">i{}
<= <id:42>i{2:"1.4.2-s5vehx"}

.app:ping

This method should reliably do nothing and should always be successful. It is used to check the connection (if message can be passed to and from client) as well as to keep connection in case of SHV Broker.

=> <id:42, method:"ping", path:".app">i{}
<= <id:42>i{}

.app:date

This is an optional method that provides access to the date and time this application is using (that includes time zone). Applications running on systems without RTC are not expected to implement this method. You must implement this any time methods this application provides to SHV works with date and time.

You should use this to detect time shift between your time and time of the device you are talking to. Date and time sent by device will be relative to this one and thus even if it has wrong time set you have change to calculate the correct one. The same applies the other way around, but in general such methods should be avoided.

Note that there is unspecified overhead of SHV RPC network in up to seconds for transferring messages and thus precision of comparison with local time must consider this.

=> <id:42, method:"date", path:".app">i{}
<= <id:42>i{2:d"2017-05-03T15:52:31.123"}

Device API

Device is a special application that represents a single physical device. It is benefical to see the difference between random application and application that runs in dedicated device and controls such device. This allows generic identification of such devices in the SHV tree.

The call to :ls(".app") can be used to identify application as being a device.

.device:name

This method must provide the device name. This is a specific generic name of the device.

=> <id:42, method:"name", path:".device">i{}
<= <id:42>i{2:"OurDevice"}

.device:version

This method must provide version (revision) of the device.

=> <id:42, method:"name", path:".device">i{}
<= <id:42>i{2:"g2"}

.device:serialNumber

This method can provide serial number of the device if that is something the device has. It is allowed to provide Null in case there is no serial number assigned to this device.

=> <id:42, method:"serialNumber", path:".device">i{}
<= <id:42>i{2:"12590"}

.device:uptime

This provide current device's uptime in seconds. It is allowed to provide Null in case device doesn't track its uptime.

=> <id:42, method:"uptime", path:".device">i{}
<= <id:42>i{2:3842}

.device:reset

Initiate the device's reset. This might not be implemented and in such case NotImplemented error should be provided.

=> <id:42, method:"reset", path:".device">i{}
<= <id:42>i{}

.device/alerts:get

Get the current device's alerts.

The .device/alerts node is property node. Its implementation is optional and thus if device doesn't raise any alerts then this node should not be present.

• 0 (date): date and time of the alert creation.
• 1 (level): int with notice level. This is value between 0 and 63 that can be used to sort alerts. The classification is to the three named levels:
  • Notice: from 0 to 20. These are alerts signal that do not affect nor primary nor secondary device's functionality but require some attention. These can also be potential issues that would affect the device's secondary functionality.
  • Warning: from 21 to 42. These are levels to be used for issues not affecting the device's primary functionality or notices for possible issues that would affect the primary functionality.
  • Error: from 43 to 63. These should be used for issues affecting the primary functionality of the device.
• 2 (id): string with notice identifier. This identifier should be chosen to be unique for not for the single device but also between devices. It should be short but at the same time precise and unique. The benefit is human readability. This ID then should be used in device's manual.
• 3 (info): any SHV value that provides additional info. This is optional field that can be used to pass some additional info related to the alert. The value interpretation should be documented but is outside of the SHV scope.

=> <id:42, method:"get", path:".device/alerts">i{}
<= <id:42>i{2:[i{0:d"2017-05-03T15:52:31.123", 1:42, 2:"EX_CMP_TEST"}]}

.device/alerts:get:chng

The alerts change must be signaled with chng signal.

<= <signal:"chng", path:".device/alerts", source:"get">i{1:[]}

Broker API

The broker needs to implement application API, as well as some additional methods and broker's side of the login sequence (unless it connects to other broker and in such case it performs client side).

The broker can be pretty much ignored when you are calling methods (sending request and receiving response messages). On the other hand, signal message retrieval needs to be requested from broker, because broker won't pass them automatically and thus knowledge about broker is required for clients wanting to receive signals.

The call to :ls(".broker") can be used to identify application as being a broker.

Signals filtering

Devices send regularly signal messages but by propagating these messages to all clients is not always desirable. For that reason signals are filtered. The default is to not send signal messages and clients need to explicitly request them with subscriptions (note that it is allowed that subscriptions would be seeded by the broker based on the login).

The speciality of these methods is that they are client specific. In general RPC should respond to all clients the same way, if it has access rights high enough, but these methods manage client's specific table of subscriptions, and thus it must work with only client specific info.

Note that all the methods under .broker/currentClient have access set to Browse. Client must always have access to these methods, so access is irrelevant in this case. On the other hand broker must deny access to the .broker/currentClient:subscribe and .broker/currentClient:unsubscribe of their mount points to their clients. That is because otherwise clients could influence subscriptions that need to be maintained by the broker itself.

Brokers can be chained by mounting broker inside another one. Such mounted broker of course won't automatically be sending its signal messages and thus they won't be propagated. Simple solution is to subscribe for all signal messages, but that is not efficient. The better solution is to derive an appropriate subscription for mounted broker and subscribe for only this limited set of signal messages. If you are implementing broker then be aware that multiple subscribes can be derived to the same one and thus unsubscribe must be performed with care to not remove still needed ones. The solution for that is to not use unsubscribe and rely on TTL instead.

.broker/currentClient:subscribe

Adds rule that allows receiving of signals (notifications) from shv path. The subscription applies to all methods of given name in given path and sub-paths. The default path is an empty and thus root of the broker, this subscribes on given method in all accessible nodes of the broker.

The parameter is resource identifier for signals. There is also an option to subscribe only for limited time by using list parameter where first argument is RPC RI and the second is TTL in seconds. The subscribe with specified TTL is automatically dropped when given number of seconds elapses. This time can be extended by calling subscribe with TTL again or it can be even removed when called without TTL.

It provides true if subscription was added and false if there was already such subscription.

=> <id:42, method:"subscribe", path:".broker/currentClient">i{1:"**:*:chng"}
<= <id:42>i{2:true}

=> <id:42, method:"subscribe", path:".broker/currentClient">i{1:["test/device/**:get:chng", 120]}
<= <id:42>i{2:true}

.broker/currentClient:unsubscribe

Reverts an operation of .broker/currentClient:subscribe.

The parameter must be resource identifier for signal used for subscription creation.

It provides true in case subscription was removed and false if it couldn't have been found.

=> <id:42, method:"unsubscribe", path:".broker/currentClient">i{1:"**:*:chng"}
<= <id:42>i{2:true}

=> <id:42, method:"unsubscribe", path:".broker/currentClient">i{1:"invalid/**:*:chng"}
<= <id:42>i{2:false}

.broker/currentClient:subscriptions

This method allows you to list all existing subscriptions for the current client.

Map of strings to int key value pairs is provided where keys are resource identifiers for signals and values are TTL remaining for the existing subscriptions. Null TTL means, that the subscription lasts forever.

=> <id:42, method:"subscriptions", path:".broker/currentClient">i{}
<= <id:42>i{2:["**:*:chng", "test/device/**:get:chng"]}

Clients

The primary use for SHV Broker is the communication between multiple clients. The information about connected clients and its parameters is beneficial not only for the client's them self but primarily to the administrators of the SHV network.

.broker:clientInfo

Information the broker has on the client.

The parameter is client's ID (Int). The provided value is Map with info about the client. The Null is provided in case there is no client with this ID.

The Map containing at least these fields:

• "clientId" with Int containing ID assigned to this client.
• "userName" with String user name used during the login sequence. This can be Null because broker can have clients it established itself and thus won't perform any login.
• "mountPoint" with String SHV path where device is mounted. This can be Null in case this is not a device.
• "subscriptions" is a list of subscriptions of this client.

Additional fields are allowed to support more complex brokers but are not required nor standardized at the moment.

=> <id:42, method:"clientInfo", path:".broker">i{1:68}
<= <id:42>i{2:{"clientId:68, "userName":"smith", "mountPoint": "iot/device", "subscriptions":["**:*:chng"]}}

=> <id:42, method:"clientInfo", path:".broker">i{1:126}
<= <id:42>i{2:null}

.broker:mountedClientInfo

Information the broker has on the client that is mounted on the given SHV path.

The parameter is SHV path (String). The provided value is same as .broker:clientInfo with info about the client that is mounted on the given path (or the parent of it). The Null is provided in case there is no mount point to which given path would belong to.

=> <id:42, method:"mountedClientInfo", path:".broker">i{1:"iot/device/node"}
<= <id:42>i{2:{"clientId:68, "userName":"smith", "mountPoint": "iot/device", "subscriptions":["**:*:chng"]}}

=> <id:42, method:"mountedClientInfo", path:".broker">i{1:"invalid"}
<= <id:42>i{2:null}

.broker/currentClient:info

Access to the information broker has for the current client. The result is client specific.

The provided value is same as if .broker:clientInfo would be called with client ID for the current client. The difference is that this method must be accessible to the current client while .broker:clientInfo is accessible only to the privileged users.

=> <id:42, method:"info", path:".broker/currentClient">i{}
<= <id:42>i{2:{"clientId:68, "userName":"smith", "mountPoint": "iot/device", "subscriptions":[{1:"chng"}]}}

.broker:clients

This method allows you get list of all clients connected to the broker. This is an administration task.

This is a mandatory way of listing clients. There also can be an optional, more convenient way, that brokers can implement to allow easier use by administrators (commonly in .broker/clientInfo), but any automatic tools should use this call instead. It is also more efficient than using .broker/client:ls.

The List of Ints is provided where integers are client IDs of all currently connected clients.

=> <id:42, method:"clients", path:".broker">i{}
<= <id:42>i{2:[68, 83]}

.broker:mounts

This method allows you get list of all mount paths of devices connected to the broker. This is an administration task.

The List of Stringss is provided where strings are mount points of all currently mounted clients.

=> <id:42, method:"mounts", path:".broker">i{}
<= <id:42>i{2:["iot/device", "shv/device/temp1"]}

.broker:disconnectClient

Forces some specific client to be immediately disconnected from the SHV broker. You need to provide client's ID as an argument. If client is established by the broker (it is connection to serial console or to some other broker) it is up to the broker what should happen, but it is suggested that this would be handled as reconnection request.

=> <id:42, method:"disconnectClient", path:".broker">i{1:68}
<= <id:42>i{}

.broker/client

It is desirable to be able to access clients directly without mounting them on a specific path. This helps with their identification by administrators. This is done by automatically mounting them in .broker/client/<clientId>. This mount won't be reported by .broker:mountPoints method, nor it should be the mount point reported in .broker:cientInfo. And due to not being mount point it also can't support signal messages.

The access to this path should be allowed only to the broker administrators. The rule of thumb is that if user can access .broker:disconnectClient, it should be also able to access .broker/client.

Property nodes

Property node is a convention where we associate value storage with some SHV path. The value can be received, optionally modified and its change can be signaled.

The type of the property depends on the presence of the methods. The following table

*:get

This method is used for getting the current value associated with SHV path. Every property node needs to have get method and every node with get method can be considered as property node.

Integer argument is maximal age in milliseconds. Value can be of any age if Null parameter is used (or omitted). The age is relevant when latest value must be received over some other medium (such as from Modbus) and thus every request would trigger a new exchange. Instead if latest exchange was withing specified age the value can be served from cache.

=> <id:42, mtehod:"get", path:"test/property">i{}
<= <id:42>i{2:"hello"}

=> <id:42, method:"get", path:"test/property">i{1:60000}
<= <id:42>i{2:"Cached"}

*:get:*chng

Value change can be optionally signaled with signal. It is used when you have desire to get info about value change without polling. Note that signal might not be emitted just on value change (that means old value can be same as the new one) and it might not be emitted on some value changes (for example if change was under some notification deadband level). To get latest value you should always use *:get instead of waiting for *chng signal but if you receive *chng then you can save yourself a *:get call.

The signal name can be either just chng or any name with that as suffix (such as fchng).

The *chng needs to provide the same value as *:get would, which is value associated with the SHV path.

<= <signal:"chng", path:"test/property", source:"get">i{1:"Hello World"}

*:set

This method is used for changing the value associated with SHV path. By providing this method alongside with *:get you are making the read-write property. Property is considered read-only if you omit this method.

The *:get should be providing value specified in *:set parameter as soon as completion of *:set method is confirmed (response is sent). In other words: the set value can be received with *:get right after *:set completes. This rules out situation where *:get reports real (measured) value while *:set specifies a reference. You should always split this to two property nodes where reference is read-write property and real value is read-only one.

=> <id:42, method:"set", path:"test/property">i{1:"Hello World"}
<= <id:42>i{}

File nodes

File node provides read and optionally also write access for the binary files over SHV RPC.

The file directories are just regular nodes because the only required functionality (that is listing files) is already provided by nodes discovery.

File nodes should not have any child nodes (*:ls should always return []).

The access levels for these methods are only suggestions (compared to some other method definition in this standard). That is because sometimes you want to allow a different levels of access to different users to same file (such as append to regular user, while write and truncate to super users).

The combination of provided method for file node will define the expected functionality of the file. For the convinience of reader the following table is provided with various, but not all, types of file node modes. The methods *:stat, and *:size are always provided.

*:stat

This method provides information about this file. It is required for file nodes.

The result is IMap with these fields:

=> <id:42, method:"stat", path:"test/file">i{}
<= <id:42>i{2:i{0:1,1:3674,2:1024}}

*:size

This provides faster access to only file size. Although it is also part of the *:stat the size of the file is commonly used to quickly identify added data to the file and thus it is provided separately as well. This method must be implemented together with *:stat.

=> <id:42, method:"size", path:"test/file">i{}
<= <id:42>i{2:3674}

*:crc

The validation of the data is common operation that can be done either to verify that all went all right or to detect that write might not be required because data is already present. To prevent from doing this validation by pulling the whole content of the file this CRC32 calculating method is provided. The client providing the file node will calculate the CRC32 instead and send only that.

This method must be implemented alongside with *:read but it can also be implemented on its own if you allow validation but not read for some reason (make sure that in such case client can't read the file anyway with short crc calculation ranges).

CRC32 algorithm to be used is the one used in IEEE 802.3 and known as plain CRC-32.

You can either pass Null and in such case checksum of the whole file will be provided, or you can pass list with offset and size (that can be Null for the all data up to the end of the file) in bytes that identifies range CRC32 should be calculate for.

No error must be raised if range specified by parameter is outside of the file. Only bytes present in the range are used to calculate CRC32 (this includes case when there are no bytes captured by specified range). This allows easier work with files that are growing.

=> <id:42, method:"crc", path:"test/file">i{}
<= <id:42>i{2:409417892}

=> <id:42, method:"crc", path:"test/file">i{1:[1024, 2048]}
<= <id:42>i{2:25819716}

*:sha1

The basic CRC32 algorithm, that is required alongside with *:read, is serviceable but it has higher probability of result collision. If you want to use it to not only detect modification of the file but instead identification of it, then SHA1 is the better choice. The implementation of this method is optional (but when it exists the *:crc must exist as well).

You can either pass Null and in such case sha1 of the whole file will be provided, or you can pass list with offset and size (that can be Null for the all data up to the end of the file) in bytes that identifies range the hash should be calculate for.

No error must be raised if range specified by parameter is outside of the file. Only bytes present in the range are used to calculate SHA1 (this includes case when there are no bytes captured by specified range). This allows easier work with files that are growing.

The result must always be 20 bytes long Bytes.

=> <id:42, method:"sha1", path:"test/file">i{}
<= <id:42>i{2:b"4\972t\cc\efj\b4\df\aa\f8e\99y/\a9\c3\feF\89"}

*:read

Method for reading data from file. This method should be implemented only if you allow reading of the file.

The parameter must be a tuple containing offset and size in bytes that identifies the range to be read. The implementation may return less data than size, but it will never return 0 bytes if any data exists at the specified offset. The range can be outside of the file boundaries and in such case zero length bytes value is returned.

=> <id:42, method:"read", path:"test/file">i{1:[0, 1024]}
<= <id:42>i{2:b"Hello World!"}

=> <id:42, method:"read", path:"test/file">i{1:[1024, 1024]}
<= <id:42>i{2:b""}

*:write

Write is optional method that can be provided if modification of the file over SHV RPC is allowed.

The parameter must be list with offset in bytes and bytes to be written on this offset address.

Write outside of the current file boundary is up to the implementation. It can extend file boundary if that is possible, or it can result into an error.

=> <id:42, method:"write", path:"test/file">i{1:[0, b"Hello World!"]}
<= <id:42>i{}

*:truncate

Change the file boundary. This method should be implemented if *:write allows write outside of the file boundary. It should not be present if that is not possible. In other words: presence of this method signals that write outside file's boundary is possible alongside with presence of *:write.

The parameter must be requested size of the file.

It is up to the implementation if some boundaries to file change are imposed, such as only increase is allowed, or maximal size of the file.

=> <id:42, method:"truncate", path:"test/file">i{1:1024}
<= <id:42>i{}

*:append

Append is a optional special way to perform write by always appending to the end of the file. Append can be provided even if *:write is not and other way around. *:truncate also doesn't have to be implemented, append is always outside file boundary.

The parameter is sequence of bytes to be appended to the file.

=> <id:42, method:"truncate", path:"test/file">i{1:b"Some text..."}
<= <id:42>i{}

Bytes Exchange node

SHV RPC is by design request-response communication protocol and most notably messages can be dropped without a retransmit nor notice and thus lost. At the same time there are use cases when it is desirable to emulate byte streams such as remote terminal. This requires improved reliability in the form of ensured delivery and both sides transmission control.

In the RPC communication there are always two sides, the asking peer (caller) and the answering peer (answerer). To get the bidirectional communication we exchange bytes between caller and answerer with method call. This gives full control over the communication flow to the caller that must initiate every exchange. The answerer holds the connection status and notifies caller on status change with SVH signal, but it doesn't send any to be exchanged bytes on its own.

The communication is initialized by requesting the new exchange point by caller from answerer. This exchange point is then used to exchange bytes between them. It is deleted either by either sides closing it by an appropriate action or if caller doesn't call exchange for idleTimeOut (in default 30 seconds).

*:newExchange

This creates new connection for this bytes exchange node. The connections are maintained as sub-nodes of this one.

The access level should be at minimum write but it might be increased depending on the resource this provides. For example you really do not want to provide access to root shell with just write access level.

The parameter is Map with options to be set to the new exchange point. The real options depend on the implementation. The minimum supported list is described in */ASSIGNED:options.

The result of this call is name of the created node. The names of the sub-nodes is up to the implementation and its algorithm for generating them but it should choose short names. The soft limit for the name is 8 characters which gives versatility for assignment for node name generation but still limits length of the name that clients can expect to remember. For security reasons the assignment should minimize the reuse of the node names as much as possible.

=> <id:42, method:"newExchange", path:"test/sh">i{}
<= <signal:"lsmod", path:"test/sh", source:"ls">i{1:{"a3":true}}
<= <id:42>i{2:"a3"}

*/ASSIGNED:exchange

This is the method that is called to exchange bytes.

The access level must be disregarded for this method and instead CallerIds must be used to control access. The message must have same CallerIds as the one for *:newExchange that created this node. Nobody else must be allowed to call this method (this is for consistency of exchanged bytes).

The parameter is IMap with following items:

The result is IMap with following items:

Note that ReadyToSend is only informative. The implementation of answerer might not know number of available bytes and this most likely it will use only 0 for "no bytes available" and 1 for "some bytes available". The parameter's ReadyToReceive thus should always contain maximum number of bytes caller is willing to receive and not copy of ReadyToSend from previous result or signal.

=> <id:42, method:"exchange", path:"test/sh/a3">i{0:7, 1:32, 3:b"Foo"}
<= <id:42>i{2:i{1:24, 3:b"Hello Foo\n"}}

*/ASSIGNED:exchange:ready

The caller is expected to call */ASSIGNED:exchange as soon as it can again to send and/or receive more data, but in some cases it might want to just wait for data to become available or space available for receive. In such a situation caller must poll */ASSIGNED:exchange periodically to detect that, but that introduces a time delay. The solution is that answerer sends this signal when either ReadyToReceive or ReadyToSend goes from zero to non-zero value. The caller still must poll the exchange method to not get stuck in case signal gets lost but in most cases this signal will speed up the return to the data exchanging again.

The value is IMap with items ReadyToReceive and ReadyToSend from */ASSIGNED:exchange result.

<= <signal:"ready", path:"test/sh/a3", source:"exchange">i{1:i{1:1}}

The more realistic exchange to the one used as an example for */ASSIGNED:exchange would contain this signal in the following way:

=> <id:42, method:"exchange", path:"test/sh/a3">i{0:7, 1:32, 3:b"Foo"}
<= <id:42>i{2:i{1:24}}
<= <signal:"ready", path:"test/sh/a3", source:"exchange">i{1:i{1:32, 2:1}}
=> <id:43, method:"exchange", path:"test/sh/a3">i{0:8, 1:32}
<= <id:43>i{2:i{1:32, 3:b"Hello Foo\n"}}

*/ASSIGNED:options

This provide access to the options associated with this connection.

The access level applies only to the callers that do not match CallerIds. Request messages with CallerIds matching that for *:newExchange that created this node must be allowed.

The result is Map with at least these fields:

• idleTimeOut with unsigned integer with number of seconds. It allows client to specify its own idle time out. If not specified the 30 seconds are used.

=> <id:42, method:"options", path:"test/sh/a3">i{}
<= <id:42>i{2:{"idleTimeOut":30}}

*/ASSIGNED:setOptions

This allows modification of option associated with this connection. Note that not all options might be modifiable and only fields specified in Map are modified (the rest is kept same or derived from new options based on the implementation).

The access level applies only to the callers that do not match CallerIds. Request messages with CallerIds matching that for *:newExchange that created this node must be allowed.

=> <id:42, method:"setOptions", path:"test/sh/a3">i{1:{"idleTimeOut":60}}
<= <id:42>i{}

*/ASSIGNED:close

This method allows caller to terminate the connection.

The access level applies only to the callers that do not match CallerIds. Request messages with CallerIds matching that for *:newExchange that created this node must be allowed.

=> <id:42, method:"close", path:"test/sh/a3">i{}
<= <signal:"lsmod", path:"test/sh", source:"ls">i{1:{"a3":false}}
<= <id:42>i{}

*/ASSIGNED:peer

This provide ClientIds from *:newExchange that created this sub-node.

This method contrary to others of this node uses normal access level control and thus only clients with Super-service access level can call it.

The result is List of Ints.

=> <id:42, method:"peer", path:"test/sh/a3">i{}
<= <id:42>i{2:[30,1,2]]}

Walkthrough of the caller

The first step is to initiate the new connection by calling *:newExchange. This will provide you with sub-node name that you will be using from now on (instead of ASSIGNED).

The next step is to subscribe for the notifications for this node (The expectation is that there is a RPC Broker in between. You can skip this step if that is not the case). You want to use full path to the assigned node, source exchange and all of signal ready.

Based on the usage you might want to inspect and tweak exchange options with */ASSIGNED:options and */ASSIGNED:setOptions respectively.

The initial exchange should not contain any data and is intended only to query for exchange state, but it is allowed to specify that it is ready to receive non-zero amount of bytes. This allows answerer to send data even on initial exchange.

The subsequent exchanges must contain only number of bytes the latest exchange or ready signal specified as being free (the latest information applies).

It is highly suggested to prioritize data retrieval because it is possible that more data can't be received by answerer unless you take some data out and thus you might wait for non-zero ReadyToReceive indefinitely.

You also must maintain exchanges (even dummy ones that do not transfer any data) in reasonable intervals to ensure that connection is not closed due to inactivity (suggestion is the half interval of idleTimeOut).

The connection can be terminated by calling */ASSIGNED:close. The answerer can terminate connection for what ever reason on its own. It will send *:ls:lsmod signal with but in general caller detects termination by receiving MethodNotFound error when calling */ASSIGNED:exchange method. There is no way to report disconnect reason.

Walkthrough of the answerer

The connection gets initiated by caller with *:newExchange method. Answerer must allocate sub-node for it. Based on the intended usage the resource needed for bytes exchanging can be initiated (such as spawning process or opening the serial device) or it can be postponed to the first bytes exchange.

The data received on exchange will be processed (commonly just pushed to buffer) and response will be created with data from processing. The sent data must be kept until next exchange alongside with received Counter value to send same data if Counter of the next exchange is the same. The received data from exchange that has matching Counter with previous one are disregarded and not processed (they are repeat of what was already received).

The number of bytes ready that can be received is provided to the caller as well as number of bytes that could be still provided. The signal ready must be emitted if one of these parameters changes from zero to non-zero because caller might not attempt next exchange for some time if they are set to zero.

The idle time between exchanges is measured and if there was no exchange for longer than idleTimeOut seconds then node is discarded and thus subsequent calls for data exchange will result into an error MethodNotFound. The same happens if */ASSIGNED:close method is called.

Design remarks

These are few points you should be aware when you are using Bytes Exchange nodes:

• The access to the node is controlled by path of message in SHV network. This path is in general constant during the connection, but there are cases when it can change without client being notified about it (when intermediate RPC Broker resets) and this will result into two issues:

  • Existing connections will be inaccessible (resulting into MethodNotFound errors)
  • Existing connection could be suddenly accessible by some other client. The exploit of this would require pretty significant setup but still is possible.

• The reason for every connection being its own sub-node instead of method parameter is to allow filtering signals for a different connections and callers.

• The usage for ready signal is to signal only readiness when exchange wasn't previously ready. It should not intentionally be emitted if previous exchange specified no ability to receive or no ability to send bytes. This is to only fasten caller's detection of readiness when it is reasonable to just be quiet and keep answerer to work.

• The possibility of lost messages is covered by counter. This provides a way to easily get data even when messages are being randomly lost by attempting call again and again with same counter value. In case of caller these multiple call attempts are intuitive (they must contain same content) but answerer must ensure the following to keep data consistent:

  • Sent data are kept until next exchange to allow their re-transmit if counter isn't changed.
  • Received data are used only when counter changes, not when it stays the same.

History

History is functionality that allows logging of signals and access to the history of changes of these signals. Client with history can provide its history to upper implementations and that way histories can be propagated through the SHV RPC network. This propagation allows central storage of the long term history while peers lower in the network can aggregate only shorter ranges. In other words: this API provides the following functionalities while clients doesn't have to implement all of them:

• Aggregated access to the logs on time bases, that is method .history/**:getLog.
• Systematic retrieval of records (for other histories to use) by providing appropriate methods bellow .history/**/.records nodes.
• Systematic retrieval of file logs (for other histories to use) by providing appropriate nodes bellow .history/**/.files nodes.

History can be implemented either directly as part of the application or it can be mounted to SHV RPC Broker. It must always be mounted in .history mount point, that is why all paths for this API are prefixed with .history/. Any other mount point is not valid for RPC History because it should be at the same node as .app is present.

.history/**

History can only record signals and provide them for other histories (through .history/.records/* and .history/.files/*) or it can also provide aggregated access to logs it stores.

If history provides aggregated access to the logs then it must contain a virtual tree of the logs sources. This allows setting limited log access based on the same path as the real SHV RPC Broker node tree. This means that if history records signal with SHV path test/device/foo/status then path .history/test/device/foo/status must be valid and discoverable.

The aggregated access to the logs doesn't have to be provided for all the logs, but all paths recorded (or to be recorded) in single log must be provided.

.history/**:getLog

Queries logs for the recorded signals in given time range.

This method must be available on all nodes in the .history/** tree with exception of .app, .records and .files. In other words this must be provided only if aggregated log access is provided and in such case it is required.

The parameter is Map with the following fields:

• "since" is DateTime since logs should be provided. The record that exactly matches this date and time is not provided. This allows followup requests from last date and time of the last returned record. The default is the time of request retrieval if not provided.
• "until" is DateTime until logs should be provided. Device might not reach this date if there is too much records in the time range. If you want to make sure that you received all records then you must follow with another request where "since" is replaced with date and time of the last provided log. Note that this date and time can precede "since" and in such case logs returned are sorted from newest to the oldest (normally they are sorted from oldest to the newest). As a special exception if "since" is equal to "until" then all logs until that time are considered and returned from newest one (just like if you would set "until" to something very small). The default is the time of request retrieval if not provided.
• "count" is optional Int as a limitation for the number of records to be at most returned. The device on its own can limit number of returned records but this can lower that even further (thus minimum is used). This should be used if you for example need to know only latest few records (you would use default for both "since" and "until" and set number of recods to "count". The device alone decides limit on number of provided records if this field is not specified. In a special case when there are multiple matching signals recorded with same date and time, then all of them must be provided even when that goes over count limit. Snapshot is not part of this limit and thus you can ask for snapshot only with count set to zero. The default if not specified (or null) is unlimited number of records unless "snapshot" is set to true and in such case it is 0.
• "snapshot" controls if virtual records should be inserted at the start (at "since" time) that copy state of the signals. This provides fixed point to start when you for example plotting data. These records are virtual and are not actually captured signals. This makes sense only for "since" being before "until" and no snapshot can be provided if that is not fulfilled.
• "ri" signal matching RPC RI that is used to filter provided records by the signal source. You should always preferably use as long path for calling getLog as possible instead of giving it to this field because access level is deduced by the request message path and with too short path you might be assigned not hight enough access rights to receive records.

The provided value is list of IMaps with following fields:

• 1(timestamp): DateTime of the record. This field is required. Note that if you requested "snapshot" then records with exactly time of "since" will be provided and will be the snapshot records.
• 2(ref): provides a way to reference the previous record to use it as the default for path, signal and source (instead of the documented defaults). It is Int where 0 is record right before this one in the list. The offset must always be to the most closest record that specifies desired default. This simplifies parsing because there is no need to remember every single received record but only the latest unique ones. It is up to the implementation if this is used or not. Simple implementations can choose to generate bigger messages and not use this field at all.
• 3(path): String with SHV path to the node relative to the path getLog was called on. The default if not specified is "".
• 4(signal): String with signal name. The default if not specified is "chng".
• 5(source): String with signal's associated method name. The default if not specified is "get".
• 6(value): with signal's value (parameter). The default if not specified is null.
• 7(userId): String with UserId carried by signal message. The default if not present is null and thus there was no user's ID in the message.
• 8(repeat): Bool with Repeat carried by signal message. The default if not present is False.

The provided records should be sorted according to the DateTime field 1 either in ascending order if "since" is before "until" or descending order if "util" is before "since".

The method itself has only Browse access level but it must filter provided logs based on their access level and thus user with low access level might not see all that is provided. Note that it is not possible to decrease access level of the user for some part of the SHV tree because he could always ask the upper node where his access level is high enough and logs would be provided.

.history/**/.records/*

These nodes provide systematic access to the records. Every record has unique ID and mustn't change once it is recorded. This ID must be increasing for new records but it doesn't have to be sequential (there can be unused IDs).

Other history implementations can take ID of their latest record and fetch everything from it to correctly synchronize.

The implementation backed in not specified but records based access matches well storage such as database or cyclic buffer.

.history/**/.records/*:fetch

This allows you to fetch records from log.

Parameter is tuple of first record ID to be provided and number of records (thus last record returned is parameter[0] + parameter[1] - 1).

The call provides list of records. Every record is IMap with following fields:

• 0(type): Int signaling record type:
  • 1(normal) for normal records.
  • 2(keep) for keep records. These are normal records repeated with newer date and time if there is no update of them in past number of records. They can also be used to seed log with values that are valid at first boot time (at log creation time).
  • 3(timeJump) time jump record. This is information that all previous recorded times should actually be considered to be with time modification. The time offset is specified in field 60. Field 1 must be also provided but others are not contrary to normal and keep records. This is recorded when time synchronization causes system clock to jump by more than a second.
  • 4(timeAbig) time ambiguity record. This is information that date and time of the new logs has no relevance compared to the previous ones. Any subsequent records of type 3 should not be applied to them. This is recorded when time jump length can't be determined (backward skip of time commonly after boot) and thus time desynchronization is detected. The only field alongside this one must be 1.
• 1(timestamp): DateTime of system when record was created. This depends on record type. For normal records this is time of signal retrieval.
• 2(path): String with SHV path to the node relative to the .history's parent. The default if not specified is "".
• 3(signal): String with signal name. The default if not specified is "chng".
• 4(source): String with signal's associated method name. The default if not specified is "get".
• 5(value): with signal's value (parameter). The default if not specified is null. specified is null.
• 6(accessLevel): Int with signal's access level. The default if not specified is Read.
• 7(userId): String with UserId carried by signal message. The default if not present is null and thus there was no user's ID in the message.
• 8(repeat): Bool with Repeat carried by signal message. The default if not present is false.
• 60(timeJump): Int with number of seconds of time skip. This is used with key 0 being 3.

Fetch that is outside of the valid record ID range must not provide error.

.history/**/.records/*:span

This allows fetch of boundaries for the record IDs and also the keep record range.

This method provides three integers in a list. The first Int is the smallest valid record ID, the second Int is the biggest valid record ID plus one (to allow case when there are no records to be signaled with same value as the first record) and the third Int is the keep record span.

The keep record span is range of records where all combinations of SHV path, signal name and signal's associated method name in the log are present. That is achieved by creating keep records that are copy of older ones when no signal for them is received for some time. It allows of fetching the full log state without going through the whole history.

.history/**/.files/*

These nodes provide file based logs. The systematic log access is ensured by only appending to the log files and never modifying them. Logs propagation is then performed by copying appended data from existing files and new files.

Files are exposed as read only file nodes. The name of the file must be date and time of the first record in the file in ISO-8601 format without timezone and with seconds precision with ".log3" extension. The new files must be created with date and time after the last log even if system clock is right now set before that time. If system time is before the latest log file name then latest log file name must be used with one second increased (to get unique file name). The date and time recorded in the log file is still the system time, the only modification here is the file name.

The file nodes must implement these methods: .history/**/.files/*/*.log3:stat, .history/**/.files/*/*.log3:size, .history/**/.files/*/*.log3:crc, .history/**/.files/*/*.log3:read and optionally .history/**/.files/*/*.log3:sha1.

The content of the file logs is line separated CPON where initial line is expected to have Map while rest should be Lists.

The first line in the log file is Map with these fields:

• "logVersion" that right now should be set to 3.0 and thus must be Decimal.
• "timeJump"is the optional true or Int with the offset in seconds that should be applied to all log files before this one. This thus not specify offset of times in this file but times recorded in files so far. The true is used for ambiguity time jumps. This is used when time desynchronization is detected.

Notice that the first line is the only way to record time jump and thus when time jump is detected you should always open a new log file.

The rest of the file must contain Lists with following columns:

• time: DateTime of system when record was created or Null for anchor logs at the start of the file. File log must start with anchor logs of all latest recorded values from the previous log. This is to provide full information in a single log file.
• path: String with SHV path to the node relative to the .history's parent. The default if not specified is "".
• signal: String with signal name. The default if not specified is "chng".
• source: String with signal's associated method name. The default if not specified is "get".
• value: signal's value (parameter). The default if not specified is null.
• accessLevel: Int with signal's access level. The default if not specified is Read.
• userId: String with UserId carried by signal message. The default if not present is null and thus there was no user's ID in the message.
• repeat: Bool with Repeat carried by signal message. The default if not present is false.

.history/**/.records/*:sync and .history/**/.files/*:sync

Trigger the synchronization manually right now.

This can't be implemented for .history/.records/* and .history/.files/* because those are logs not synchronized but collected.

This method triggers synchronization or does nothing if synchronization is already in the progress.

.history/**/.records/*:lastSync and .history/**/.files/*:lastSync

This provides information when last synchronization was performed.

This can't be implemented for .history/.records/* and .history/.files/* because those are logs not synchronized but collected.

The provided value is either DateTime that must be when last synchronization was performed and Null if synchronization is in progress right now. Implementations should use time when they last called **/.history/**/.records/*:fetch or **/.history/**/.files/*:ls.

Time management in logs

Logs are recorded with device's UTC time. The RPC History then only copies these logs from one instance to the other without modification. This means that date and time is always kept as it was on the device that recorded it. This is ideal when device has the correct real time clock but that might not be true and thus time modifications come into play.

There are two types of time modifications recorded in the logs. We have either known time jump or unknown time desynchronization.

The know time jump is detected on device when some log was already recorded and suddenly system time doesn't correspond to the monotonic time since the last record. The discrepancies up to 1 seconds should be disregarded and covered up by time tweaking (that is because we record skips in seconds) to ensure that time is still growing (no step backs in time compared to the previous log is allowed unless time jump is recorded). This time jump happens commonly if some tool synchronizes or in general updates system time. We expect that this modification is always the correct one (that our time up to now was shifted by skip) and time jump is recorded. getLog implementation thus must shift virtually all recorded times. It can't modify date and time recorded in those records but all previous records since the time jump up to the any time desynchronization must be considered to be shifted by recorded time jump. The multiple time jumps must be added together when you are reaching for older records and have multiple time jumps in between.

The time desynchronization can happen only on first record after boot because otherwise we know the previous time and can thus calculate the time jump. The common detection for time desynchronization is the check of the latest record, if it is in the future then desynchronization occurred and must be recorded. Desynchronization creates break point in the time sequence and thus shifts described in the previous paragraph are not performed after for records before desynchronization. The only exception is in an unlikely event when logs after desynchronization (after all time shifts from jumps are applied) are recorded as happening before last log before desynchronization and in such case time shift is introduced to move logs before desynchronization to be right before logs after desynchronization. This can really happen only if someone sets date and time in the future, then powers down the device, resets RTC and sets the correct date and time after boot.

The optimal implementation of getLog for both records and files is to keep index of modified times with reference to record ID or file with offset to speed up lookup for getLog. The memory constrained devices can implement it in less time optimal way by keeping only references to the time jumps and calculate the correct time for every record when loaded. The logs time sequence is always kept regardless of date and time and thus this time shifting only moves the whole blocks of consistent logs.

RPC transport layer

RPC messages can be transmitted over different layers. Any message transport layer that guarantees the following functionalities can be used for SHV RPC:

• Messages need to be transferred completely without error or transport error needs to be detected.
• Any message can be lost without transport error reporting error.
• Communication needs to be point-to-point, or transport layer needs to emulate that.
• Messages can be of any size

Exchanged messages have data always start with a single byte identifying the format of the subsequent bytes in the message (if any):

• 00 - ResetSession
• 01 - Chainpack encoded RPC Message
• 02 - Cpon encoded RpcMessage (deprecated)
• 03 - Json encoded RpcMessage (deprecated)

ResetSession message is used to reset current session. Basically it works in the same manner as socket reconnection, but it can be used also on layers without connection tracking like serial port. When ResetSession is received, then receiver should clear the peer's state. ResetSession message can be used by client to log-out or by broker to kick-out client. ResetSession message contains only Format part, the message is excluded. It is 01 00 on stream layer or STX 00 ETX or STX 00 ETX D2 02 EF 8D (if CRC is on) on serial layer. This message must be sent before hello on layers without connection tracking (TTY, CAN). It might be optionally sent also on other layers like socket.

There are protocols defined for some common transport layers that do not support message transport layer as required by SHV RPC. This includes the following:

• Stream transport layers
• CAN Bus

Stream transport layers

Exchange of the data in streams is common abstraction that is provided by most of the transport layers (such as TCP/IP). The important operation to support streams in SHV RPC is to split stream to distinct messages. For this purpose there are two different protocols defined; Block and Serial protocols.

Block

The communication on bidirectional stream where delivery is ensured and checked. The transport layer establishes and maintains point-to-point connection on its own.

Message is transmitted in stream as one complete segment where message length is sent before data. The receiving side first reads size and thus knows how much data it should expect to receive. Message length is encoded as Chainpack unsigned integer.

+--------+------+
| length | data |
+--------+------+

The transport error is detected if there is no byte received from other side for more than 5 seconds during the message transfer.

Transport errors are handled by an immediate disconnect. It is expected that connecting side is able to reestablish connection.

The primary transport layer used with this is TCP/IP, but this also applies to Unix sockets or pair of pipes.

Serial

This is communication over data stream with possible on the line errors (such as data corruption).

The message data is encapsulated in start and stop byte and on link layers without error checking it is also verified with CRC32. The dedicated bytes for this are escaped in the message data.

+-----+------+-----------+---------+
| STX | data | ETX / ATX | *CRC32* |
+-----+------+-----------+---------+

• STX start of message 0xA2
• ETX end of the message 0xA3
• ATX abort the message 0xA4
• ESC escape 0xAA
  • STX in data will be coded as ESC 0x02
  • ETX in data will be coded as ESC 0x03
  • ATX in data will be coded as ESC 0x04
  • ESC in data will be coded as ESC 0x0A
• data - escaped message data
• CRC32 - escaped BigEndian CRC32 of data (same as used in IEEE 802.3 and known as plain CRC-32) only on channels that do not provide data corruption prevention on its own and only after ETX (not after ATX).

The transport error is detected if there is no byte received from other side for more that 5 seconds during the message transfer or when STX or ATX is received before EXT or if CRC32 do not match received data (on channels with possible corruption such as RS232 and not TCP/IP).

Transport errors do not have to be handled explicitly because any subsequent message can still be consistent. The invalid message should be just dropped.

The primary transport layer is RS232 with hardware flow control, but usage with other streams, such as TCP/IP or Unix domain named socket, is also possible.

In some cases the restart of the connection might not be available. That is for example when application just doesn't have rights for it or even when such restart would not be propagated to the target, for what ever reason. To solve this the empty message is reserved (that is message STX ETX 0x0). Once client receives a valid empty message then it must drop any state it keeps for it.

CAN-FD transport layer

The communication over CAN (Control Area Network) bus. Compared to other transport layers that are point-to-point this is bus and thus it must not only provide message transfer but also a connection tracking.

CAN has the following abilities:

• Delivery of the CAN frames is not ensured (it can be lost)
• Single CAN frame can be transmitted multiple times
• CAN frames are delivered in order based on CAN ID priority and never out of order
• Collision is resolved by avoiding it based on the CAN ID (lower has higher priority)
• CAN frames correctness is ensured with CRC32
• Flow control is handled by CAN overload frame

CAN supports few different types of the frames:

• Data frames: regular frames used to transfer data
• Remote frames: frame that caries no data and data length (0x0 - 0xf) is just informative. SHV RPC uses these as following signal frames:
  • Message abort frame with data length 0x0
  • Device advertisement frame with data length 0x1
  • Device discover frame with data length 0x2

CAN bus is designed for control applications but SHV RPC is rather configuration and management interface and thus the design here prioritizes fair queueing over message delivery deadline guaranties. This is because we expect that SHV RPC will overcommit the bus (contrary to common control applications).

CAN bus has limited bandwidth and thus it is not desirable in most cases to emit all signals device has (contrary to standard SHV RPC device design). The SHV native way to introduce filtering is to use RPC Broker and thus it is highly suggested that devices on CAN bus should expose RPC Broker to it.

CAN ID

CAN ID can be either 29 bits or 11 bits. SHV RPC uses exclusively only 11 bits ID.

+-------------+-----------+---------+--------------------+
| NotLast [1] | First [1] | QoS [1] | Device address [8] |
+-------------+-----------+---------+--------------------+

• NotLast is 0 when no subsequent CAN frame will follow this one and 1 otherwise. This prioritizes message termination on the bus.
• First is 1 when this is initial CAN frame and 0 otherwise. This penalizes start of the new message and thus prefers SHV RPC message finish.
• QoS should be flipped on every SHV RPC message sent. It ensures that devices with high CAN ID (low priority) get chance to transmit their messages. It is allowed that device that is quiet for considerable amount of time to set it to any state (commonly beneficial for that device).
• Device address this must be unique address of the device transmitting CAN frame or bitwise not of it when QoS is 1. The addresses 0x0 and 0xff are reserved. The bit flipping of the device address based on the QoS ensures that high priority CAN IDs in QoS 1 are low priority ones in the QoS 0 and vice versa, thus devices should get somewhat equal chance to transmit their messages.

Note: The advantage of using only 11 bits is with CAN-FD where initial arbitration runs in slower speed and by not using 29 bits it will be shorter and thus communication faster.

First data byte

The first data byte in the CAN frame has special meaning.

For CAN frames with First bit set (1) in CAN ID the first byte must be destination device address. This identifies the device this SHV RPC message is intended for.

For CAN frames with First bit unset (0) in CAN ID the first byte must be sequence number that starts with 0x0 for the second CAN frame (third is 0x1 and so on). If sequence number reaches 0xff then it just simply wraps around to 0x0.

The complete and valid SHV RPC message thus starts with CAN frame with First set, continues with CAN frames where First is not set and NotLast is set and first data byte is sequence, and terminates by CAN frame where NotLast is not set. The consistency of the message (that no CAN frame is lost) is ensured with counter in first data byte.

Transport error is detected if CAN frame with First not set in CAN ID is received with byte having number in first data byte out of order (while ignoring frames with same number as the last received CAN frame from that specific device). Such SHV RPC message is silently dropped and error is ignored as subsequent messages can be consistent again without any action.

The message can be terminated by CAN RTR frame (Remote transmission request) with both NotLast and First unset and data length set to 0x0.

Connection

Connection between devices is automatically established with first message being exchanged. There can be only one connection channel between two devices. To disengage it the ResetSession message can be sent (that is regular CAN frame with NotLast not set and First set in CAN ID and data containing only byte with destination device address and one 00 byte).

Broadcast

Due to the CAN bus bandwidth limitations it is suggested to expose SHV RPC Broker instead of just plain device, but sometimes it is beneficial to not add the signal filtering and instead to automatically broadcast signals. Such signals are not intended for any specific device and are just submitted on the bus with special destination device address 0xff.

The only allowed SHV RPC message type for destination device address 0xff is RpcSignal. Other message types received with this destination device address must be ignored and devices should not send them.

Handling of the broadcast signals is up to the application it receives it. SHV RPC Brokers will propagate them further if given device is mounted and subscribed.

One concrete example when this is beneficial is for date and time synchronization device. Such device can send signals with precise time to let other devices to synchronize with it.

SHV Device discovery

SHV devices on CAN bus must be possible to discover to not only be able to dynamically mount them to SHV RPC Broker but also to actively diagnose the CAN bus.

Once device is ready to receive messages on CAN bus it should send CAN RTR frame (Remote transmission request) with data length set to 0x1 (the CAN ID should have NotLast not set and First set which applies to all CAN RTR frames). This ensures that it gets discovered as soon as possible.

Device that wants to perform discovery can send CAN RTR frame (Remote transmission request) with data length set to 0x2. Device that receives this CAN frame must respond with CAN RTR frame with data length set to 0x1.

Address collision resolution

The standard deployment should prevent address collisions by allocating them for the whole bus before deployment, but there is also a use case for on site ad hoc connection and in such situation it is unclear what address should be chosen. The device should select random unallocated address but that won't prevent collision, only minimize them.

All CAN devices (that includes SHV clients) must listen for others using their address and must send CAN RTR (Remote transmission request) frame with size set to 0x3 when they receive CAN frame they did not send. CAN devices with same dynamic address receiving this RTR frame must choose a different one.

The best practice is to choose address from upper range (close to 255) and initially send dummy CAN RTR frame with size set to 0xf. Do this in 200ms intervals five times. The communication with other CAN devices can start if no CAN RTR frame with size 0x3 is received in the meantime.

CAN RTR frames index

This is full list of all different CAN RTR frames used in the protocol:

SHV RPC URL

Unified Resource Locators are common way to specify connection. This is definition of such URL for Silicon Heaven RPC.

Examples of URLs for Silicon Heaven RPC:

tcp://user@localhost:3755?password=pass
tcp://user@localhost:3755?password=pass&devid=42
unix:/run/shvbroker.sock

The base format is:

URL = scheme ":" ["//" [username "@"] authority] [path] ["?" options]

options is sequence of attribute-value pairs split by = an joined by & (example: password=test&devid=foo). Generally supported options are:

• password: Plain text password used to login to the server.
• shapass: Password hashed with SHA1 used to login to the server.
• user: Alternative way to set user name that overrides user in URL. The default user name if neither is used is local user's name or the platform specific alternative.
• devid: Identify to the other side as device with this ID.
• devmount: Identify to the other side as device and request mount to the given location.

TCP/IP protocol

scheme = "tcp"
authority = host [":" port]

The default host is localhost and port is 3755. Any non-empty path is invalid as it has no meaning in IP.

This uses TCP/IP with Block transport layer.

TCP/IP serial protocol

scheme = "tcps"
authority = host [":" port]

This is variant of TCP/IP protocol. It is same except of used transport layer. Please refer to the previous section for more info. The only difference is the default port that is 3765.

This uses TCP/IP with Serial transport layer.

TCP/IP protocol with SSL

scheme = "ssl"
authority = host [":" port]

The default host is localhost and port is 3756. Any non-empty path is invalid as it has no meaning in IP.

The additional supported options are:

• ca: Path to the file with CA certificates used to verify the peer.
• cert: Path to the file with certificate. For clients connecting to the server this is client certificate that is validated by server for access and in some cases can replace password. For server this is certificate clients verify to validate if they are connecting to the correct server.
• key: Path to the file with secret part of the cert. This must be specified alongside with cert.
• crl: Path to the file with certification revocation list. This is used to invalidate client certificates on the server.
• verify: can be used with either true or false to control if server should be verified or not. The default, if not specifies, is true. Setting false forces client to accept any certificate as valid.

This uses TLS TCP/IP with Block transport layer.

TCP/IP serial protocol with SSL

scheme = "ssls"
authority = host [":" port]

This is variant of TCP/IP protocol with SSL. It is same except of used transport layer. Please refer to the previous section for more info. The only difference is the default port that is 3766.

This uses TLS TCP/IP with Serial transport layer.

Unix/Local domain socket

scheme = "unix"

There is no default path and thus empty path is considered invalid. Any non-empty authority is also considered as invalid because it has no meaning.

This uses Unix sockets for local interprocess communication with Block transport layer.

Unix/Local domain socket serial protocol

scheme = "unixs"

This is variant of Unix/Local domain socket. It is same except of used transport layer. Please refer to the previous section for more info.

This uses Unix sockets for local interprocess communication with Serial transport layer.

Serial / RS232

scheme = ("serial" | "tty")

path needs to point to valit serial device. There is no default path and thus empty path is considered invalid. Any non-empty authority is also considered as invalid because it has no meaning.

The additional supported options are:

• baudrate: Specifies baudrate used for the serial communication.

Other common serial-port parameters are at the moment specified as not configurable and are expected to be: eight bits per word, no parity, single stop bit, enabled hardware flow control, disabled software flow control.

This uses serial console or terminal like interface as bidirectional stream channel with Serial transport layer.

WebSocket

scheme = ("ws")
authority = host [":" port]

The default host is localhost and port is 8755.

WebSocket over SSL

scheme = ("wss")
authority = host [":" port]

The default host is localhost and port is 8766.

• ca: Path to the file with CA certificates used to verify the peer.
• cert: Path to the file with certificate. For clients connecting to the server this is client certificate that is validated by server for access and in some cases can replace password. For server this is certificate clients verify to validate if they are connecting to the correct server.
• key: Path to the file with secret part of the cert. This must be specified alongside with cert.
• crl: Path to the file with certification revocation list. This is used to invalidate client certificates on the server.
• verify: can be used with either true or false to control if server should be verified or not. The default, if not specifies, is true. Setting false forces client to accept any certificate as valid.

SHV RPC RI

This is definition of Resource Identifiers for SHV RPC. SHV has two types of resources; those are methods and signals.

Resource identifiers are human readable and are used thorough this documentation as well as in the broker's signal filtering. They can also be used by other tools.

The format is either PATH:METHOD for methods or PATH:METHOD:SIGNAL for signals.

The first field PATH is the SHV path. It can be glob pattern (rules from POSIX.2, 3.13 with added support for ** that matches multiple nodes). Note that empty PATH is the root of the SHV node tree.

The second field METHOD is the SHV RPC method name. It can be wildcard pattern (rules from POSIX.2, 3.13). The empty method name is invalid and thus is invalid pattern.

The third field SIGNAL that is used only for signals is the SHV RPC signal name while METHOD in such case is its source. It can be wildcard pattern (rules from POSIX.2, 3.13). The empty signal name is invalid and thus is invalid pattern.

The examples of Resource Identifiers for methods and method matching:

The examples of Resource Identifiers for signals and signals matching:

Please note that the method RI matches all signals associated with the corresponding method. The right to invoke a method also grants the right to receive any signals of its associated method.

Implementations

This is the list of projects implementing at least ChainPack and/or CPON data formats. Most of the provide the abstraction on SHV RPC.

C

shvc

C implementation targeting Unix-like operating systems. This includes support for embedded OS NuttX, and thus this implementation is concerned with memory size more than others. It is using C23 with GNU extensions.

This project also implements some tools that can be used from CLI and SHV RPC Broker.

libshv

Implementation of ChainPack and CPON formats. The transport layers or any SHV RPC APIs are not provided. The implementation is malloc() free and depends only on C99.

C++

libshv

The most advanced implementation of SHV but also with highest historical baggage relative to the previous SHV 3.0 versions.

The additional applications are provided based on this library in shvapp which include SHV RPC Broker.

JavaScript

libshv-js

Implementation intended to be used in web browsers to access SVH RPC over WebSocket. It provides SHV RPC message exchange functionality, but higher levels of SHV APIs integration is not provided.

Python

pySHV

Pure Python implementation intended as SHV 3.0 reference.

In combination with Python testing frameworks (such as pytest) it is a very good option for testing any SHV application.

This project also implements SHV RPC Broker and History recording as a separately usable applications.

Rust

libshvrpc-rs

This project is split across multiple repositories and thus libraries providing a way to select the minimal requirements for the specific application.

• libshvproto-rs that provides ChainPack and CPON support.
• libshvrpc-rs providing basic support for SHV RPC based on libshvproto-rs.
• libshvclient-rs is the ideal library if you intend to implement client application based on the libshvrpc-rs.

Tools

These are tools and utilities to be used with SHV RPC.

Clients

These are clients for generic access of the SHV RPC. They provide an interactive interface to discover nodes and methods as well as to interact with them.

• shvspy: Qt based graphical client
• shvcli: command line interface client

Brokers

These are standalone server applications implementing SHV RPC Broker functionality.

• shvapp: provides shvbroker
• pyshv: provides pyshvbroker
• shvbroker-rs: provides shvbroker

History aggregators

These are standalone server applications implementing SHV RPC History functionality. They aggregate history logs.

No standalone implementation that fulfils the SVH 3.0 requirements is available, yet (07.2024).

CLI tools

These are generic command line tools that are designed to work with resources exposed over SHV RPC. They can be used in scripts or just from interactive session. They are tools for performing some specific operations.

Method call

Generic way to call a single method. This is minimal functionality required to get SHV RPC to work and thus it is provided by most of the implementations in one way or the other.

• shvapp: provides shvcall
• shvc: provides shvc
• shvcall-rs: provides shvcall

Signals retrieval

Tools to subscribe and receive SHV RPC signals.

• shvc: provides shvcsub

File copy

The tool to copy from and to SHV RPC File nodes.

• shvc: provides shvcp

Exchange interaction

Exchange nodes provide bi-directional stream of data. This is consistent with CLI and thus it is beneficial to sometimes attach this stream to the console or to some pipes.

• shvc: provides shvctio

ChainPack/CPON conversion

• libshv: provides cp2cp
• pyshv: provides pycpconv

Formal declaration of SHV tree

SHV tree provides API and thus it is beneficial to have a standard way to describe this API in format that is readable by humans as well as computer. This is provided by SHVTree project.