Computer scientists seem to reinvent the wheel quite often. For the more cynical observers every new Remote-Procedure-Call implementation looks quite like CORBA or Sun-RPC, but under a different name, different interfaces and - more often than not - some restricted features or inefficiencies. Welcome to Ubus! [2, 4]

What is Ubus

Ubus is a (local) message bus, specifically for the embedded Linux distribution OpenWRT [1]. OpenWRT is a distribution which brings Linux to many Routers, Access Points or Internet Modems. For example Freifunk mostly uses OpenWRT based installations for their devices.

Ubus contains from a server process (ubusd), which provides the message bus. Client applications can then connect to the Ubus, ask for specific objects on the bus and call methods of these objects, or they can create new objects and provide methods. There are no attributes on objects in the bus, only methods, i.e. remote procedure calls. In a sense, the ubusd-process is more of a broker, handling the discovery of objects.

Ubus is based on libubox [3], which offers a message-queue like programming interface, allowing C programs to wait for new messages, some functions for marshalling and demarshalling of user data inside so called blobmessages. libubus is single threaded in design. There is a single blob_buf b, so it is only possible to handle one message at a time, adding another thread, which also tries to interact with Ubus will make it crash.

In the remainder of this post I'm gonna refer to the application providing an object and methods as the server, and the one using an object and its methods as the client application.

Trying things out

While working on the server is might come in handy to try things out against a - sort of - reference implementation of a client. On OpenWRT there is the ubus tool preinstalled. It can be used to query the ubus objects and their methods and to call methods or subscribe to events. These are some examples:

ubus list
ubus -v list
ubus call test sendHello "{'id': 12, 'message': 'Testmessage'}"

Defining Objects and Methods on the Server

First, we need to make a type for ubus to use. A type has a name and some methods. Methods are represented by a name which is used to identify the methods on the client-side. Methods also need the callback to be called when the method is requested by a client and it needs a so called polcy, if it takes any parameters. A policy represents the possible arguments of the function and their types. Not, that return values do not need a policy, but you may want one nonetheless, more on that later.

Let's say we want an object "test", which has two methods: "hello", taking no parameters and returning "hello world" and a second one, taking an id of type int and a string "msg".

#include <libubox/blobmsg_json.h>
#include <libubus.h>

// This enum is helpful, but not necessary.
// Its entries allow easier access to the methods later on
enum {
    HELLO_ID,
    HELLO_MSG,
    __HELLO_MAX
};


// the policy describes the paramters of a method
static const struct blobmsg_policy hello_policy[] = {
    // id is of type int
    [HELLO_ID] = { .name = "id", .type = BLOBMSG_TYPE_INT32 },
    // message is of type string
    [HELLO_MSG] = { .name = "msg", .type = BLOBMSG_TYPE_STRING },
};

// forward definitions of our callbacks
int testHello(struct ubus_context *ctx, struct ubus_object *obj,
        struct ubus_request_data *req, const char *method,
        struct blob_attr *msg);
int sendHello(struct ubus_context *ctx, struct ubus_object *obj,
        struct ubus_request_data *req, const char *method,
        struct blob_attr *msg);

// setting up a list of methods, one without arguments (hello)
// and one with the sendHelloPolicy, so (int id, char* msg)
static const struct ubus_method test_methods[] = {
    UBUS_METHOD_NOARG("hello", test_hello),
    UBUS_METHOD("sendHello", sendHello, sendHelloPolicy),
};

// defining the type "test" with the above methods
static struct ubus_object_type test_object_type =
    UBUS_OBJECT_TYPE("test", test_methods);

// and defining an object of type test with the methods
static struct ubus_object test_object = {
    .name = "test",
    .type = &test_object_type,
    .methods = test_methods,
    .n_methods = ARRAY_SIZE(test_methods),
};

We have now the data structures describing an object "test" of type "test" with the method "hello" and "sendHello". The list of possible types is this. Note there are arrays and tables which we'll need to cover later a bit more in depth:

enum blobmsg_type {
    BLOBMSG_TYPE_UNSPEC,
    BLOBMSG_TYPE_ARRAY,
    BLOBMSG_TYPE_TABLE,
    BLOBMSG_TYPE_STRING,
    BLOBMSG_TYPE_INT64,
    BLOBMSG_TYPE_INT32,
    BLOBMSG_TYPE_INT16,
    BLOBMSG_TYPE_INT8,
    BLOBMSG_TYPE_BOOL = BLOBMSG_TYPE_INT8,
    BLOBMSG_TYPE_DOUBLE,
    __BLOBMSG_TYPE_LAST,
    BLOBMSG_TYPE_LAST = __BLOBMSG_TYPE_LAST - 1,
    BLOBMSG_CAST_INT64 = __BLOBMSG_TYPE_LAST,
}

Now let's make the ubus daemon aware of our object:

int main(int argc, char **argv) {
    // initialize the ubox library
    uloop_init();

    // connect to the ubus socket
    // the parameter may be a path to the ubus socket
    ctx = ubus_connect(NULL);
    if (!ctx) {
        fprintf(stderr, "Failed to connect to ubus\n");
        return -1;
    }

    ubus_add_uloop(ctx);

    // add our object to the ubus object space
    int ret = ubus_add_object(ctx, &test_object);
    if (ret) {
        fprintf(stderr, "Failed to add object: %s\n", ubus_strerror(ret));
    }

    // wait in the uloop event queue
    uloop_run();

    // deinit ubus and clean up
    ubus_free(ctx);
    uloop_done();

    return 0;
}

So now, when we build and run our code, we will see no output of our program, but we will see that there is a new object "test" on the ubus and we will see, that it has two methods.

# build
gcc server.c -lubus -lubox

# run
./a.out

# see
ubus list
ubus list -v

Methods and Parameters

We've seen before the prototype of our callback functions. Let's first have a look into how we demarshal parameters out of the ubus objects:

// - ctx, obj and req are pointers to internal state of the ubus library
// we could check which object is wanted, but we'll not gonna bother here
// - method is the name of the method called, useful if there is one
// callback for more than 1 method
// - msg is the payload of the request
int sendHello(struct ubus_context *ctx, struct ubus_object *obj,
        struct ubus_request_data *req, const char *method,
        struct blob_attr *msg) {

    // tb is an array of pointers to blob_attr.
    // this serves as indices into the message, from which we'll
    // be able to get our parameters
    struct blob_attr *tb[__HELLO_MAX];

    // parse the message and fill the tb-pointers
    blobmsg_parse(hello_policy, ARRAY_SIZE(hello_policy), tb, blob_data(msg), blob_len(msg));

    // if there was a message present, print it.
    if (tb[HELLO_MSG]) {
        printf ("Received %s\n", blobmsg_get_string(tb[HELLO_MSG]));
    }

    if (tb[HELLO_ID) {
        printf ("Recieved ID: %s\n", blobmsg_get_u32(tb[HELLO_ID]));
    }

    // Todo: respond

    return 0;
}

For most simpler requests, it is sufficient to call blobmsg_parse for demarshalling the message into the tb array. Its prototype is:

int blobmsg_parse(const struct blobmsg_policy *policy, int policy_len,
                  struct blob_attr **tb, void *data, unsigned int len)

blobmsg_parse takes the policy and its length, so it knows, which values to expect. The pointer to the array tb is used to write indizes into the tb array, so we can access the message parameter as tb[1] or tb[HELLO_MSG], as the message-entry of the policy is on place 1. The last two parameter is the pointer to the data and its length.

Afterwards we'll have a filled tb array. If an entry is NULL, the parameter was not found, so the client probably did not send it. Trying to access the parameter will crash our application, so we should always check its existence before using and values and have either sane defaults or ones, that imply that this parameter is missing.

After parsing, we can access our data by using blobmsg_get_.... This will return either the value (for scalar values) or a pointer to a string (if it's a string). For string we'd need to copy it, if we'd need it later on, I'm gonna skip that. Not, that there are no blogmsg_get_... variants for tables and arrays.

Arrays and Tables

The fun really starts with arrays and tables. Arrays are in a sense lists of entries, they may even have differing types inside them. I'll assume same typed entries for the time being. Tables are like structs in C. In a way they are arrays with names entries, which is how they can be used in ubus.

Arrays

blobmsg_parse will not get us very far here. Let's assume the following policy of a delete call of a list object:

enum {
    DELETE_IDS,
    __DELETE_MAX
};

static const struct blobmsg_policy delete_policy[] = {
    [DELETE_IDS] = { .name = "ids", .type = BLOBMSG_TYPE_ARRAY },
};

The delete method takes a single parameter, which has the array-type. However, we don't have to tell ubus at this point which types are inside the array, just that it's an array of things.

Now we can use blobmsg_parse to get us the ids parameter, but then we'll still have a blob_attr* pointer to demarshall. Let's say, we'd want to copy out the array for later use.

static int delete_cb(struct ubus_context *ctx, struct ubus_object *obj,
        struct ubus_request_data *req, const char *method,
        struct blob_attr *msg ) {
    struct blob_attr *tb[__DELETE_MAX];

    blobmsg_parse(delete_policy, __DELETE_MAX, tb, blob_data(msg), blob_len(msg));
    if ( tb[DELETE_IDS] ) {
        // get the length of the array
        int len = blobmsg_check_array(tb[DELETE_IDS], BLOBMSG_TYPE_INT32);
        int *arr = (int*) malloc ( len * sizeof(int) );
        int i=0;

        struct blob_attr *cur;
        size_t rem;
        // this macro sets up a for loop over all entries in the array
        // cur is always the current object
        // tb[...] is the array to be iterated over
        // and rem is an int for the remainder
        blobmsg_for_each_attr(cur, tb[DELETE_IDS], rem) {
            if (blobmsg_type(cur) != BLOBMSG_TYPE_INT32) {
                continue;
            }

            arr[i++] = blobmsg_get_u32( cur );
            // do stuff
        }
    }

    // respond
    return 0;
}

So we need to get the length of the array. You could iterate over the list two times, once for checking the size and the second one for copying out the values. This is probably what you need to do when the array has differing types in it.

The foreach-macro is a bit on the kernel-level C coding style, which is why I will not copy it in here. You can look it up and try to understand it, I will not.

Tables

Tables are in a sense the same as arrays, but its values have names. So lets assume, we'd want to add to a key-value store with a typed value (i.e. the parameters are a name, a type-reference and a value).

We would initialize our policy like this, and maybe create a struct for future reference in our key-value store program:

struct kv_table {
    char* key;
    int type;
    union {
        void* value;
        uint32_t int_val;
        bool bool_val;
        // ...
    };
};

enum {
    APPEND_TABLE,
    __APPEND_MAX
};

static const struct blobmsg_policy append_policy[] = {
    [APPEND_TABLE] = { .name = "table", .type = BLOBMSG_TYPE_TABLE },
};

So then let's take a look into parsing a table. This is a bit boring if the only thing you get passed is a table, but think about nesting table-structures into an array. Then you cannot just add your table names to the policy, but have to iterate over the array and then the table.

static int append_cb(struct ubus_context *ctx, struct ubus_object *obj,
        struct ubus_request_data *req, const char *method,
        struct blob_attr *msg ) {
    struct blob_attr *tb[__APPEND_MAX];

    blobmsg_parse(append_policy, __APPPEND_MAX, tb, blob_data(msg), blob_len(msg));
    if ( tb[APPEND_TABLE] ) {
        struct kv_table payload;
        payload.type = -1;

        struct blob_attr *cur;
        size_t rem;
        // as above we'll iterate over the entries of the table
        blobmsg_for_each_attr(cur, tb[APPEND_TABLE], rem) {
            char* name = blobmsg_name ( cur );
            if ( strcmp ( "name", key ) == 0 ) {
                // you could (should) check the type of the entry here
                payload.name = blobmsg_get_string ( cur );
            } else if ( strcmp ( "type", name ) == 0 ) {
                payload.type = blobmsg_get_u32 ( cur );
            } else if ( strcmp ( "value", name ) == 0 ) {
                if ( payload.type == -1 ) {
                    // check blobmsg_type
                    // ...
                } else {
                    switch ( payload.type ) {
                        case 1:
                            payload.bool_val = blobmsg_get_bool ( cur );
                            break
                        // ...
                    }
                }
            }

            // do stuff
        }
    }

    // respond
    return 0;
}

Sending Responses

Other than the parameters, there is no policy needed on what to return. For replying to requests we need the call ubus_send_reply:

int ubus_send_reply(struct ubus_context *ctx, struct ubus_request_data *req,
        struct blob_attr *msg);

We need to give it the ubus context and the original request and a blob_attr message to send. Now let's go back to the "hello" method from above, which should just return "hello world".

First we need to initialize a struct blob_buf. We can have that either static in our C-file or on the stack inside the function. After initialization, we can add values to the buffer, here we just add an attribute "message" and send it away with the ubus_send_reply call. We give that the head pointer to the list of parameters it created.

struct blob_buf b;
blob_buf_init(&b, 0);
blobmsg_add_string(&b, "message", "Hello World");

return ubus_send_reply(ctx, req, buf.head);

Nested Responsed: Arrays and Tables

Sending arrays and tables behaves quite the same, but adds nesting. We will open an attribute of our buffer and add entries to it, either names entries (table) or entries without a name (array). Nesting needs a cookie, which is returned on opening and will be needed for closing, as the nesting operations change the internal state of our buffer-structure and we want to reset it after we're done.

// as a reference: the _array and the _table variants just call
// _nested with a single bool value changed:
void *blobmsg_open_nested(struct blob_buf *buf, const char *name, bool array);

static inline void *
blobmsg_open_array(struct blob_buf *buf, const char *name)
{
    return blobmsg_open_nested(buf, name, true);
}

static inline void *
blobmsg_open_table(struct blob_buf *buf, const char *name)
{
    return blobmsg_open_nested(buf, name, false);
}

// our code:
void* cookie = blobmsg_open_array ( &buf, "ids" );
    blobmsg_add_u32 ( &buf, NULL, 12 );
    blobmsg_add_u32 ( &buf, NULL, 13 );
    blobmsg_add_u32 ( &buf, NULL, 14 );

blobmsg_close_array ( &buf, cookie );

When working with tables, the NULL-parameters are used as names for the fields of the table.

Client: Sending Requests

Until now we've only looked at the server code. However sending requests is equally important. The ubus tool only gets us so far. First we need to find the object we want to use.

int rc = ubus_lookup_id ( ctx, "test", &id );
if ( rc ) {
    printf ("Could not find object test\n");
    return rc
}

ubus_lookup_id takes the name of the object we want to have and returns then id of that object. This is an ubus-internal id, we'll be using it to refer to this object, when we call methods of that object.

For calling methods we need either ubus_invoke or ubus_invoke_async. I will tell you a little secret: the synchronous ubus_invoke uses the asynchronous ubus_invoke_async internally and (busy) waits for the response. No matter which one we'll use, we will need a complete-callback, which is called, when the response was received. This is the only place, we will be able to see the response-data.

// ubus_data_handler_t, type of the callback
typedef void (*ubus_data_handler_t)(struct ubus_request *req,
        int type, struct blob_attr *msg);


// prototype of ubus_invoke
int ubus_invoke(struct ubus_context *ctx, uint32_t obj, const char *method,
        struct blob_attr *msg, ubus_data_handler_t cb, void *priv,
        int timeout);

// usage:
ubus_invoke ( ctx, id, "hello", b.head, myCallback, NULL, 3000 );

ubus_invoke will return, when the response is received and the transaction complete or the timeout (3000ms) is hit. The return value will indicate whether or not the transaction was successful. In the example call, b is again a buffer we filled with values, and NULL is a pointer to private data. This data will be handed right to the complete-callback, so it may be used to hand it memory to write the received response to.

The callback could look like this:

enum {
    HELLO_RESP_MSG,
    __HELLO_RESP_MAX
};

static const struct blobmsg_policy hello_response_policy[] = {
    [HELLO_RESP_MSG] = { .name = "message", .type = BLOBMSG_TYPE_STRING },
};

void myCallback ( struct ubus_request *req, int type, struct blob_attr *msg ) {
    struct blob_attr *tb[__HELLO_RESP_MAX];
    // void* priv = req->priv;

    blobmsg_parse(hello_response_policy, __HELLO_RESP_MAX, tb, blob_data(msg), blob_len(msg));

    if ( tb[HELLO_RESP_MSG] ) {
        printf ("received: %s\n, blobmsg_get_string(tb[HELLO_RESP_MSG]);
    }
}

Using a policy for return values does make response handling a bit easier. You can of course just handle the parameters yourself. Note, that the private-data pointer is inside of the request-structure.

This is also the point where a code generator would make sense, which generates stubs from a descriptor format, so the boilerplate code is generated, like handling of parameters and return values. This takes some of the flexibility out of it, but makes it easier to use.

Events and Subscriptions

Ubus also allows subscribing to objects and sending events. We can subscribe to any object, even ones without a a name (for example when a call returns an object id). As a subscriber, we need a ubus_subscriber callback, which contains a callback for every event and a callback for when the object was removed. When subscribing we implicitely create a new object, to which all events are sent.

typedef int (*ubus_handler_t)(struct ubus_context *ctx, struct ubus_object *obj,
        struct ubus_request_data *req,
        const char *method, struct blob_attr *msg);
typedef void (*ubus_remove_handler_t)(struct ubus_context *ctx,
        struct ubus_subscriber *obj, uint32_t id);

struct ubus_subscriber {
    struct ubus_object obj;

    ubus_handler_t cb;
    ubus_remove_handler_t remove_cb;
};

int ubus_subscribe(struct ubus_context *ctx, struct ubus_subscriber *obj, uint32_t id);

When we get a remove-callback, we can assume, that the object is gone at will not send any events anymore. Whenever we get an event until then, the cb-callback will be called, again allowing us to parse the messages. The sender can notify us under different names (or methods), which can be used to identify certain states or statechanges.

On the sender-side, first we'll add a subscribe-callback, which is called when a subscriber appears or disappears:

void subscriber ( struct ubus_context *ctx, struct ubus_object *obj ) {
    printf ("Subsciber change!\n");
}

static struct ubus_object test_object = {
    .name = "test",
    .type = &test_object_type,
    .methods = test_methods,
    .n_methods = ARRAY_SIZE(test_methods),
    .subscribe_cb = subscriber
};

When an event occurs on the server side, we can notify our subscribers by using ubus_notify.

int ubus_notify(struct ubus_context *ctx, struct ubus_object *obj,
        const char *type, struct blob_attr *msg, int timeout);

Again the msg is payload data like above. type can be used to identify different state changes and will be passed to the subscriber as a method.

Timeouts

As many of the message loop systems do, uloop also provides timeouts. Give uloop_timeout_add a struct uloop_timeout and a timeout, it will execute the appropriate callback function when the timeout has expired.

void myTimoutCallback ( struct uloop_timeout* t ) {
    //
}

struct uloop_timeout myTimeout = {
    .cb = myTimeoutCallback
};

uloop_timeout_add ( &myTimeout, 3000 );

The callback receives a pointer to the original timeout structure. This is unfortunate, because we cannot add private data to the timeout callback, at least if we don't want to get dirty. If we decide to play dirty, we can use the container_of macro, which every Linux Kernel developer knows so well. libubox has contianer_of itself in the list.h. It is defined as follows:

#ifndef container_of
#define container_of(ptr, type, member) \
    ({  \
        const __typeof__(((type *) NULL)->member) *__mptr = (ptr);  \
        (type *) ((char *) __mptr - offsetof(type, member));        \
    })
#endif

container_of takes a pointer, a type and a member of that type. Let's sway we have a structure like this:

struct privateData {
    struct uloop_timeout t;
    int private;
};

A uloop_timeout is embedded into our structure. In this special case, it is (most probably, if the compiler likes us) the first member of the structure, so a normal cast would suffice:

// do not copy!
void myTimoutCallback ( struct uloop_timeout* t ) {
    struct privateData* priv = (struct privateData*) t;
}

However, if the compiler decides to reorder our structure, we are out of look and just created a bug, which is quite hard to trace. But we can use container_of, which will make the compiler do the offset-calculation. We have a pointer to the embedded uloop_timeout, and want a pointer to the enclosing structure, which is right around the uloop_timeout in memory.

So we tell the compiler the pointer we have, the member inside the enclosing structure, were this pointer points to, and the type of the enclosing structure. Let's alter the structure and see an example:

struct privateData2 {
    int priv2;
    struct uloop_timeout t;
    int private;
};


void myTimoutCallback ( struct uloop_timeout* timeout ) {
    struct privateData* priv = container_of(timeout, t, struct privateData2);
}

Let's see what it does:

... [ priv2 |       t       | private ] ...
    0       4               x        x+4
           / \
            |
         timeout

container_of calculates the offset of timeout inside our enclosing structure. We tell it the member to which timeout points, which is t, then calculates the offset of that member inside the struct and substracts that from our original pointer. The problem here is, that we cannot be totally sure, that this structure is always around the timeout. So when using container_of make sure and be absolutely certain, that you always have the embedding structure around the embedded structure. The compiler won't warn you.

Conclusion

Ubus is a (more or less) simple message bus for embedded Linux distributions (like OpenWRT). You can call methods, subscribe and send events and add timeouts. Unfortunately ubus is not designed for multithreaded applications, throwing segementation faults left, right and center when a second thread comes anywhere near ubus-code.

As always with these message-loops you need to call uloop_run repeatedly (or once to that matter), if you want to get new events or messages. If your C-code takes a while, you will not get new messages - of course you don't, you only have one single thread.

A big drawback is, that ubus is essentially written like the Linux Kernel, which sometimes makes it hard to read. Also there is next to no documentation for any of it, apart from the code (at least that I can find). Going through the examples only brings you that far, reading through the tests of libubox gives a bit more insight, but still there is no place, which really explains these things. Unfortunately there is also no (publically available) code generator of stub-code.

But now there is at least some documentation. If you find any problems or mistakes, please let me know!

References

[1] https://openwrt.org/ [2] https://git.openwrt.org/project/ubus.git [3] https://git.openwrt.org/project/libubox.git [4] https://openwrt.org/docs/techref/ubus