Embeddable scripting languages are for integration testing too!

Source

tl;dr needed some fake clients, went the scripting route instead of building several little executables.

What?

At work, among other things, I'm working on a debugging/postmortem analysis component for a bigger, complex and not-so-tested system. This little service is really important and used during integration tests, so "it has to work".

While I managed to find and remove lots of bugs early (based TDD!), some behaviors of the service "as a whole" were not tested as much as I'd like.

Explaining the service a little more

Please, don't close the tab and keep reading. I need to explain some more details about the service I want to put under test.

This little "thing" receives messages via ZeroMQ, has to write them to files and stream the received information to other clients to do some realtime graphs/visual analysis, or whatever they prefer. Clients can communicate to the component using a simple, custom C++ library that "hides" the ZeroMQ client code, so we have to test it, that messages are received correctly and that the commands sent by clients are executed.

For the sake of simplicity, and to give you both an idea of what I'm discussing and a runnable proof of concept, I wrote a toy example on Bitbucket.

An example of the library code is the following:

void ClientClass::logValue(uint8_t value_to_send) {
  std::cout << "sending value: " << (int)value_to_send << std::endl;
  char *buffer = new char[2];
  buffer[0] = 1;
  buffer[1] = value_to_send;

  int res = zmq_send(socket, buffer, 2, 0);
  assert(res == 2);

  delete[] buffer;
}

The idea

At first, I tried to simulate as much cases as possible in a single "testing client" -a fake program to simulate actual clients-, but I understood the hard way that it was a very stupid idea: tests became difficult to extend very, very soon, and it was difficult to modify it to test something without breaking something else. After the enlightment, I started thinking about having several fake clients, each one tailored on the situation I wanted to test (message bursts, ntp client/server malfunctions, etc).

I didn't want to include these little executables to my daily dev build, though: it'd take too much time to do a full rebuild of the system, and time is precious. This constraint made me think about using a scripting language: "just write a script in some sort of DSL and you're done", I thought.

ChaiScript, or: extending an embeddable language

So, I started thinking about how to implement such a language. How difficult it could be? The more I though about it, the more use-cases and features I deemed necessary. From a bunch of log* functions, it evolved into a limited but general-purpose language with variables, functions and cycles. Building your own language is a cool, fun and "formative" experience, I didn't have the time to actually write an interpreter or a compiler and maintain it in addition to the real project. The right thing to do was to pick an existing solution and integrate it inside the real project.

So, after a little searching on the Wired, I came across ChaiScript, a simple Javascript-like language written in C++, designed to be embedded by C++ applications. The usual cmake . && make && make install will build and install it from sources.

Let's write the server first...

As said before, the server is just a simple ZeroMQ ROUTER in a DEALER-ROUTER communication scheme. For the sake of simplicity, in this article we will work on a simplified example, a proof of concept.

The protocol between the server and its clients is very simple: messages cannot be bigger than 20 bytes, and two kinds of payloads can be sent.

byte  0: message_type (0=ascii, 1=number)
bytes 1-20: payload

If message_type is 0, an ASCII string is sent to the server, if 1 it's a unsigned int number between 0 and 255 -only one byte will be used to store it-.

#include <zmq.h>
#include <iostream>
#include <cstring>

int main() {
  void *context = zmq_ctx_new();
  void *socket_ = zmq_socket(context, ZMQ_ROUTER);

  // set a timeout
  unsigned int recv_timeout_ms = 10000;
  zmq_setsockopt(socket_, ZMQ_RCVTIMEO, &recv_timeout_ms,
                 sizeof(recv_timeout_ms));

  if (zmq_bind(socket_, "tcp://127.0.0.1:6987") == -1) {
    std::cout << "err: could not bind, " << zmq_strerror(zmq_errno())
              << std::endl;
    return 1;
  }

  // needed to skip identity messages
  bool is_recv_message_identity = false;
  bool timeoutted = false;

  while (!timeoutted) {
    char buffer[20];
    bzero(buffer, 20 * sizeof(char));
    int recv_res = zmq_recv(socket_, buffer, 19, 0);

    if (recv_res == -1) {
      if (zmq_errno() == 11) { // EAGAIN
        timeoutted = true;
        continue;
      } else {
        std::cout << "err: could not receive [" << zmq_errno() << "] "
                  << zmq_strerror(zmq_errno()) << std::endl;
        return 2;
      }
    }

    // skip identity messages: they're not the payload we are looking for,
    //      and we don't need to do identity-based logic in this PoC.
    is_recv_message_identity = is_recv_message_identity ? false : true;
    if (is_recv_message_identity)
      continue;

    if (buffer[0] == '\0') {
      std::cout << "received a string!" << std::endl;
      std::cout << "the received string is: >>>" << std::string(buffer + 1)
                << "<<<" << std::endl;
    } else {
      std::cout << "received a number!" << std::endl;
      std::cout << "the received number is: " << (int)buffer[1] << std::endl;
    }
  }

  int sock_res = zmq_close(socket_);
  if (sock_res == -1)
    return sock_res;
  int ctx_res = zmq_ctx_shutdown(context);
  return ctx_res;
}

... then, the client script runner

Being a proof of concept discussed here, we'll keep things simple and limit our example to just two functions: logDebug(str) and logValue(unsigned int), that send a string or an unsigned int to the server via ZeroMQ.

#include <chaiscript/chaiscript.hpp>
#include <chaiscript/utility/utility.hpp>
#include <iostream>
#include <mahlib.h>

int main(int argc, char* argv[]) {
  // handle parameters. if no parameter is given, use "test.chai"
  // as default script name.
  std::string script_name = "test.chai";
  if(argc == 2){
    script_name = std::string(argv[1]);
  }
  else if(argc > 2){
    std::cout << "usage: " << argv[0] << " <script.chai>" << std::endl;
    return 1;
  }

  // create the ChaiScript interpreter
  chaiscript::ChaiScript chai;

  // extend ChaiScript with our ClientClass class,
  // so it can recognize our class' usage in the module,
  // call the right function at the right time, etc.
  chaiscript::ModulePtr m = chaiscript::ModulePtr(new chaiscript::Module());
  chaiscript::utility::add_class<ClientClass>(*m,
     "ClientClass",
     { chaiscript::constructor<ClientClass(const std::string)>()},
     { {chaiscript::fun(&ClientClass::logDebug), "logDebug"},
       {chaiscript::fun(&ClientClass::logValue), "logValue"}
     });
  chai.add(m);

  // the (augmented) interpreter picks the script and executes it.
  // it never asked for this, by the way.
  chai.eval_file(script_name);

  return 0;
}

Time to write scripts!

Let's now write some scripts and test the whole system!

// define a simple function
def foo(n) {
  return n*2+5
}

// define a client, directly inside the script!
var client = ClientClass("tcp://127.0.0.1:6987");

// call client's methods.
client.logValue(foo(33));
client.logDebug("from chai via zmq!");
client.logDebug("hey, what happens if I send a string that is obviously longer than 18 characters? eh? EEEEHHHH???");

What does the server prints out on the console?

# the client is run as:
# LD_LIBRARY_PATH="/home/winter/built/chai/lib/chaiscript/" ./client test.chai

[winter@timeofeve] [/dev/pts/4] [master]
[~/blogging/chai-poc]> make run_server
./server
received a number!
the received number is: 71
received a string!
the received string is: >>>from chai via zmq!<<<
received a string!
the received string is: >>>hey, what happens <<<

Let's now show another script we can run.

// define a client, directly inside the script!
var client = ClientClass("tcp://127.0.0.1:6987");

client.logDebug("hello!");

for(var i=65; i<92; ++i){
    client.logValue(i);
}

client.logDebug("goodbye!");

And the result is...

# the client is run as:
# LD_LIBRARY_PATH="/home/winter/built/chai/lib/chaiscript/" ./client wow.chai

[winter@timeofeve] [/dev/pts/4]
[~/blogging/chai-poc]> make run_server
./server
received a string!
the received string is: >>>hello!<<<
received a number!
the received number is: 65
[snip]
received a number!
the received number is: 91
received a string!
the received string is: >>>goodbye!<<<

What's the magic? We were able to write two scripts in a higher-level language and use the same application to run them, without even compiling once!

Would you want to test this automatically, you may write a simple script in Python that starts the server, launches the client with the right script and then checks the content of the files generated by the server! That's just an idea, though: your needs define how you'd write your integration tests!

The nice turnout: faster experimentation and bug-hunting

It didn't took long to discover -and be favourably surprised!- that the new scripted client allows for a lot of experimentation.

In particular, I'm now working on the graphical user interface of one of the systems that receive data from the server, and it let me check edge cases in the graphs interaction, in real time.

The workflow is very simple:

  1. Have a doubt/want to test a different use case
  2. Write a simple script, changing the order of the function calls, how data are generated, etc
  3. Run the new script, with the same client(!)
  4. Check how GUI reacts and fix glitches

So, you now have both some code that can be run to verify that issues are solved. Easy and useful, isn't it?

I'll throw it in too: as the whole thing is written in C++, I can use Suzuha (with a couple of ad-hoc function calls) inside the host application. Suzuha is a simple shim for gettimeofday(3) and settimeofday(3), so I can make clients send messages from the past (or the future!) without having to recompile them.

Nice. Isn't it a bit overkill though?

Probably. I just want to give my clients something that works. Being able to anticipate possible problems and be sure that bugs found on the field won't reappear anymore is very important, and I think that thorough testing is one of the best ways to achieve it.

Do you like the idea? Do you want to suggest something better, or share your experience? Send me a tweet , or consider offering me a Ko-Fi !