HIL in software test automation toolchain

Introducing HIL in software testing.

Typical toolchain for Test-Driven Development (TDD) in software project may consist of continuous integration framework, static code analysis tool, unit test tool, and possibly acceptance test tool. In case of OSS tools in use, the chain may consists of Jenkins, cpptest, ccpunit, and Robot framework, as an example among many. That's a good toolchain for pure software development, but in case of embedded systems, it's missing the HIL aspect.

Robot Framework project defines itself as a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). Robot has gained popularity recently within my organization. The framework is written in Python, which seems to be popular language among many of our developers.

Robot is well suitable for testing web user interfaces, communication protocols, database operations, and all such cases when the behavior of the software under testing can be monitored via external interfaces of some sort, without introducing any test instrumentation into the software itself. In acceptance testing, the intention is to test the production software the way it is supposed to be used in real use cases, and that's why such an instrumentation like in case of unit testing  is not acceptable.

Robot is keyword-driven, which enables different abstractions levels, as one keyword can be defined from other keywords. In addition to Standard Test Libraries,  there are extensive set of community contributed External Test Libraries for many different purposes.When implementing test cases, one can easily combine keywords from different libraries, without need to do any actual coding. A graphical Robot IDE (RIDE) is available to make it easy to write and maintain test case sets.

Robot supports Remote Server concept similar to gdb-server.With help of Remote Server, Robot Framework may communicate with test libraries located in a different machine, or written with different language. It enables distributed testing as well. Remote server uses XML-RPC over HTTP to communicate between the Framework and Server.

Robot Framework Remote Server architecture.

Readily available Remote Servers exists for Python, Ruby, Java, .NET, Perl and more. As the XML-RPC is well documented, one can easily implement servers for other languages as well. There even exists a lightweight XML-RPC server implementation for Arduino.

Robot Remote Server concept sounds like the solution to integrate software testing with other domains, especially hardware testing, as discusses in my previous posting. National Instruments LabView is possibly the most popular solution for hardware testing. Implementing XML-RPC Server in LabView and integrating it with Test Library in LabView makes it possible to communicate with hardware test routines from software test cases.

Disclaimer:
Personally I'm not a big fan of XML. XML-RPC consumes 4 times more characters compared to plain XML, which by itself is a bloat compared to JSON for example. XML-RPC is fixed to be transferred over HTTP, which is not the most efficient method for two-way communication. JSON is not tied to any specific carrier protocol, WebSocket is possibly the most common and obvious choice. WebSocket by itself is more efficient for back and forth data transfer than HTTP. I hope one day someone will introduce JSON/WebSocket implementation to substitute XML-RPC.