Guide for Writing Rbq-Lua Unit Tests

last updated Sat Jun 5 2004

Purpose of Unit Tests

Unit tests are an important component in the development of software, especially for large and complex software projects. The purpose of a unit test is to verify the functionality of a small piece of the overall software. Often, such a piece can be a single function. A unit test will call the function in a variety of ways, with both valid and invalid parameters, and check that the function returns the expected result. After a comprehensive set of unit tests has been developed to check the behavior of the components of a software program, these tests can be run on a regular basis. If ongoing change in the code breaks something or introduces bugs, the unit tests can expose such problems in an automatic and comprehensive manner. This is in contrast to a human software developer, who typically does brief, limited and inconsistent checks of the program after making code changes.

Unit tests will not catch all bugs. Many bugs are exposed only through the combined action of multiple components, and must be found in other ways. Still, unit tests are a good baseline for maintaining program stability.

The Rbq-Lua API

The Rbq-Lua API is the Application Programming Interface between the core application code, written in C++, and high-level scripts, written in Lua.

The core application code provides a number of operations that a user can perform, such as loading models into the program, retrieving data about those models, controlling the visual display, and so on. Such operations can be performed by clicking the menus, toolbars and typing commands into the graphical user interface. Operations like these can also be performed by executing scripting code that is contained in text files, where the scripts are written in the Lua scripting language.

In Lua scripts, the user can define global and local variables, expressions, if statements, for loops, and functions. The embedded Lua interpretor also provides a basic set of library functions to do string manipulation, mathematical calculation and file i/o. However, to interact with the Ribosome Builder application, a Lua script can only do so by calling a special set of functions that have been registered by the core application. These functions are all written in C and are defined in a single C++ file called CRbqLua.cpp. They all begin with the prefix 'rbq_'. Inside these functions, parameters are converted from the Lua scripting context into C numeric and string data types. Then, after the result of the function has been retrieved by calling C++ functions in the application core code, the return values are converted into Lua data types and passed back, via the script interpretor, to the scripting code.

HTML documentation for the Rbq-Lua API is automatically generated from comment header blocks that precede these functions. This documentation describes the purpose of each function, the number and type of parameters and return values, and sometimes additional notes and cross-referencing to related functions. The documentation organizes the functions into different categories. Occaisionally, a function will appear in more than one category because its purpose can not be fully localized to a single category.

The Lua High-Level Libraries

The Lua Libraries are modules of code written in Lua that are built on top of the Rbq-Lua interface functions. Each module is a file, located in the 'lua' subdir, beginning with an underscore '_'. The modules are organized by category. For example, the _PdbUtil.lua file contains functions specific to PDB data. In addition to each Lua library module file, there is another file with the same name, but lacking the underscore, which serves as an 'include' file to load the module as needed.

The functions in the Lua Libraries also have HTML documentation that is automatically generated from comment header blocks that precede the functions. Some of the functions are simple wrappers around the rbq_XXX versions that are implemented in C, but many other functions implement higher-level operations in a variety of categories. These Lua library functions can also be used in the writing the Rbq-Lua unit tests to provide support operations such as opening documents, building paths, and unit-test specific operations, as provided by the _TestUtil.lua module, discussed in more detail below.

At some point, unit tests for the functions of the Lua Libraries should also be created.

Location of Rbq-Lua Unit Tests in the Source Tree

The source code for the project is organized in a hierarchical manner. The files for the basic portion of the program are in the 'bas' module in CVS. Inside of this bas directory, the source code is located in 2 subdirectories, 'app' and 'lib'. Inside of the 'app' subdirectory are 2 subdirectories 'rbq' and 'htsrv', for the two stand-alone executable programs that are currently part of the project. Within the 'rbq' directory, the Rbq-Lua tests are located in the 'tst/rbqlua' subdirectory. So, from the top-level, the full path to the directory for Rbq-Lua unit tests is 'bas/app/rbq/tst/rbqlua'.

Within this directory, there is a 'funcs' subdirectory. In the funcs subdirectory, the unit tests for each rbq_XXX function are located in subdirectories with the same name as the function. For an 'immediate' unit test of a function, a file called 'test.lua' is created within the function directory. For example, the 'immediate' unit test for the function 'rbq_InsPdbFile()' would have the full path 'bas/app/rbq/tst/rbqlua/funcs/rbqInsPdbFile/test.lua'.

Two Types of Unit Tests: 'Immediate' and 'Tick'

There are two types of Rbq-Lua unit tests. The first type is called 'immediate', because it refers to a test script that can check the desired behaviour of a function within a single instance of executing code. This means that the script is run, the test is done, and then execution of the scripting code terminates, returning to the caller.

This execution model is sufficient for testing some but not all of the functions in the Rbq-Lua API. There are functions which cannot be properly tested in such a 'single-pass' execution model. For example, some functions can initiate changes in the graphics display which are performed only through a sequence of queued events in the Qt windowing system. This requires that the testing script terminate execution and return to the caller in order for the desired behaviour to subsequently appear.

To test such delayed behaviour, a second type of unit test has been defined. This type is called 'tick', because it makes use of the 'tick' event of script objects to implement the unit test. The design of a 'tick' unit test is a little more complicated than that of an 'immediate' unit test. For example, the unit test for the function rbq_DrawOrthoText() is tickTest.lua. Because it defines script object events, it also consists of two additional files, tickTest.tick.lua and tickTest.draw.lua.

Running Multiple Unit Tests with the Test Driver

All the Rbq-Lua tests that are currently defined can be run in succession by executing a test driver script, 'runAllTests.lua', located in the 'bas/app/rbq/tst/rbqlua' directory. For greater convenience, this script can be run by choosing a menu item from the application, located in the 'Script', 'Other', 'Testing' menu.

In addition, the driver script will create two files, a test result file and a test log file, that will receive data from each unit script as it is run. The result file simply holds the PASS/FAIL status of each test. The log file can hold more detailed information, depending upon what the implementer of a unit test script has decided to log.

In addition to running all unit tests, it is sometimes convienient to just run a subset of tests, such as the test you have just written. This can also be done from a menu item in the same submenu mentioned above. The additional requirement is that the desired subset of tests to be run must be specified in the file 'bas/app/rbq/tst/rbqlua/subset.testNames'.

An Example Unit Test

To give a better idea how Rbq-Lua unit tests work, the structure of an existing test is briefly discussed. The rbq_InsPdbFile unit test verifies the function that inserts a PDB file into the current document. The test consists of a main section and 5 functions.

The main section at the bottom includes three Lua Library modules that define some functions used by the script:

	load( "FileUtil.lua" );
	load( "PdbUtil.lua" );
	load( "TestUtil.lua" );

Then it defines a table that holds references to the 5 functions above, each of which tests a particular scenario involving the Rbq-Lua function:

	local tCases =

The next line calls a function in the TestUtil.lua module to do some setup work at the start of the unit test. This setup work currently just clears the 'failed' flag and logs a test start message to the test logfile, but it is also a placeholder for future intialization:

	testBegin( g_sTestName );

The next section is just a for loop to call each function to do a particular test case:

	local iCase;
	for iCase = 1, getn( tCases ) do
		local tCase = tCases[iCase];
		if not tCase() then
			do break; end

Finally, a cleanup function is called, which outputs the test PASS/FAIL status to the test result file:

	testEnd( g_sTestName );

Test Cases

Test Cases are used within a particular unit test to check the function behaviour with particular inputs. Typically, for a function that requires at least one parameter, a test case might call the function with zero parameters and verify that the function does not crash, and returns a fail value. Another test case may call the function with valid parameters and check that a valid result returned.

The number of possible test cases and the code within a particular test case function will vary, depending upon the function being tested. However, some conventions currently exist:

The TestUtil Library

As mentioned above, the TestUtil library contains functions to support the running of unit tests. It defines the common initialization and cleanup functions used at the start and end of each test. It also provides a general-purpose testLog() function to log extra information as needed during the running of a test. As other functions are created over time to support commonly used operations for testing, they will be placed in this library module.

In a script, before calling a function in the TestUtil library, the library must be 'included' by with a 'load' statement:

	load( "TestUtil.lua" );

Such 'load' statements must also be done when using functions in any other application Lua Library module.

Zero-based Indices and One-based Indices

The Lua language has a clean and simple syntax that is intended to appeal to the non-professional programmer, which makes it ideal as a scripting language for users of molecular biology applications. However, one consequence is that there is an unfortunate inconsistency in the use of numerical indices at the Rbq-Lua interface.

In the Lua language, indices are one-based. For example, the first element in a Lua table is accessed with the index '1'. The Lua built-in library functions such as strsub() also refer to the first character in a string as starting at the '1' position. In contrast, in the C and C++ code of the core application, all offsets begin at position '0'. As a result, there will be some mixing of 0-based and 1-based indices in the Rbq-Lua API code.

The rule for determining 1-based versus 0-based indices is as follows: when calling rbq_XXX functions, either directly or indirectly, 0-based parameters will be used. When calling built-in Lua functions, either directly or indirectly, 1-based parameters will be used. If there is any question as to which is the case, the documentation should clearly state whether the parameters and return values are 0-based or 1-based.

Creating a Known Testing Context

Unit tests should be capable of being run individually and in sequence. As a result, a unit test should not make any assumptions about the state of the program when the unit test is begun. Typically, a unit test will define some initial setup code to establish a known state in which to conduct the tests.

For example, unit tests that load and check the presence of a pdb model may first close any current document and create a new, empty document. A unit test that checks the result of a graphical function will first set the graphics window to be a specified size, so that the graphic snapshot will match a reference file. The helper script 'rlutSetGraphicWin.lua' can be used for setting the graphic window state.


A common test case is to call an rbq-lua function with invalid parameters and verify that the function fails. The default behaviour in such cases is for the 'lua_error()' function to be called, resulting in termination of script execution and display of a call stack trace. However, while this behaviour is desireable for normal development of scripts, it would interfere with the running of unit test scripts.

The calling of 'lua_error()' is temporarily disabled through a call to the 'enableLuaError()' function during unit test cases that call rbq-lua functions with invalid parameters.