Test::Nginx has built-in support for performance testing or benchmarking. It can invoke external load testing tools like ab and weighttp to load each test case as hard as possible.

To enable this benchmark testing mode, you can specify the TEST_NGINX_BENCHMARK system environment variable before running the prove command. For example,

This will run all the test cases in t/foo.t in benchmark mode. In particular, the first number, 2000 in the environment variable value indicates the total number of requests used to flood the server while the second number, 2, means that the number of concurrent connections the client will use.

If the test case uses an HTTP 1.1 request (which is the default), then the test scaffold will invoke the weighttp tool. If it is an HTTP 1.0 request, then the test scaffold invokes the ab tool.

This test mode requires the unbuffer command-line utility from the expect package, as well as the ab and weighttp load testing tools. On Ubuntu/Debian systems, we can install most of the dependencies with the command

You may need to build and install weighttp from source on Ubuntu/Debian yourself due to the lack of the Debian package.

For the Mac OS X system, on the other hand, we can use homebrew to install it like this:

Now let’s consider the following example.

Then we run this test file in the benchmark mode, like this:

The output should look like this:

The most important line in this:

We can see that this test case can achieve 75393 requests per second and 12218 KB per second. Not bad for a single NGINX worker process!

It is also important to keep an eye on failed requests. We surely do not care about the performance of error pages. We can get the number of error responses by checking the following output lines:

We are glad to see that all our requests succeeded in this run.

If we want to benchmark the performance of multiple NGINX worker processes so as to utilize multiple CPU cores, then we can add the following lines to the test file prologue, before the line run_tests():

This way we can have 4 NGINX worker processes sharing the load.

Behind the scenes, the test scaffold assembles the command line involving weighttp from the test block specification, in this case, the command line looks like this:

There exists complicated cases, however, where the test scaffold fails to derive the exact command line equivalent.

We can also enforce HTTP 1.0 requests in our test block by appending the "HTTP/1.0" string to the value of the --- request section:

In this case, the test scaffold will invoke the ab tool to flood the matching HTTP 1.0 request. The output might look like this:

The most important output lines, in this case, are

Different hardware and operating systems may lead to very different results. Therefore, it generally does not make sense at all to directly compare numbers obtained from different machines and systems.

Clever users can write some external scripts to record and compare these numbers across different runs, so as to keep track of performance changes in the web server or application. Such comparison scripts must take into account any measurement errors and any disturbances from other processes running in the same system.

Performance benchmark is a large topic and we gives it a more detailed treatment in a dedicated chapter.

By default, the test scaffold always starts a fresh instance of the NGINX server right before running each individual test block and stops the server right after the checks of the current test block are all done. This ensures that there is no side effects among test blocks, especially those running successively. But it can also be desired to ensure everything also works fine when the NGINX server is just reloading its configurations without a full server restart. Such configuration reloading is usually done via sending the HUP signal to the master process of NGINX. So we usually call it "HUP reload".

One example of OpenResty features that behaves different upon HUP reload than server restart is the shared dictionary mechanism (lua_shared_dict) that does not wipe out any existing data in the shared memory storage during HUP reload. When testing this feature or application code relying on this feature, it is wise to test how it behaves upon HUP reload. We saw in the past that some 3rd-party NGINX C modules dealing with shared memory, for example, have bugs across HUP reloads, like nasty memory leaks.

Test::Nginx has built-in support for the HUP reload test mode, which can be enabled by specifying the TEST_NGINX_USE_HUP=1 environment:

Then we can run our existing test suite as usual but now HUP signal is used by the test scaffold to reload the NGINX configuration specified by different test blocks. The NGINX server will only be automatically shut down when the test harness finishes running each test file.

UNIX signals like HUP usually work asynchronously. Thus, there is a delay between the test scaffold finishes sending the HUP signal to the NGINX server and the NGINX server forks off a new worker process using the newly loaded configuration and starts accepting new connections with the new worker. For this reason, there is a (small) chance that the request of a test block is served by an NGINX worker process still using the configuration specified by the previous test block. Although Test::Nginx tries hard to wait as long as it can with some simple heuristics, some test blocks may still experience some intermittent test failures due to the mismatch of the NGINX configuration. Be prepared for such false positives when using the HUP reload testing mode. This is also one of the reasons why the HUP reload mode is not the default. We hope this issue can be further improved in the future.

Another limitation with the HUP reload mode is that HUP reloads only happen upon test block boundaries. There are cases where it is desired to issue HUP reload in the middle of a test block. We can achieve that by using some custom Lua code in your test block to send a HUP signal yourself, as in

One of the biggest enemies in web servers or web applications that are supposed to run in a 24x7 manner is memory issues. Memory issues include memory leaks, memory invalid reads (like reading beyond the buffer boundary), and memory invalid writes (like buffer overflow). In case of memory leaks, the processes can take up more and more memory in the system and eventually exhaust all the physical memory available, leading to unresponsive systems or triggering the system to start killing processes with force. Memory invalid accesses, on the other hand, can lead to process crashes (like segmentation faults), or worse, leading to nondeterminism in the process' s behavior (like giving out wrong results).

Valgrind is a powerful tool for programmers to detect a wide range of memory issues, including many memory leaks and many memory invalid accesses. This is usually for debugging lower level code like the OpenResty core (including the NGINX core), the Lua or LuaJIT VM, as well as those Lua libraries involved with C and/or FFI. Plain Lua code without using FFI is considered "safe" and is not subject to most of the memory issues.

Test::Nginx provides a testing mode that can automatically use Valgrind to run the existing tests and check if there is any memory issues that can be caught by Valgrind. This test mode is called "Valgrind mode". To enable this mode, just set the environment TEST_NGINX_USE_VALGRIND, as in

Then just run the test files as usual.

Let’s consider the following example.

Here we use the ffi.new API to allocate a C string buffer of 3 bytes long and initialize the buffer with the bytes 48, 49, and 0, in the decimal ASCII code. Then we call the standard C function strlen via the ffi.C API with our C string buffer.

It is worth noting that we need to first declare the strlen function prototype via the ffi.cdef API. Since we declare the C function in the request handler (content_by_lua_block), we should only declare it once instead of upon every request. To achieve that, we use a Lua if statement to check if the symbol strlen is already declared (when strlen is not declared or defined, the Lua expression C.strlen would throw out a Lua exception, which can make the pcall call fail).

This example contains no memory issues since we properly initialize our C string buffer by setting the null terminator character (\0) at end of our C string. The C function strlen should correctly report back the length of the string, which is 2, without reading beyond our buffer boundary. Now we run this test file with the Valgrind mode enabled using the default OpenResty installation’s nginx:

There should be a lot of output. The first few lines should look like this:

Ouch! Valgrind reports a memory invalid read error. Fortunately it is just a false positive due to the optimization inside the LuaJIT VM when it is trying to create a new Lua string. The LuaJIT code repository maintains a file named lj.supp that lists all the known Valgrind false positives that can be used to suppress these messages. We can simply copy that file over and rename it to valgrind.suppress in the current working directory. Then Test::Nginx will automatically feed this valgrind.suppress file into Valgrind while running the tests in Valgrind mode. Let’s try that:

This time, the test scaffold is calmed:

We might encounter other Valgrind false positives like some of those in the NGINX core or the OpenSSL library. We can add those to the valgrind.suppress file as needed. The Test::Nginx test scaffold always outputs suppression rules that can be added directly to the suppression file. For the example above, the last few lines of the output are like below.

The suppression rule generated is the stuff between the curly braces (including the curly braces themselves):

We could have simply copied and pasted this rule into the valgrind.suppress file. It is worth mentioning however, we can make this rule more general to exclude the C function frames belonging to the NGINX core and the ngx_lua module (near the bottom of the rule) since this false positive is related to LuaJIT only.

Let’s continue our experiment with our current example. Now we edit our test case and change the following line

to

That is, we replace the null character (with ASCII code 0) to a non-null character whose ASCII code is 50. This change makes our C string buffer lacks any null terminators and thus calling strlen on it will result in memory reads beyond our buffer boundary.

Unfortunately running this edited test file fail to yield any Valgrind error reports regarding this memory issue:

The response body check fails as expected. This time strlen returns 4, which is larger than our buffer size, 3. This is a clear indication of memory buffer over-read. So why does Valgrind fail to catch this?

To answer this question, we need some knowledge about how LuaJIT allocates memory. By default, LuaJIT uses its own memory allocator atop the system allocator (usually provided by the standard C library). For performance reasons, LuaJIT pre-allocates large memory blocks than request. Because Valgrind has no knowledge about LuaJIT’s own allocator and Lua user-level buffer boundary definitions, it can be cheated and can get confused.

To remove this limitation, we can enforce LuaJIT to use the system allocator instead of its own. To achieve this, we need build LuaJIT with special compilation options like below.

The most important option is -DLUAJIT_USE_SYSMALLOC which forces LuaJIT to use the system allocator. The other options are important for our debugging as well, for example, the CCDEBUG=-g option is to enable debug symbols in the LuaJIT binary while -DLUAJIT_USE_VALGRIND enables some other special collaborations with Valgrind inside the LuaJIT VM.

If we are using the OpenResty bundle, we can simply build another special version of OpenResty like below:

This will build and install a special debug version of OpenResty for Valgrind checks to the file system location /opt/openresty-valgrind.

Now let’s try running our previous buggy example with this special OpenResty and Valgrind:

This time Valgrind succeeds in catching the memory bug!

We omit the rest of the output for brevity. Here Valgrind reports an invalid read of one byte of memory in the C function strlen, which is exactly what we’d expect. Mission accomplished!

From this example, we can see how application-level memory allocation optimizations and management can compromise the effectiveness of Valgrind’s memory issue detection. Similarly, the NGINX core also comes with its own memory allocator via "memory pools". Such memory pools tend to allocate page-sized memory blocks for small allocations and thus can also inversely affect Valgrind' s detection. OpenResty provides a patch for the NGINX core to disable the memory pool optimizations altogether. The easiest way to use the patch is to specify the --with-no-pool-patch option when running the ./configure script while building OpenResty.

This Valgrind mode is used by OpenResty developers on a daily basis and has helped locate countless memory manage bugs in the OpenResty C and Lua/FFI code base. Interestingly, this test mode also located memory issues in the official NGINX core and the official LuaJIT core. Unlike analyzing core dumps, Valgrind can almost always find the first scene of memory offends, studying the memory error reports can usually give rise to immediate code fixes.

As with all the other tools, Valgrind has its own limitations and cannot find all the memory issues even when we carefully disable application level memory allocators as demonstrated above. For example,

Understanding the weakness of the tool is as important as understanding its strengths. We shall see an alternative approach in the next section for detecting leaks in the application-level memory managers.

As we have seen from the previous section, Valgrind is great at detecting a wide range of memory leaks and memory invalid accesses. But Valgrind also suffers from limitations in detecting leaks in application-level memory managers such as garbage collectors (GC) and memory pools, which is also quite common in reality. To see this, let’s consider the following simple example that leaks in LuaJIT’s GC-managed memory.

This example demonstrates a common mistake made by many OpenResty beginners. The package.path field specifies the search paths used by the require builtin function for loading pure Lua modules. This string value is hooked up in the global Lua table package which has the same lifetime as the current Lua virtual machine (VM) instance. Since Lua VM instances usually have the same lifetime as NGINX worker processes (unless the lua_code_cache directive is turned off in nginx.conf), prepending a new string to the value of package.path in a request handler like content_by_lua_block apparently results in a memory leak.

Unfortunately Valgrind cannot find this leak at all since the leak happens in the GC-managed memory inside the Lua VM because all such leaked memory will always get released upon GC destruction (or VM destruction) before the current process exits, which fools Valgrind to think that there is no leaks at all. Interested readers can try running this example with the "Valgrind test mode" as explained in the previous section.

To address this limitation of Valgrind, Test::Nginx::Socket introduces a new test mode called "naive memory leak check mode", or just "check leak mode" for short. In this mode, the test scaffold performs the following things:

To make use of this mode, just specify the TEST_NGINX_CHECK_LEAK=1 environment, before running existing test files, as in

Assuming the t/a.t test file contains the test block example given above, we should get an output similar to the following.

The special output lines from this test mode have the prefix LeakTest:. The first such line lists all the data points for the memory footprint size in the unit of kilo bytes (KB), collected every 0.02 seconds. And the second line is the slope (k) of the data line that best fits these data points. And in this case, k equals to 22.6.

The slope of the line can usually serve as an indication for the speed of memory leaking. The larger the slope is, the faster the leak is. A 2-digit data line slope here is very likely an indication of memory leak. To be sure, we plot these data points in a graph using the gnuplot tool.

There are quite some fluctuations in the graph. This is due to how garbage collector normally behaves. It usually allocates page-sized or even larger memory blocks than actually requested for performance reasons and delays the release of unused memory blocks because of the sweep phase or something else. Still, it is clear that the memory usage is going up over all.

We can try enforcing a full garbage collection cycle upon the entry of our request handler, like this:

This way we can ensure that there is no memory garbage hanging around after the point we call the Lua builtin function collectgarbage().

Now the output looks like this:

We can see this time, the slope of the best-fitting line is much smaller, but still much larger than 0.

The line graph is now much smoother, as expected:

And we can see that the line is still going upward relatively steadily over time.

Large fluctuations and variations in the memory footprint may create noises in our data samples and even result in false positives. We already saw how big fluctuations may result in large data-fitting line slopes. It is usually a good idea to enforce full garbage collection cycles frequently to reduce such noises at least in GC-managed memory. The collectgarbage() function, however, is quite expensive in terms of CPU resources and may hurt the over-all performance very badly. Ensure you do not call it often (like in every request) in the "benchmark test mode" introduced above or even in production applications.

In reality, this brute-force "check leak" test mode has helped catching quite a lot of real memory leaks in OpenResty’s test suites over the years. Most of those leaks made their way around the Valgrind test mode since they happened in GC-managed memory or NGINX’s memory pools.

Nevertheless, there exists one big drawback of this test mode. Unlike Valgrind, it cannot give any detailed information about the locations where leaks (may) happen. All it reports are just data samples and other metrics that verify just the existence of a leak (at least to some extend). We shall see in a later chapter how we can use the "memory leak flame graphs" to overcome this limitation even for leaks and big swings in GC-managed or pool-managed memory.