=== TEST 1:
--- main_config
env MY_ENVIRONMENT;
--- http_config
init_worker_by_lua_block {
print("init")
}
--- config
location = /t {
echo ok;
}
--- request
GET /t
--- response_body
okPreparing Tests
As we have seen in the previous sections, Test::Nginx provides a simple
declarative format to express test cases. Each test case is represented
by a test block. A test block consists of a title, an optional description,
and several data sections for specifying inputs and expected outputs. In
this section, we will have a close look at how to prepare such test cases
for different test requirements.
Designing test cases is an art, in many ways. It may, sometimes, take even
more time and effort than implementing the feature to be tested, according
to our own experience. Test::Nginx tries hard to make writing tests as
simple as possible but it still cannot automate the whole test case design
process. Only you know exactly what to test and how it can be tested anyway.
This section will focus on the basic primitives provided by Test::Nginx
that you can take advantage of to devise clever and effective test cases.
Preparing NGINX Configuration
In a test block, we can use different data sections to specify our custom
snippets in different positions of the final nginx.conf configuration
file generated by Test::Nginx.
The most common one is the config section which is used to insert custom
snippets inside the server {} configuration block for the default test
server. We can also use the http_config section to insert our custom
content into the http {} configuration block of nginx.conf. The main_config
section can be used to insert content into the top-level scope of the NGINX
configuration. Let’s consider the following example.
This test block will generate an nginx.conf file with the following basic
structure:
...
env MY_ENVIRONMENT;
http {
...
init_worker_by_lua_block {
print("init")
}
server {
...
location = /t {
echo ok;
}
}
}Please pay attention to how the main_config, http_config, and config
data sections' values are mapped into different locations in the NGINX
configuration file.
When in doubt, we can always check out the actual nginx.conf file generated
by the test scaffold at the location t/servroot/conf/nginx.conf in the
current working directory (usually just being the root directory of the
current project).
Test::Nginx generates a new nginx.conf file for each test block, which
makes it possible for each test block to become self-contained. By default,
the test scaffold automatically starts a new NGINX server before running
each test block and shuts down the server immediately after running the
block. Fortunately, NGINX is a lightweight server and it is usually very
fast to start and stop. Thus, the test blocks are not that slow to run
as it might look.
Preparing Requests
The simplest way to prepare a request is to use the request data section,
as in
--- request
GET /t?a=1&b=2The HTTP/1.1 protocol is used by default. You can explicitly make it use the HTTP/1.0 protocol if desired:
--- request
GET /t?a=1&b=2 HTTP/1.0Leading spaces or empty lines in the value of the request section are
automatically discarded. You can even add comments by leading them with
a # character, as in
--- request
# this is a simple test:
GET /tYou can add some additional request headers at the same time through the
more_headers section as below.
--- request
GET /t
--- more_headers
Foo: bar
Bar: bazPipelined Requests
Preparing pipelined HTTP requests are also possible. But you need to use
the pipelined_requests section instead of request. For instance,
=== TEST 1: pipelined requests
--- config
location = /t {
echo ok;
}
--- pipelined_requests eval
["GET /t", "GET /t"]
--- response_body eval
["ok\n", "ok\n"]It is worth noting that we use the eval filter with the pipelined_requests
section to treat the literal value of that section as Perl code. This way
we can construct a Perl array of the request strings, which is the expected
data format for the pipelined_requests section. Similarly we need a similar
trick for the response_body section when checking outputs. With an array
of expected response body data, we can expect and check different values
for different individual request in the pipeline. Note, however, not every
data section supports the same array-typed value semantics as response_body.
Checking Responses
We have already visited the response_body and error_code data sections
for checking the response body data and response status code, respectively.
The response_body data section always performs an exact whole-string
comparison between the section value and the actual response body. It tries
to be clever when long string value comparison fails. Consider the following
sample output from prove.
t/foo.t .. 1/?
# Failed test 'TEST 1: long string test - response_body - response is expected (req 0)'
# at .../test-nginx/lib/Test/Nginx/Socket.pm line 1282.
# got: ..."IT 2.x is enabled.\x{0a}\x{0a}"...
# length: 409
# expected: ..."IT 2.x is not enabled.\x{0a}"...
# length: 412
# strings begin to differ at char 400 (line 1 column 400)
# Looks like you failed 1 test of 2.
/tmp/foo.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
Test Summary Report
-------------------
/tmp/foo.t (Wstat: 256 Tests: 2 Failed: 1)
Failed test: 2
Non-zero exit status: 1
Files=1, Tests=2, 0 wallclock secs (0.01 usr 0.00 sys + 0.09 cusr 0.03 csys = 0.13 CPU)
Result: FAIL
From this test report, we can clearly see that
-
it is the test block with the title
TEST 1: long string testthat is failing, -
it is the
response_bodydata section check that fails, -
the actual response body data is 409 bytes long while the expected value is 412 bytes, and
-
the expected value has an additional
notword in the string fragmentIT 2.x is enabledand the difference starts at the offset 400 in the long string.
Behind the scene, Test::Nginx uses the Perl module Test::LongString
to do the long string comparisons. It is also particularly useful while
checking response body data in binary formats.
If your response body data is in a multi-line textual format, then you
may also want to use a diff-style output when the data does not match.
To achieve this, we can call the no_long_string() Perl function before
the run_tests() function call in the prologue part of the test file.
Below is such an example.
use Test::Nginx::Socket 'no_plan';
no_long_string();
run_tests();
__DATA__
=== TEST 1:
--- config
location = /t {
echo "Life is short.";
echo "Moon is bright.";
echo "Sun is shining.";
}
--- request
GET /t
--- response_body
Life is short.
Moon is deem.
Sun is shining.Note the no_long_string() call in the prologue part. It is important
to place it before the run_tests() call otherwise it would be too late
for it to take effect, obviously.
Invoking the prove utility (or any shell wrappers for it) to run this
test file gives the following details about the test failure:
# Failed test 'TEST 1: - response_body - response is expected (req 0)' # at .../test-nginx/lib/Test/Nginx/Socket.pm line 1277. # @@ -1,3 +1,3 @@ # Life is short. # -Moon is deem. # +Moon is bright. # Sun is shining. # Looks like you failed 1 test of 2.
It is obvious that the second line of the response body output is different.
You can even further disable the diff-style comparison mode by adding
a no_diff() Perl function call in the prologue part. Then the failure
report will look like this:
# Failed test 'TEST 1: - response_body - response is expected (req 0)' # at .../test-nginx/lib/Test/Nginx/Socket.pm line 1277. # got: 'Life is short. # Moon is bright. # Sun is shining. # ' # expected: 'Life is short. # Moon is deem. # Sun is shining. # ' # Looks like you failed 1 test of 2.
That is, Test::Nginx just gives full listing of the actual response body
data and the expected one without any abbreviations or hand-holding.
Pattern Matching on Response Bodies
When the request body may change in some ways or you just care about certain
key words in a long data string, you can specify a Perl regular expression
to do a pattern match against the actual request body data. This is achieved
by the response_body_like data section. For example,
--- response_body_like: age: \d+Be careful when you are using the multi-line data section value form. A
trailing newline character appended to your section value may make your
pattern never match. In this case the chomp filter we introduced in an
early section can be very helpful here. For example,
--- response_body_like chomp
age: \d+You can also use the eval filter to construct a Perl regular expression
object with a Perl expression, as in
--- response_body_like eval
qr/age: \d+/This is the most flexible form to specify a pattern.
|
Note
|
Perl uses the qr quoting structure to explicitly construct regular
expression objects. You can use various different quoting forms like qr/…/,
qr!…!, qr#…#, and qr{…}.
|
Checking Response Headers
The response_headers data section can be used to validate response header
entries. For example,
--- response_headers
Foo: bar
Bar: baz
!BlahThis section dictates 3 tests actually:
-
The response header
Foomust appear and must take the valuebar; -
The response header
Barmust appear and must take the valuebaz; and -
The response header
Blahmust not appear or take an empty value.
Checking NGINX Error Logs
In addition to responses, the NGINX error log file is also an important output channel for an NGINX server setup.
True-False Tests
One immediate testing requirement is to check whether or not a piece of
text appears in any error log messages. Such checks can be done via the
data sections error_log and no_error_log, respectively. The former
ensures that some lines in the error log file contain the string specified
as the section value while the latter tests the opposite: ensuring that
no line contains the pattern.
For example,
--- error_log
Hello world from my serverThen the string Hello world from my server (without the trailing new-line)
must appear in at least one line of the NGINX error log. You can specify
multiple strings in separate lines of the section value to perform different
checks, for instance,
--- error_log
This is a dog!
Is it a cat?Then it performs two error log checks, one is to ensure that the string
This is a dog! appears in some error log lines. The order of these two
string patterns do not matter at all.
If one of the string pattern failed to match any lines in the error log
file, then we would get a test failure report from prove like below.
# Failed test 'TEST 1: simple test - pattern "This is a dog!" matches a line in error.log (req 0)'
If you want to specify a Perl regular expression (regex) as one of the
patterns, then you should use the eval section filter to construct a
Perl-array as the section value, as in
--- error_log eval
[
"This is a dog!",
qr/\w+ is a cat\?/,
]As we have seen earlier, Perl regexes can be constructed via the qr/…/
quoting syntax. Perl string patterns in the Perl array specified by double
quotes or single quotes are still treated as plain string patterns, as
usual. If the array contains only one regex pattern, then you can omit
the array itself, as in
--- error_log eval
qr/\w+ is a cat\?/Test::Nginx puts the error log file of the test NGINX server in the file
path t/servroot/logs/error.log. As a test writer, we frequently check
out this file directly when things go wrong. For example, it is common
to make mistakes or typos in the patterns we specify for the error_log
section. Also, scanning the raw log file can give us insight about the
details of the NGINX internal working when the NGINX debugging logs are
enabled in the NGINX build.
The no_error_log section is very similar to error_log but it checks
the nonexistence of the string patterns in the NGINX error log file. One
of the most frequent uses of the no_error_log section is to ensure that
there is no error level messages in the log file.
--- no_error_log
[error]If, however, there is a line in the nginx error log file that contains
the string [error], then the test fails. Below is such an example.
# Failed test 'TEST 1: simple test - pattern "[error]" should not match any line in error.log but matches line "2016/02/01 11:59:50 [error] 1788\#0: *1 lua entry thread aborted: runtime error: content_by_lua(nginx.conf:42):2: bad"'
This is a great way to find the details of the error quickly by just looking at the test report.
Like error_log, this section also supports Perl array values and Perl
regex values through the eval filter.
Grep Tests
The error_log and no_error_log sections are very handy in quickly checking
the appearance of contain patterns in the NGINX error log file. But they
have serious limitations in that it is impossible to impose stronger constraints
on the relative order of the messages containing the patterns nor on the
number of their occurrences.
To address such limitations, Test::Nginx::Socket provides an alternative
way to check NGINX error logs in a way similar to the famous UNIX tool,
grep. The sections grep_error_log and grep_error_log_out are used
for this purpose. The test writer uses the grep_error_log section to
specify a pattern, with which the test framework scans through the NGINX
error log file and collect all the matched parts of the log file lines
along the way, forming a final result. This aggregated log data result
is then matched against the expected value specified as the value of the
grep_error_log_out section, in a similar way as with the response_body
section discussed above.
It is easiest to explain with a simple example.
=== TEST 1: simple grep test for error logs
--- config
location = /t {
content_by_lua_block {
print("it is matched!")
print("it is matched!")
print("it is matched!")
}
}
--- request
GET /t
--- grep_error_log: it is matched!
--- grep_error_log_out
it is matched!
it is matched!
it is matched!Here we use the Lua function print() provided by the
ngx_http_lua
module to generate NGINX error log messages at the notice level. This
test case tests the number of the log messages containing the string it
is matched!. It is important to note that only the matched part of the
log file lines are collected in the final result instead of the whole log
lines. This simplifies the comparison a lot since NGINX error log messages
can contain varying details like timestamps and connection numbers.
A more useful form of this test is to specify a Perl regex pattern in the
grep_error_log section. Consider the following example.
=== TEST 1: simple grep test for error logs
--- config
location = /t {
content_by_lua_block {
print("test: before sleeping...")
ngx.sleep(0.001) -- sleeping for 1ms
print("test: after sleeping...")
}
}
--- request
GET /t
--- grep_error_log eval: qr/test: .*?\.\.\./
--- grep_error_log_out
test: before sleeping...
test: after sleeping...We specify a Perl regex pattern, test: .*?\.\.\., here to filter out
all the error log messages starting with test: and ending with ….
And naturally in this test we also require the relative order of these
two messages, that is, before sleeping must appear before after sleeping.
Otherwise, we shall see failure reports like below:
# Failed test 'TEST 1: simple grep test for error logs - grep_error_log_out (req 0)'
# at ..../lib/Test/Nginx/Socket.pm line 1048.
# got: "test: after sleeping...\x{0a}test: before sleeping...\x{0a}"
# length: 49
# expected: "test: before sleeping...\x{0a}test: after sleeping...\x{0a}"
# length: 49
# strings begin to differ at char 7 (line 1 column 7)
As with the response_body section, we can also call the no_long_string()
Perl function before run_tests() in the test file prologue, so as to
disable the long string output mode and enable the diff mode. Then the
test failure would look like this:
# Failed test 'TEST 1: simple grep test for error logs - grep_error_log_out (req 0)' # at .../lib/Test/Nginx/Socket.pm line 1044. # @@ -1,2 +1,2 @@ # -test: before sleeping... # test: after sleeping... # +test: before sleeping...
Obviously, for this test case, the diff format looks better.
Extra Delay Before Log Checks
By default, Test::Nginx::Socket performs the NGINX error log checks not
long after it receives the complete HTTP response for the test request.
Sometimes, when the log messages are generated by the server after sending
out the response, the error log checks may be carried out too early
that the messages are not yet written into the log file. In this case,
we can specify an extra delay via the wait data section for the test
scaffold to wait for the error log messages. Here is an example:
=== TEST 1: wait for the timer
--- config
location = /t {
content_by_lua_block {
local function f(premature)
print("HERE!")
end
assert(ngx.timer.at(0.1, f))
}
}
--- request
GET /t
--- error_log
HERE!
--- no_error_log
[error]
--- wait: 0.12Here we create a timer via the ngx.timer.at Lua function, which expires
after 0.1 seconds. Due to the asynchronous nature of timers, the request
handler does not wait for the timer to expire and immediately finishes
processing the current request and sends out a response with an empty body.
To check for the log message HERE! generated by the timer handler f,
we have to specify an extra delay for the test scaffold to wait. The 0.12
seconds time is specified in this example but any values larger than 0.1
would suffice. Without the wait section, this test case would fail with
the following output:
# Failed test 'TEST 1: wait for the timer - pattern "HERE!" matches a line in error.log (req 0)'
Obviously the test scaffold checks the error log too soon, even before the timer handler runs.
Section Review
Test::Nginx::Socket offers a rich set of data sections for specifying
various different input data and expected output data, ranging from NGINX
configuration file snippets, test requests, to expected responses and error
log messages. We have already demonstrated the power of data driven testing
and declarative test case crafting. We want to achieve multiple goals at
the same time, that is, not only to make the tests self-contained and highly
readable, but also to make the test report easy to interpret and analyze
when some of the tests fail. Raw files automatically generated by the test
scaffold, like t/servroot/conf/nginx.conf and t/servroot/logs/error.log,
should be checked frequently when manually debugging the test cases. The
next section extends the discussion of this section with a focus on testing
erroneous cases.