export PATH=/usr/local/openresty/nginx/sbin:$PATHRunning Tests
Like most Perl-based testing frameworks, Test:Nginx relies on Perl’s
prove
command-line utility to run the test files. The prove utility is usually
shipped with the standard perl distribution so we should already have it
when we have perl installed.
Test::Nginx always invokes a real NGINX server and a real socket client
to run the tests. It automatically uses the nginx program found in the
system environment PATH. It is your responsibility to specify the right
nginx in your PATH environment for the test suite. Usually we just
specify the path of the nginx program inside the OpenResty installation
tree. For example,
Here we assume that OpenResty is installed to the default prefix, i.e.,
/usr/local/openresty/.
You can always use the which command to verify if the PATH environment
is indeed set properly:
$ which nginx
/usr/local/openresty/nginx/sbin/nginxFor convenience, we usually wrap such environment settings in a custom
shell script so that we do not risk polluting the system-wide or account-wide
environment settings nor take on the burden of manually setting the environments
manually for every shell session. For example, I usually have a local bash
script named go in each project I work on. A typical go script might
look like below
#!/usr/bin/env bash
export PATH=/usr/local/openresty/nginx/sbin:$PATH
exec prove "$@"Then we can use this ./go script to substitute the prove utility in
any of the subsequent commands involving prove.
Because Test::Nginx makes heavy use of environment variables for the
callers to fine tune the testing behaviors (as we shall see in later sections),
such shell wrapper scripts also make it easy to manage all these environment
variable settings and hard to get things wrong.
|
Note
|
Please do not confuse the name of this bash script with Google’s Go programming language. It has nothing to do with the Go language in any way. |
Running A Single File
If you want to run a single test file, say, t/foo.t, then all you need
to do is just to type the following command in your terminal.
prove t/foo.tHere inside t/foo.t we employ the simple test file example presented
in the previous section. We repeat the content below for the reader’s
convenience.
t/foo.t
use Test::Nginx::Socket 'no_plan';
run_tests();
__DATA__
=== TEST 1: hello, world
This is just a simple demonstration of the
echo directive provided by ngx_http_echo_module.
--- config
location = /t {
echo "hello, world!";
}
--- request
GET /t
--- response_body
hello, world!
--- error_code: 200It is worth mentioning that we could run the following command instead
if we have a custom wrapper script called ./go for prove (as mentioned
earlier in this section):
./go foo.tWhen everything goes well, it generates an output like this:
t/foo.t .. ok All tests successful. Files=1, Tests=2, 0 wallclock secs (0.02 usr 0.01 sys + 0.08 cusr 0.03 csys = 0.14 CPU) Result: PASS
This is a very concise summary. The first line tells you all tests were passed while the second line gives you a summary of the number of test files (1 in this case), the number of tests (2 in this case), and the wallclock and CPU times used to run all the tests.
It is interesting to see that we have only one test block in the sample
test file but in the test summary output by prove we see that the number
of tests are 2. Why the difference? We can easily find it out by asking
prove to generate a detailed test report for all the individual tests.
This is achieved by passing the -v option (meaning "verbose") to the
prove command we used earlier:
prove -v t/foo.tNow the output shows all the individual tests performed in that test file:
t/foo.t .. ok 1 - TEST 1: hello, world - status code ok ok 2 - TEST 1: hello, world - response_body - response is expected (req 0) 1..2 ok All tests successful. Files=1, Tests=2, 0 wallclock secs (0.01 usr 0.01 sys + 0.07 cusr 0.03 csys = 0.12 CPU) Result: PASS
Obviously, the first test is doing the status code check, which is dictated
by the error_code data section in the test block, and the second test
is doing the response body check, required by the response_body section.
Now the mystery is solved.
It is worth mentioning that the --- error_code: 200 section is automatically
assumed when no error_code section is explicitly provided in the test
block. So our test block above can be simplified by removing the --- error_code:
200 line without affecting the number of tests. This is because that checking
200 response status code is so common that Test::Nginx makes it the default.
If you expect a different status code, like 500, then just add an explicit
error_code section.
From this example, we can see that one test block can contain multiple
tests and the number of tests for any given test block can be determined
or predicted by looking at the data sections performing output checks.
This is important when we provide a "test plan" ourselves to the test file
where a "test plan" is the exact number of tests we expect the current
test file to run. If a different number of tests than the plan were actually
run, then the test result would be considered malicious even when all the
tests are passed successfully. Thus, a test plan adds a strong constraint
on the total number of tests expected to be run. For our t/foo.t file
here, however, we intentionally avoid providing any test plans by passing
the 'no_plan' argument to the use statement that loads the Test::Nginx::Socket
module. We will revisit the "test plan" feature and explain how to provide
one in a later section.
Running Multiple Files
Running multiple test files is straightforward; just specify the file
names on the prove command line, as in
prove -v t/foo.t t/bar.t t/baz.tIf you want to run all the test files directly under the t/ directory,
then using a shell wildcard can be handy:
prove -v t/*.tIn the case that you have sub-directories under t/, you can specify
the -r option to ask prove to recursively traverse the while directory
tree rooted at t/ to find test files:
prove -r t/This command is also the standard way to run the whole test suite of a project.
Running Individual Test Blocks
Test::Nginx makes it easy to run an individual test block in a given
file. Just add the special data section ONLY to that test block you want
to run individually and prove will skip all the other test blocks while
running that test file. For example,
=== TEST 1: hello, world
This is just a simple demonstration of the
echo directive provided by ngx_http_echo_module.
--- config
location = /t {
echo "hello, world!";
}
--- request
GET /t
--- response_body
hello, world!
--- ONLYNow prove won’t run any other test blocks (if any) in the same test file.
This is very handy while debugging a particular test block. You can focus on one test case at a time without worrying about other unrelated test cases stepping in your way.
When using the Vim editor, we can quickly insert
a --- ONLY line to the test block we are viewing in the vim file buffer,
and then type :!prove % in the command mode of vim without leaving the
editor window. This works because vim automatically expands the special
% placeholder with the path of the current active file being edited.
This workflow is great since you never leave your editor window and you
never have to type the title (or other IDs) of your test block nor the
path of the containing test file. You can quickly jump between test blocks
even across different files. Test-driven development usually demands very
frequent interactions and iterations, and Test::Nginx is particularly
optimized to speed up this process.
Sometimes you may forget to remove the --- ONLY line from some test files
even after debugging, this will incorrectly skip all the other tests in
those files. To catch such mistakes, Test::Nginx always reports a warning
for files using the ONLY special section, as in
$ prove t/foo.t
t/foo.t .. # I found ONLY: maybe you're debugging?
t/foo.t .. ok
All tests successful.
Files=1, Tests=2, 0 wallclock secs (0.01 usr 0.00 sys + 0.09 cusr 0.03 csys = 0.13 CPU)
Result: PASSThis way it is much easier to identify any leftover --- ONLY lines.
Similar to ONLY, Test::Nginx also provides the LAST data section
to make the containing test block become the last test block being run
in that test file.
|
Note
|
The special data sections ONLY and LAST are actually features
inherited from the Test::Base module.
|
Skipping Tests
We can specify the special SKIP data section to skip running the containing
test block unconditionally. This is handy when we write a test case that
is for a future feature or a test case for a known bug that we haven’t
had the time to fix right now. For example,
=== TEST 1: test for the future
--- config
location /t {
some_fancy_directive;
}
--- request
GET /t
--- response_body
blah blah blah
--- SKIPIt is also possible to skip a whole test file in the prologue part. Just
replace the use statement with the following form.
use Test::Nginx::Socket skip_all => "some reasons";Then running the test file gives something like follows.
t/foo.t .. skipped: some reasons
|
Note
|
It is also possible to conditionally skip a whole test file but it
requires a little bit of Perl programming. Interested readers can try using
a BEGIN {} before the use statement to calculate the value of
the skip_all option on the fly.
|
Test Running Order
Test File Running Order
Test files are usually run by the alphabetical order of their file names.
Some people prefer explicitly controlling the running order of their test
files by prefixing the test file names with number sequences like 001-,
002-, and etc.
The test suite of the ngx_http_lua module follows this practice, for example, which has test file names like below
t/000-sanity.t t/001-set.t t/002-content.t t/003-errors.t ... t/139-ssl-cert-by.t
Although the prove utility supports running test files in multiple parallel
jobs via the -jN option, Test::Nginx does not really support this mode
since all the test cases share exactly the same test server directory,
t/servroot/, and the same listening ports, as we have already seen, while
parallel running requires strictly isolated running environments for each
individual thread of execution. One can still manually split the test files
into different groups and run each group on a different (virtual) machine
or an isolated environment like a Linux container.
Test Block Running Order
By default, the Test::Nginx scaffold shuffles the test blocks in each
file and run them in a random order. This behavior encourages writing
self-contained and independent test cases and also increases the chance
of hitting a bug by actively mutating the relative running order of the
test cases. This may, indeed, confuse new comers, coming from a more traditional
testing platform.
We can always disable this test block shuffling behavior by calling the
Perl function, no_shuffle(), imported by the Test::Nginx::Socket module,
before the run_tests() call in the test file prologue. For example,
use Test::Nginx::Socket 'no_plan';
no_shuffle();
run_tests();
__DATA__
...With the no_shuffle() call in place, the test blocks are run in the exact
same order as their appearance in the test file.