| 1 | =head1 NAME |
| 2 | |
| 3 | Test::Harness::TAP - Documentation for the TAP format |
| 4 | |
| 5 | =head1 SYNOPSIS |
| 6 | |
| 7 | TAP, the Test Anything Protocol, is Perl's simple text-based interface |
| 8 | between testing modules such as Test::More and the test harness |
| 9 | Test::Harness. |
| 10 | |
| 11 | =head1 TODO |
| 12 | |
| 13 | Exit code of the process. |
| 14 | |
| 15 | =head1 THE TAP FORMAT |
| 16 | |
| 17 | TAP's general format is: |
| 18 | |
| 19 | 1..N |
| 20 | ok 1 Description # Directive |
| 21 | # Diagnostic |
| 22 | .... |
| 23 | ok 47 Description |
| 24 | ok 48 Description |
| 25 | more tests.... |
| 26 | |
| 27 | For example, a test file's output might look like: |
| 28 | |
| 29 | 1..4 |
| 30 | ok 1 - Input file opened |
| 31 | not ok 2 - First line of the input valid |
| 32 | ok 3 - Read the rest of the file |
| 33 | not ok 4 - Summarized correctly # TODO Not written yet |
| 34 | |
| 35 | =head1 HARNESS BEHAVIOR |
| 36 | |
| 37 | In this document, the "harness" is any program analyzing TAP output. |
| 38 | Typically this will be Perl's I<prove> program, or the underlying |
| 39 | C<Test::Harness::runtests> subroutine. |
| 40 | |
| 41 | A harness must only read TAP output from standard output and not |
| 42 | from standard error. Lines written to standard output matching |
| 43 | C</^(not )?ok\b/> must be interpreted as test lines. All other |
| 44 | lines must not be considered test output. |
| 45 | |
| 46 | =head1 TESTS LINES AND THE PLAN |
| 47 | |
| 48 | =head2 The plan |
| 49 | |
| 50 | The plan tells how many tests will be run, or how many tests have |
| 51 | run. It's a check that the test file hasn't stopped prematurely. |
| 52 | It must appear once, whether at the beginning or end of the output. |
| 53 | |
| 54 | The plan is usually the first line of TAP output and it specifies how |
| 55 | many test points are to follow. For example, |
| 56 | |
| 57 | 1..10 |
| 58 | |
| 59 | means you plan on running 10 tests. This is a safeguard in case your test |
| 60 | file dies silently in the middle of its run. The plan is optional but if |
| 61 | there is a plan before the test points it must be the first non-diagnostic |
| 62 | line output by the test file. |
| 63 | |
| 64 | In certain instances a test file may not know how many test points |
| 65 | it will ultimately be running. In this case the plan can be the last |
| 66 | non-diagnostic line in the output. |
| 67 | |
| 68 | The plan cannot appear in the middle of the output, nor can it appear more |
| 69 | than once. |
| 70 | |
| 71 | =head2 The test line |
| 72 | |
| 73 | The core of TAP is the test line. A test file prints one test line test |
| 74 | point executed. There must be at least one test line in TAP output. Each |
| 75 | test line comprises the following elements: |
| 76 | |
| 77 | =over 4 |
| 78 | |
| 79 | =item * C<ok> or C<not ok> |
| 80 | |
| 81 | This tells whether the test point passed or failed. It must be |
| 82 | at the beginning of the line. C</^not ok/> indicates a failed test |
| 83 | point. C</^ok/> is a successful test point. This is the only mandatory |
| 84 | part of the line. |
| 85 | |
| 86 | Note that unlike the Directives below, C<ok> and C<not ok> are |
| 87 | case-sensitive. |
| 88 | |
| 89 | =item * Test number |
| 90 | |
| 91 | TAP expects the C<ok> or C<not ok> to be followed by a test point |
| 92 | number. If there is no number the harness must maintain |
| 93 | its own counter until the script supplies test numbers again. So |
| 94 | the following test output |
| 95 | |
| 96 | 1..6 |
| 97 | not ok |
| 98 | ok |
| 99 | not ok |
| 100 | ok |
| 101 | ok |
| 102 | |
| 103 | has five tests. The sixth is missing. Test::Harness will generate |
| 104 | |
| 105 | FAILED tests 1, 3, 6 |
| 106 | Failed 3/6 tests, 50.00% okay |
| 107 | |
| 108 | =item * Description |
| 109 | |
| 110 | Any text after the test number but before a C<#> is the description of |
| 111 | the test point. |
| 112 | |
| 113 | ok 42 this is the description of the test |
| 114 | |
| 115 | Descriptions should not begin with a digit so that they are not confused |
| 116 | with the test point number. |
| 117 | |
| 118 | The harness may do whatever it wants with the description. |
| 119 | |
| 120 | =item * Directive |
| 121 | |
| 122 | The test point may include a directive, following a hash on the |
| 123 | test line. There are currently two directives allowed: C<TODO> and |
| 124 | C<SKIP>. These are discussed below. |
| 125 | |
| 126 | =back |
| 127 | |
| 128 | To summarize: |
| 129 | |
| 130 | =over 4 |
| 131 | |
| 132 | =item * ok/not ok (required) |
| 133 | |
| 134 | =item * Test number (recommended) |
| 135 | |
| 136 | =item * Description (recommended) |
| 137 | |
| 138 | =item * Directive (only when necessary) |
| 139 | |
| 140 | =back |
| 141 | |
| 142 | =head1 DIRECTIVES |
| 143 | |
| 144 | Directives are special notes that follow a C<#> on the test line. |
| 145 | Only two are currently defined: C<TODO> and C<SKIP>. Note that |
| 146 | these two keywords are not case-sensitive. |
| 147 | |
| 148 | =head2 TODO tests |
| 149 | |
| 150 | If the directive starts with C<# TODO>, the test is counted as a |
| 151 | todo test, and the text after C<TODO> is the explanation. |
| 152 | |
| 153 | not ok 13 # TODO bend space and time |
| 154 | |
| 155 | Note that if the TODO has an explanation it must be separated from |
| 156 | C<TODO> by a space. |
| 157 | |
| 158 | These tests represent a feature to be implemented or a bug to be fixed |
| 159 | and act as something of an executable "things to do" list. They are |
| 160 | B<not> expected to succeed. Should a todo test point begin succeeding, |
| 161 | the harness should report it as a bonus. This indicates that whatever |
| 162 | you were supposed to do has been done and you should promote this to a |
| 163 | normal test point. |
| 164 | |
| 165 | =head2 Skipping tests |
| 166 | |
| 167 | If the directive starts with C<# SKIP>, the test is counted as having |
| 168 | been skipped. If the whole test file succeeds, the count of skipped |
| 169 | tests is included in the generated output. The harness should report |
| 170 | the text after C< # SKIP\S*\s+> as a reason for skipping. |
| 171 | |
| 172 | ok 23 # skip Insufficient flogiston pressure. |
| 173 | |
| 174 | Similarly, one can include an explanation in a plan line, |
| 175 | emitted if the test file is skipped completely: |
| 176 | |
| 177 | 1..0 # Skipped: WWW::Mechanize not installed |
| 178 | |
| 179 | =head1 OTHER LINES |
| 180 | |
| 181 | =head2 Bail out! |
| 182 | |
| 183 | As an emergency measure a test script can decide that further tests |
| 184 | are useless (e.g. missing dependencies) and testing should stop |
| 185 | immediately. In that case the test script prints the magic words |
| 186 | |
| 187 | Bail out! |
| 188 | |
| 189 | to standard output. Any message after these words must be displayed |
| 190 | by the interpreter as the reason why testing must be stopped, as |
| 191 | in |
| 192 | |
| 193 | Bail out! MySQL is not running. |
| 194 | |
| 195 | =head2 Diagnostics |
| 196 | |
| 197 | Additional information may be put into the testing output on separate |
| 198 | lines. Diagnostic lines should begin with a C<#>, which the harness must |
| 199 | ignore, at least as far as analyzing the test results. The harness is |
| 200 | free, however, to display the diagnostics. Typically diagnostics are |
| 201 | used to provide information about the environment in which test file is |
| 202 | running, or to delineate a group of tests. |
| 203 | |
| 204 | ... |
| 205 | ok 18 - Closed database connection |
| 206 | # End of database section. |
| 207 | # This starts the network part of the test. |
| 208 | # Daemon started on port 2112 |
| 209 | ok 19 - Opened socket |
| 210 | ... |
| 211 | ok 47 - Closed socket |
| 212 | # End of network tests |
| 213 | |
| 214 | =head2 Anything else |
| 215 | |
| 216 | Any output line that is not a plan, a test line or a diagnostic is |
| 217 | incorrect. How a harness handles the incorrect line is undefined. |
| 218 | Test::Harness silently ignores incorrect lines, but will become more |
| 219 | stringent in the future. |
| 220 | |
| 221 | =head1 EXAMPLES |
| 222 | |
| 223 | All names, places, and events depicted in any example are wholly |
| 224 | fictitious and bear no resemblance to, connection with, or relation to any |
| 225 | real entity. Any such similarity is purely coincidental, unintentional, |
| 226 | and unintended. |
| 227 | |
| 228 | =head2 Common with explanation |
| 229 | |
| 230 | The following TAP listing declares that six tests follow as well as |
| 231 | provides handy feedback as to what the test is about to do. All six |
| 232 | tests pass. |
| 233 | |
| 234 | 1..6 |
| 235 | # |
| 236 | # Create a new Board and Tile, then place |
| 237 | # the Tile onto the board. |
| 238 | # |
| 239 | ok 1 - The object isa Board |
| 240 | ok 2 - Board size is zero |
| 241 | ok 3 - The object isa Tile |
| 242 | ok 4 - Get possible places to put the Tile |
| 243 | ok 5 - Placing the tile produces no error |
| 244 | ok 6 - Board size is 1 |
| 245 | |
| 246 | =head2 Unknown amount and failures |
| 247 | |
| 248 | This hypothetical test program ensures that a handful of servers are |
| 249 | online and network-accessible. Because it retrieves the hypothetical |
| 250 | servers from a database, it doesn't know exactly how many servers it |
| 251 | will need to ping. Thus, the test count is declared at the bottom after |
| 252 | all the test points have run. Also, two of the tests fail. |
| 253 | |
| 254 | ok 1 - retrieving servers from the database |
| 255 | # need to ping 6 servers |
| 256 | ok 2 - pinged diamond |
| 257 | ok 3 - pinged ruby |
| 258 | not ok 4 - pinged saphire |
| 259 | ok 5 - pinged onyx |
| 260 | not ok 6 - pinged quartz |
| 261 | ok 7 - pinged gold |
| 262 | 1..7 |
| 263 | |
| 264 | =head2 Giving up |
| 265 | |
| 266 | This listing reports that a pile of tests are going to be run. However, |
| 267 | the first test fails, reportedly because a connection to the database |
| 268 | could not be established. The program decided that continuing was |
| 269 | pointless and exited. |
| 270 | |
| 271 | 1..573 |
| 272 | not ok 1 - database handle |
| 273 | Bail out! Couldn't connect to database. |
| 274 | |
| 275 | =head2 Skipping a few |
| 276 | |
| 277 | The following listing plans on running 5 tests. However, our program |
| 278 | decided to not run tests 2 thru 5 at all. To properly report this, |
| 279 | the tests are marked as being skipped. |
| 280 | |
| 281 | 1..5 |
| 282 | ok 1 - approved operating system |
| 283 | # $^0 is solaris |
| 284 | ok 2 - # SKIP no /sys directory |
| 285 | ok 3 - # SKIP no /sys directory |
| 286 | ok 4 - # SKIP no /sys directory |
| 287 | ok 5 - # SKIP no /sys directory |
| 288 | |
| 289 | =head2 Skipping everything |
| 290 | |
| 291 | This listing shows that the entire listing is a skip. No tests were run. |
| 292 | |
| 293 | 1..0 # skip because English-to-French translator isn't installed |
| 294 | |
| 295 | =head2 Got spare tuits? |
| 296 | |
| 297 | The following example reports that four tests are run and the last two |
| 298 | tests failed. However, because the failing tests are marked as things |
| 299 | to do later, they are considered successes. Thus, a harness should report |
| 300 | this entire listing as a success. |
| 301 | |
| 302 | 1..4 |
| 303 | ok 1 - Creating test program |
| 304 | ok 2 - Test program runs, no error |
| 305 | not ok 3 - infinite loop # TODO halting problem unsolved |
| 306 | not ok 4 - infinite loop 2 # TODO halting problem unsolved |
| 307 | |
| 308 | =head2 Creative liberties |
| 309 | |
| 310 | This listing shows an alternate output where the test numbers aren't |
| 311 | provided. The test also reports the state of a ficticious board game in |
| 312 | diagnostic form. Finally, the test count is reported at the end. |
| 313 | |
| 314 | ok - created Board |
| 315 | ok |
| 316 | ok |
| 317 | ok |
| 318 | ok |
| 319 | ok |
| 320 | ok |
| 321 | ok |
| 322 | # +------+------+------+------+ |
| 323 | # | |16G | |05C | |
| 324 | # | |G N C | |C C G | |
| 325 | # | | G | | C +| |
| 326 | # +------+------+------+------+ |
| 327 | # |10C |01G | |03C | |
| 328 | # |R N G |G A G | |C C C | |
| 329 | # | R | G | | C +| |
| 330 | # +------+------+------+------+ |
| 331 | # | |01G |17C |00C | |
| 332 | # | |G A G |G N R |R N R | |
| 333 | # | | G | R | G | |
| 334 | # +------+------+------+------+ |
| 335 | ok - board has 7 tiles + starter tile |
| 336 | 1..9 |
| 337 | |
| 338 | =head1 AUTHORS |
| 339 | |
| 340 | Andy Lester, based on the original Test::Harness documentation by Michael Schwern. |
| 341 | |
| 342 | =head1 ACKNOWLEDGEMENTS |
| 343 | |
| 344 | Thanks to |
| 345 | Pete Krawczyk, |
| 346 | Paul Johnson, |
| 347 | Ian Langworth |
| 348 | and Nik Clayton |
| 349 | for help and contributions on this document. |
| 350 | |
| 351 | The basis for the TAP format was created by Larry Wall in the |
| 352 | original test script for Perl 1. Tim Bunce and Andreas Koenig |
| 353 | developed it further with their modifications to Test::Harness. |
| 354 | |
| 355 | =head1 COPYRIGHT |
| 356 | |
| 357 | Copyright 2003-2005 by |
| 358 | Michael G Schwern C<< <schwern@pobox.com> >>, |
| 359 | Andy Lester C<< <andy@petdance.com> >>. |
| 360 | |
| 361 | This program is free software; you can redistribute it and/or |
| 362 | modify it under the same terms as Perl itself. |
| 363 | |
| 364 | See L<http://www.perl.com/perl/misc/Artistic.html>. |
| 365 | |
| 366 | =cut |