Commit | Line | Data |
---|---|---|
920dae64 AT |
1 | =head1 NAME |
2 | ||
3 | Test::Harness::TAP - Documentation for the TAP format | |
4 | ||
5 | =head1 SYNOPSIS | |
6 | ||
7 | TAP, the Test Anything Protocol, is Perl's simple text-based interface | |
8 | between testing modules such as Test::More and the test harness | |
9 | Test::Harness. | |
10 | ||
11 | =head1 TODO | |
12 | ||
13 | Exit code of the process. | |
14 | ||
15 | =head1 THE TAP FORMAT | |
16 | ||
17 | TAP's general format is: | |
18 | ||
19 | 1..N | |
20 | ok 1 Description # Directive | |
21 | # Diagnostic | |
22 | .... | |
23 | ok 47 Description | |
24 | ok 48 Description | |
25 | more tests.... | |
26 | ||
27 | For example, a test file's output might look like: | |
28 | ||
29 | 1..4 | |
30 | ok 1 - Input file opened | |
31 | not ok 2 - First line of the input valid | |
32 | ok 3 - Read the rest of the file | |
33 | not ok 4 - Summarized correctly # TODO Not written yet | |
34 | ||
35 | =head1 HARNESS BEHAVIOR | |
36 | ||
37 | In this document, the "harness" is any program analyzing TAP output. | |
38 | Typically this will be Perl's I<prove> program, or the underlying | |
39 | C<Test::Harness::runtests> subroutine. | |
40 | ||
41 | A harness must only read TAP output from standard output and not | |
42 | from standard error. Lines written to standard output matching | |
43 | C</^(not )?ok\b/> must be interpreted as test lines. All other | |
44 | lines must not be considered test output. | |
45 | ||
46 | =head1 TESTS LINES AND THE PLAN | |
47 | ||
48 | =head2 The plan | |
49 | ||
50 | The plan tells how many tests will be run, or how many tests have | |
51 | run. It's a check that the test file hasn't stopped prematurely. | |
52 | It must appear once, whether at the beginning or end of the output. | |
53 | ||
54 | The plan is usually the first line of TAP output and it specifies how | |
55 | many test points are to follow. For example, | |
56 | ||
57 | 1..10 | |
58 | ||
59 | means you plan on running 10 tests. This is a safeguard in case your test | |
60 | file dies silently in the middle of its run. The plan is optional but if | |
61 | there is a plan before the test points it must be the first non-diagnostic | |
62 | line output by the test file. | |
63 | ||
64 | In certain instances a test file may not know how many test points | |
65 | it will ultimately be running. In this case the plan can be the last | |
66 | non-diagnostic line in the output. | |
67 | ||
68 | The plan cannot appear in the middle of the output, nor can it appear more | |
69 | than once. | |
70 | ||
71 | =head2 The test line | |
72 | ||
73 | The core of TAP is the test line. A test file prints one test line test | |
74 | point executed. There must be at least one test line in TAP output. Each | |
75 | test line comprises the following elements: | |
76 | ||
77 | =over 4 | |
78 | ||
79 | =item * C<ok> or C<not ok> | |
80 | ||
81 | This tells whether the test point passed or failed. It must be | |
82 | at the beginning of the line. C</^not ok/> indicates a failed test | |
83 | point. C</^ok/> is a successful test point. This is the only mandatory | |
84 | part of the line. | |
85 | ||
86 | Note that unlike the Directives below, C<ok> and C<not ok> are | |
87 | case-sensitive. | |
88 | ||
89 | =item * Test number | |
90 | ||
91 | TAP expects the C<ok> or C<not ok> to be followed by a test point | |
92 | number. If there is no number the harness must maintain | |
93 | its own counter until the script supplies test numbers again. So | |
94 | the following test output | |
95 | ||
96 | 1..6 | |
97 | not ok | |
98 | ok | |
99 | not ok | |
100 | ok | |
101 | ok | |
102 | ||
103 | has five tests. The sixth is missing. Test::Harness will generate | |
104 | ||
105 | FAILED tests 1, 3, 6 | |
106 | Failed 3/6 tests, 50.00% okay | |
107 | ||
108 | =item * Description | |
109 | ||
110 | Any text after the test number but before a C<#> is the description of | |
111 | the test point. | |
112 | ||
113 | ok 42 this is the description of the test | |
114 | ||
115 | Descriptions should not begin with a digit so that they are not confused | |
116 | with the test point number. | |
117 | ||
118 | The harness may do whatever it wants with the description. | |
119 | ||
120 | =item * Directive | |
121 | ||
122 | The test point may include a directive, following a hash on the | |
123 | test line. There are currently two directives allowed: C<TODO> and | |
124 | C<SKIP>. These are discussed below. | |
125 | ||
126 | =back | |
127 | ||
128 | To summarize: | |
129 | ||
130 | =over 4 | |
131 | ||
132 | =item * ok/not ok (required) | |
133 | ||
134 | =item * Test number (recommended) | |
135 | ||
136 | =item * Description (recommended) | |
137 | ||
138 | =item * Directive (only when necessary) | |
139 | ||
140 | =back | |
141 | ||
142 | =head1 DIRECTIVES | |
143 | ||
144 | Directives are special notes that follow a C<#> on the test line. | |
145 | Only two are currently defined: C<TODO> and C<SKIP>. Note that | |
146 | these two keywords are not case-sensitive. | |
147 | ||
148 | =head2 TODO tests | |
149 | ||
150 | If the directive starts with C<# TODO>, the test is counted as a | |
151 | todo test, and the text after C<TODO> is the explanation. | |
152 | ||
153 | not ok 13 # TODO bend space and time | |
154 | ||
155 | Note that if the TODO has an explanation it must be separated from | |
156 | C<TODO> by a space. | |
157 | ||
158 | These tests represent a feature to be implemented or a bug to be fixed | |
159 | and act as something of an executable "things to do" list. They are | |
160 | B<not> expected to succeed. Should a todo test point begin succeeding, | |
161 | the harness should report it as a bonus. This indicates that whatever | |
162 | you were supposed to do has been done and you should promote this to a | |
163 | normal test point. | |
164 | ||
165 | =head2 Skipping tests | |
166 | ||
167 | If the directive starts with C<# SKIP>, the test is counted as having | |
168 | been skipped. If the whole test file succeeds, the count of skipped | |
169 | tests is included in the generated output. The harness should report | |
170 | the text after C< # SKIP\S*\s+> as a reason for skipping. | |
171 | ||
172 | ok 23 # skip Insufficient flogiston pressure. | |
173 | ||
174 | Similarly, one can include an explanation in a plan line, | |
175 | emitted if the test file is skipped completely: | |
176 | ||
177 | 1..0 # Skipped: WWW::Mechanize not installed | |
178 | ||
179 | =head1 OTHER LINES | |
180 | ||
181 | =head2 Bail out! | |
182 | ||
183 | As an emergency measure a test script can decide that further tests | |
184 | are useless (e.g. missing dependencies) and testing should stop | |
185 | immediately. In that case the test script prints the magic words | |
186 | ||
187 | Bail out! | |
188 | ||
189 | to standard output. Any message after these words must be displayed | |
190 | by the interpreter as the reason why testing must be stopped, as | |
191 | in | |
192 | ||
193 | Bail out! MySQL is not running. | |
194 | ||
195 | =head2 Diagnostics | |
196 | ||
197 | Additional information may be put into the testing output on separate | |
198 | lines. Diagnostic lines should begin with a C<#>, which the harness must | |
199 | ignore, at least as far as analyzing the test results. The harness is | |
200 | free, however, to display the diagnostics. Typically diagnostics are | |
201 | used to provide information about the environment in which test file is | |
202 | running, or to delineate a group of tests. | |
203 | ||
204 | ... | |
205 | ok 18 - Closed database connection | |
206 | # End of database section. | |
207 | # This starts the network part of the test. | |
208 | # Daemon started on port 2112 | |
209 | ok 19 - Opened socket | |
210 | ... | |
211 | ok 47 - Closed socket | |
212 | # End of network tests | |
213 | ||
214 | =head2 Anything else | |
215 | ||
216 | Any output line that is not a plan, a test line or a diagnostic is | |
217 | incorrect. How a harness handles the incorrect line is undefined. | |
218 | Test::Harness silently ignores incorrect lines, but will become more | |
219 | stringent in the future. | |
220 | ||
221 | =head1 EXAMPLES | |
222 | ||
223 | All names, places, and events depicted in any example are wholly | |
224 | fictitious and bear no resemblance to, connection with, or relation to any | |
225 | real entity. Any such similarity is purely coincidental, unintentional, | |
226 | and unintended. | |
227 | ||
228 | =head2 Common with explanation | |
229 | ||
230 | The following TAP listing declares that six tests follow as well as | |
231 | provides handy feedback as to what the test is about to do. All six | |
232 | tests pass. | |
233 | ||
234 | 1..6 | |
235 | # | |
236 | # Create a new Board and Tile, then place | |
237 | # the Tile onto the board. | |
238 | # | |
239 | ok 1 - The object isa Board | |
240 | ok 2 - Board size is zero | |
241 | ok 3 - The object isa Tile | |
242 | ok 4 - Get possible places to put the Tile | |
243 | ok 5 - Placing the tile produces no error | |
244 | ok 6 - Board size is 1 | |
245 | ||
246 | =head2 Unknown amount and failures | |
247 | ||
248 | This hypothetical test program ensures that a handful of servers are | |
249 | online and network-accessible. Because it retrieves the hypothetical | |
250 | servers from a database, it doesn't know exactly how many servers it | |
251 | will need to ping. Thus, the test count is declared at the bottom after | |
252 | all the test points have run. Also, two of the tests fail. | |
253 | ||
254 | ok 1 - retrieving servers from the database | |
255 | # need to ping 6 servers | |
256 | ok 2 - pinged diamond | |
257 | ok 3 - pinged ruby | |
258 | not ok 4 - pinged saphire | |
259 | ok 5 - pinged onyx | |
260 | not ok 6 - pinged quartz | |
261 | ok 7 - pinged gold | |
262 | 1..7 | |
263 | ||
264 | =head2 Giving up | |
265 | ||
266 | This listing reports that a pile of tests are going to be run. However, | |
267 | the first test fails, reportedly because a connection to the database | |
268 | could not be established. The program decided that continuing was | |
269 | pointless and exited. | |
270 | ||
271 | 1..573 | |
272 | not ok 1 - database handle | |
273 | Bail out! Couldn't connect to database. | |
274 | ||
275 | =head2 Skipping a few | |
276 | ||
277 | The following listing plans on running 5 tests. However, our program | |
278 | decided to not run tests 2 thru 5 at all. To properly report this, | |
279 | the tests are marked as being skipped. | |
280 | ||
281 | 1..5 | |
282 | ok 1 - approved operating system | |
283 | # $^0 is solaris | |
284 | ok 2 - # SKIP no /sys directory | |
285 | ok 3 - # SKIP no /sys directory | |
286 | ok 4 - # SKIP no /sys directory | |
287 | ok 5 - # SKIP no /sys directory | |
288 | ||
289 | =head2 Skipping everything | |
290 | ||
291 | This listing shows that the entire listing is a skip. No tests were run. | |
292 | ||
293 | 1..0 # skip because English-to-French translator isn't installed | |
294 | ||
295 | =head2 Got spare tuits? | |
296 | ||
297 | The following example reports that four tests are run and the last two | |
298 | tests failed. However, because the failing tests are marked as things | |
299 | to do later, they are considered successes. Thus, a harness should report | |
300 | this entire listing as a success. | |
301 | ||
302 | 1..4 | |
303 | ok 1 - Creating test program | |
304 | ok 2 - Test program runs, no error | |
305 | not ok 3 - infinite loop # TODO halting problem unsolved | |
306 | not ok 4 - infinite loop 2 # TODO halting problem unsolved | |
307 | ||
308 | =head2 Creative liberties | |
309 | ||
310 | This listing shows an alternate output where the test numbers aren't | |
311 | provided. The test also reports the state of a ficticious board game in | |
312 | diagnostic form. Finally, the test count is reported at the end. | |
313 | ||
314 | ok - created Board | |
315 | ok | |
316 | ok | |
317 | ok | |
318 | ok | |
319 | ok | |
320 | ok | |
321 | ok | |
322 | # +------+------+------+------+ | |
323 | # | |16G | |05C | | |
324 | # | |G N C | |C C G | | |
325 | # | | G | | C +| | |
326 | # +------+------+------+------+ | |
327 | # |10C |01G | |03C | | |
328 | # |R N G |G A G | |C C C | | |
329 | # | R | G | | C +| | |
330 | # +------+------+------+------+ | |
331 | # | |01G |17C |00C | | |
332 | # | |G A G |G N R |R N R | | |
333 | # | | G | R | G | | |
334 | # +------+------+------+------+ | |
335 | ok - board has 7 tiles + starter tile | |
336 | 1..9 | |
337 | ||
338 | =head1 AUTHORS | |
339 | ||
340 | Andy Lester, based on the original Test::Harness documentation by Michael Schwern. | |
341 | ||
342 | =head1 ACKNOWLEDGEMENTS | |
343 | ||
344 | Thanks to | |
345 | Pete Krawczyk, | |
346 | Paul Johnson, | |
347 | Ian Langworth | |
348 | and Nik Clayton | |
349 | for help and contributions on this document. | |
350 | ||
351 | The basis for the TAP format was created by Larry Wall in the | |
352 | original test script for Perl 1. Tim Bunce and Andreas Koenig | |
353 | developed it further with their modifications to Test::Harness. | |
354 | ||
355 | =head1 COPYRIGHT | |
356 | ||
357 | Copyright 2003-2005 by | |
358 | Michael G Schwern C<< <schwern@pobox.com> >>, | |
359 | Andy Lester C<< <andy@petdance.com> >>. | |
360 | ||
361 | This program is free software; you can redistribute it and/or | |
362 | modify it under the same terms as Perl itself. | |
363 | ||
364 | See L<http://www.perl.com/perl/misc/Artistic.html>. | |
365 | ||
366 | =cut |