Commit | Line | Data |
---|---|---|
86530b38 AT |
1 | =head1 NAME |
2 | ||
3 | Test::Tutorial - A tutorial about writing really basic tests | |
4 | ||
5 | =head1 DESCRIPTION | |
6 | ||
7 | ||
8 | I<AHHHHHHH!!!! NOT TESTING! Anything but testing! | |
9 | Beat me, whip me, send me to Detroit, but don't make | |
10 | me write tests!> | |
11 | ||
12 | I<*sob*> | |
13 | ||
14 | I<Besides, I don't know how to write the damned things.> | |
15 | ||
16 | ||
17 | Is this you? Is writing tests right up there with writing | |
18 | documentation and having your fingernails pulled out? Did you open up | |
19 | a test and read | |
20 | ||
21 | ######## We start with some black magic | |
22 | ||
23 | and decide that's quite enough for you? | |
24 | ||
25 | It's ok. That's all gone now. We've done all the black magic for | |
26 | you. And here are the tricks... | |
27 | ||
28 | ||
29 | =head2 Nuts and bolts of testing. | |
30 | ||
31 | Here's the most basic test program. | |
32 | ||
33 | #!/usr/bin/perl -w | |
34 | ||
35 | print "1..1\n"; | |
36 | ||
37 | print 1 + 1 == 2 ? "ok 1\n" : "not ok 1\n"; | |
38 | ||
39 | since 1 + 1 is 2, it prints: | |
40 | ||
41 | 1..1 | |
42 | ok 1 | |
43 | ||
44 | What this says is: C<1..1> "I'm going to run one test." [1] C<ok 1> | |
45 | "The first test passed". And that's about all magic there is to | |
46 | testing. Your basic unit of testing is the I<ok>. For each thing you | |
47 | test, an C<ok> is printed. Simple. B<Test::Harness> interprets your test | |
48 | results to determine if you succeeded or failed (more on that later). | |
49 | ||
50 | Writing all these print statements rapidly gets tedious. Fortunately, | |
51 | there's B<Test::Simple>. It has one function, C<ok()>. | |
52 | ||
53 | #!/usr/bin/perl -w | |
54 | ||
55 | use Test::Simple tests => 1; | |
56 | ||
57 | ok( 1 + 1 == 2 ); | |
58 | ||
59 | and that does the same thing as the code above. C<ok()> is the backbone | |
60 | of Perl testing, and we'll be using it instead of roll-your-own from | |
61 | here on. If C<ok()> gets a true value, the test passes. False, it | |
62 | fails. | |
63 | ||
64 | #!/usr/bin/perl -w | |
65 | ||
66 | use Test::Simple tests => 2; | |
67 | ok( 1 + 1 == 2 ); | |
68 | ok( 2 + 2 == 5 ); | |
69 | ||
70 | from that comes | |
71 | ||
72 | 1..2 | |
73 | ok 1 | |
74 | not ok 2 | |
75 | # Failed test (test.pl at line 5) | |
76 | # Looks like you failed 1 tests of 2. | |
77 | ||
78 | C<1..2> "I'm going to run two tests." This number is used to ensure | |
79 | your test program ran all the way through and didn't die or skip some | |
80 | tests. C<ok 1> "The first test passed." C<not ok 2> "The second test | |
81 | failed". Test::Simple helpfully prints out some extra commentary about | |
82 | your tests. | |
83 | ||
84 | It's not scary. Come, hold my hand. We're going to give an example | |
85 | of testing a module. For our example, we'll be testing a date | |
86 | library, B<Date::ICal>. It's on CPAN, so download a copy and follow | |
87 | along. [2] | |
88 | ||
89 | ||
90 | =head2 Where to start? | |
91 | ||
92 | This is the hardest part of testing, where do you start? People often | |
93 | get overwhelmed at the apparent enormity of the task of testing a | |
94 | whole module. Best place to start is at the beginning. Date::ICal is | |
95 | an object-oriented module, and that means you start by making an | |
96 | object. So we test C<new()>. | |
97 | ||
98 | #!/usr/bin/perl -w | |
99 | ||
100 | use Test::Simple tests => 2; | |
101 | ||
102 | use Date::ICal; | |
103 | ||
104 | my $ical = Date::ICal->new; # create an object | |
105 | ok( defined $ical ); # check that we got something | |
106 | ok( $ical->isa('Date::ICal') ); # and it's the right class | |
107 | ||
108 | run that and you should get: | |
109 | ||
110 | 1..2 | |
111 | ok 1 | |
112 | ok 2 | |
113 | ||
114 | congratulations, you've written your first useful test. | |
115 | ||
116 | ||
117 | =head2 Names | |
118 | ||
119 | That output isn't terribly descriptive, is it? When you have two | |
120 | tests you can figure out which one is #2, but what if you have 102? | |
121 | ||
122 | Each test can be given a little descriptive name as the second | |
123 | argument to C<ok()>. | |
124 | ||
125 | use Test::Simple tests => 2; | |
126 | ||
127 | ok( defined $ical, 'new() returned something' ); | |
128 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); | |
129 | ||
130 | So now you'd see... | |
131 | ||
132 | 1..2 | |
133 | ok 1 - new() returned something | |
134 | ok 2 - and it's the right class | |
135 | ||
136 | ||
137 | =head2 Test the manual | |
138 | ||
139 | Simplest way to build up a decent testing suite is to just test what | |
140 | the manual says it does. [3] Let's pull something out of the | |
141 | L<Date::ICal/SYNOPSIS> and test that all its bits work. | |
142 | ||
143 | #!/usr/bin/perl -w | |
144 | ||
145 | use Test::Simple tests => 8; | |
146 | ||
147 | use Date::ICal; | |
148 | ||
149 | $ical = Date::ICal->new( year => 1964, month => 10, day => 16, | |
150 | hour => 16, min => 12, sec => 47, | |
151 | tz => '0530' ); | |
152 | ||
153 | ok( defined $ical, 'new() returned something' ); | |
154 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); | |
155 | ok( $ical->sec == 47, ' sec()' ); | |
156 | ok( $ical->min == 12, ' min()' ); | |
157 | ok( $ical->hour == 16, ' hour()' ); | |
158 | ok( $ical->day == 17, ' day()' ); | |
159 | ok( $ical->month == 10, ' month()' ); | |
160 | ok( $ical->year == 1964, ' year()' ); | |
161 | ||
162 | run that and you get: | |
163 | ||
164 | 1..8 | |
165 | ok 1 - new() returned something | |
166 | ok 2 - and it's the right class | |
167 | ok 3 - sec() | |
168 | ok 4 - min() | |
169 | ok 5 - hour() | |
170 | not ok 6 - day() | |
171 | # Failed test (- at line 16) | |
172 | ok 7 - month() | |
173 | ok 8 - year() | |
174 | # Looks like you failed 1 tests of 8. | |
175 | ||
176 | Whoops, a failure! [4] Test::Simple helpfully lets us know on what line | |
177 | the failure occured, but not much else. We were supposed to get 17, | |
178 | but we didn't. What did we get?? Dunno. We'll have to re-run the | |
179 | test in the debugger or throw in some print statements to find out. | |
180 | ||
181 | Instead, we'll switch from B<Test::Simple> to B<Test::More>. B<Test::More> | |
182 | does everything B<Test::Simple> does, and more! In fact, Test::More does | |
183 | things I<exactly> the way Test::Simple does. You can literally swap | |
184 | Test::Simple out and put Test::More in its place. That's just what | |
185 | we're going to do. | |
186 | ||
187 | Test::More does more than Test::Simple. The most important difference | |
188 | at this point is it provides more informative ways to say "ok". | |
189 | Although you can write almost any test with a generic C<ok()>, it | |
190 | can't tell you what went wrong. Instead, we'll use the C<is()> | |
191 | function, which lets us declare that something is supposed to be the | |
192 | same as something else: | |
193 | ||
194 | #!/usr/bin/perl -w | |
195 | ||
196 | use Test::More tests => 8; | |
197 | ||
198 | use Date::ICal; | |
199 | ||
200 | $ical = Date::ICal->new( year => 1964, month => 10, day => 16, | |
201 | hour => 16, min => 12, sec => 47, | |
202 | tz => '0530' ); | |
203 | ||
204 | ok( defined $ical, 'new() returned something' ); | |
205 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); | |
206 | is( $ical->sec, 47, ' sec()' ); | |
207 | is( $ical->min, 12, ' min()' ); | |
208 | is( $ical->hour, 16, ' hour()' ); | |
209 | is( $ical->day, 17, ' day()' ); | |
210 | is( $ical->month, 10, ' month()' ); | |
211 | is( $ical->year, 1964, ' year()' ); | |
212 | ||
213 | "Is C<$ical-E<gt>sec> 47?" "Is C<$ical-E<gt>min> 12?" With C<is()> in place, | |
214 | you get some more information | |
215 | ||
216 | 1..8 | |
217 | ok 1 - new() returned something | |
218 | ok 2 - and it's the right class | |
219 | ok 3 - sec() | |
220 | ok 4 - min() | |
221 | ok 5 - hour() | |
222 | not ok 6 - day() | |
223 | # Failed test (- at line 16) | |
224 | # got: '16' | |
225 | # expected: '17' | |
226 | ok 7 - month() | |
227 | ok 8 - year() | |
228 | # Looks like you failed 1 tests of 8. | |
229 | ||
230 | letting us know that C<$ical-E<gt>day> returned 16, but we expected 17. A | |
231 | quick check shows that the code is working fine, we made a mistake | |
232 | when writing up the tests. Just change it to: | |
233 | ||
234 | is( $ical->day, 16, ' day()' ); | |
235 | ||
236 | and everything works. | |
237 | ||
238 | So any time you're doing a "this equals that" sort of test, use C<is()>. | |
239 | It even works on arrays. The test is always in scalar context, so you | |
240 | can test how many elements are in a list this way. [5] | |
241 | ||
242 | is( @foo, 5, 'foo has 5 elements' ); | |
243 | ||
244 | ||
245 | =head2 Sometimes the tests are wrong | |
246 | ||
247 | Which brings us to a very important lesson. Code has bugs. Tests are | |
248 | code. Ergo, tests have bugs. A failing test could mean a bug in the | |
249 | code, but don't discount the possibility that the test is wrong. | |
250 | ||
251 | On the flip side, don't be tempted to prematurely declare a test | |
252 | incorrect just because you're having trouble finding the bug. | |
253 | Invalidating a test isn't something to be taken lightly, and don't use | |
254 | it as a cop out to avoid work. | |
255 | ||
256 | ||
257 | =head2 Testing lots of values | |
258 | ||
259 | We're going to be wanting to test a lot of dates here, trying to trick | |
260 | the code with lots of different edge cases. Does it work before 1970? | |
261 | After 2038? Before 1904? Do years after 10,000 give it trouble? | |
262 | Does it get leap years right? We could keep repeating the code above, | |
263 | or we could set up a little try/expect loop. | |
264 | ||
265 | use Test::More tests => 32; | |
266 | use Date::ICal; | |
267 | ||
268 | my %ICal_Dates = ( | |
269 | # An ICal string And the year, month, date | |
270 | # hour, minute and second we expect. | |
271 | '19971024T120000' => # from the docs. | |
272 | [ 1997, 10, 24, 12, 0, 0 ], | |
273 | '20390123T232832' => # after the Unix epoch | |
274 | [ 2039, 1, 23, 23, 28, 32 ], | |
275 | '19671225T000000' => # before the Unix epoch | |
276 | [ 1967, 12, 25, 0, 0, 0 ], | |
277 | '18990505T232323' => # before the MacOS epoch | |
278 | [ 1899, 5, 5, 23, 23, 23 ], | |
279 | ); | |
280 | ||
281 | ||
282 | while( my($ical_str, $expect) = each %ICal_Dates ) { | |
283 | my $ical = Date::ICal->new( ical => $ical_str ); | |
284 | ||
285 | ok( defined $ical, "new(ical => '$ical_str')" ); | |
286 | ok( $ical->isa('Date::ICal'), " and it's the right class" ); | |
287 | ||
288 | is( $ical->year, $expect->[0], ' year()' ); | |
289 | is( $ical->month, $expect->[1], ' month()' ); | |
290 | is( $ical->day, $expect->[2], ' day()' ); | |
291 | is( $ical->hour, $expect->[3], ' hour()' ); | |
292 | is( $ical->min, $expect->[4], ' min()' ); | |
293 | is( $ical->sec, $expect->[5], ' sec()' ); | |
294 | } | |
295 | ||
296 | So now we can test bunches of dates by just adding them to | |
297 | C<%ICal_Dates>. Now that it's less work to test with more dates, you'll | |
298 | be inclined to just throw more in as you think of them. | |
299 | Only problem is, every time we add to that we have to keep adjusting | |
300 | the C<use Test::More tests =E<gt> ##> line. That can rapidly get | |
301 | annoying. Instead we use C<no_plan>. This means we're just running | |
302 | some tests, don't know how many. [6] | |
303 | ||
304 | use Test::More 'no_plan'; # instead of tests => 32 | |
305 | ||
306 | now we can just add tests and not have to do all sorts of math to | |
307 | figure out how many we're running. | |
308 | ||
309 | ||
310 | =head2 Informative names | |
311 | ||
312 | Take a look at this line here | |
313 | ||
314 | ok( defined $ical, "new(ical => '$ical_str')" ); | |
315 | ||
316 | we've added more detail about what we're testing and the ICal string | |
317 | itself we're trying out to the name. So you get results like: | |
318 | ||
319 | ok 25 - new(ical => '19971024T120000') | |
320 | ok 26 - and it's the right class | |
321 | ok 27 - year() | |
322 | ok 28 - month() | |
323 | ok 29 - day() | |
324 | ok 30 - hour() | |
325 | ok 31 - min() | |
326 | ok 32 - sec() | |
327 | ||
328 | if something in there fails, you'll know which one it was and that | |
329 | will make tracking down the problem easier. So try to put a bit of | |
330 | debugging information into the test names. | |
331 | ||
332 | Describe what the tests test, to make debugging a failed test easier | |
333 | for you or for the next person who runs your test. | |
334 | ||
335 | ||
336 | =head2 Skipping tests | |
337 | ||
338 | Poking around in the existing Date::ICal tests, I found this in | |
339 | F<t/01sanity.t> [7] | |
340 | ||
341 | #!/usr/bin/perl -w | |
342 | ||
343 | use Test::More tests => 7; | |
344 | use Date::ICal; | |
345 | ||
346 | # Make sure epoch time is being handled sanely. | |
347 | my $t1 = Date::ICal->new( epoch => 0 ); | |
348 | is( $t1->epoch, 0, "Epoch time of 0" ); | |
349 | ||
350 | # XXX This will only work on unix systems. | |
351 | is( $t1->ical, '19700101Z', " epoch to ical" ); | |
352 | ||
353 | is( $t1->year, 1970, " year()" ); | |
354 | is( $t1->month, 1, " month()" ); | |
355 | is( $t1->day, 1, " day()" ); | |
356 | ||
357 | # like the tests above, but starting with ical instead of epoch | |
358 | my $t2 = Date::ICal->new( ical => '19700101Z' ); | |
359 | is( $t2->ical, '19700101Z', "Start of epoch in ICal notation" ); | |
360 | ||
361 | is( $t2->epoch, 0, " and back to ICal" ); | |
362 | ||
363 | The beginning of the epoch is different on most non-Unix operating | |
364 | systems [8]. Even though Perl smooths out the differences for the most | |
365 | part, certain ports do it differently. MacPerl is one off the top of | |
366 | my head. [9] We I<know> this will never work on MacOS. So rather than | |
367 | just putting a comment in the test, we can explicitly say it's never | |
368 | going to work and skip the test. | |
369 | ||
370 | use Test::More tests => 7; | |
371 | use Date::ICal; | |
372 | ||
373 | # Make sure epoch time is being handled sanely. | |
374 | my $t1 = Date::ICal->new( epoch => 0 ); | |
375 | is( $t1->epoch, 0, "Epoch time of 0" ); | |
376 | ||
377 | SKIP: { | |
378 | skip('epoch to ICal not working on MacOS', 6) | |
379 | if $^O eq 'MacOS'; | |
380 | ||
381 | is( $t1->ical, '19700101Z', " epoch to ical" ); | |
382 | ||
383 | is( $t1->year, 1970, " year()" ); | |
384 | is( $t1->month, 1, " month()" ); | |
385 | is( $t1->day, 1, " day()" ); | |
386 | ||
387 | # like the tests above, but starting with ical instead of epoch | |
388 | my $t2 = Date::ICal->new( ical => '19700101Z' ); | |
389 | is( $t2->ical, '19700101Z', "Start of epoch in ICal notation" ); | |
390 | ||
391 | is( $t2->epoch, 0, " and back to ICal" ); | |
392 | } | |
393 | ||
394 | A little bit of magic happens here. When running on anything but | |
395 | MacOS, all the tests run normally. But when on MacOS, C<skip()> causes | |
396 | the entire contents of the SKIP block to be jumped over. It's never | |
397 | run. Instead, it prints special output that tells Test::Harness that | |
398 | the tests have been skipped. | |
399 | ||
400 | 1..7 | |
401 | ok 1 - Epoch time of 0 | |
402 | ok 2 # skip epoch to ICal not working on MacOS | |
403 | ok 3 # skip epoch to ICal not working on MacOS | |
404 | ok 4 # skip epoch to ICal not working on MacOS | |
405 | ok 5 # skip epoch to ICal not working on MacOS | |
406 | ok 6 # skip epoch to ICal not working on MacOS | |
407 | ok 7 # skip epoch to ICal not working on MacOS | |
408 | ||
409 | This means your tests won't fail on MacOS. This means less emails | |
410 | from MacPerl users telling you about failing tests that you know will | |
411 | never work. You've got to be careful with skip tests. These are for | |
412 | tests which don't work and I<never will>. It is not for skipping | |
413 | genuine bugs (we'll get to that in a moment). | |
414 | ||
415 | The tests are wholly and completely skipped. [10] This will work. | |
416 | ||
417 | SKIP: { | |
418 | skip("I don't wanna die!"); | |
419 | ||
420 | die, die, die, die, die; | |
421 | } | |
422 | ||
423 | ||
424 | =head2 Todo tests | |
425 | ||
426 | Thumbing through the Date::ICal man page, I came across this: | |
427 | ||
428 | ical | |
429 | ||
430 | $ical_string = $ical->ical; | |
431 | ||
432 | Retrieves, or sets, the date on the object, using any | |
433 | valid ICal date/time string. | |
434 | ||
435 | "Retrieves or sets". Hmmm, didn't see a test for using C<ical()> to set | |
436 | the date in the Date::ICal test suite. So I'll write one. | |
437 | ||
438 | use Test::More tests => 1; | |
439 | ||
440 | my $ical = Date::ICal->new; | |
441 | $ical->ical('20201231Z'); | |
442 | is( $ical->ical, '20201231Z', 'Setting via ical()' ); | |
443 | ||
444 | run that and I get | |
445 | ||
446 | 1..1 | |
447 | not ok 1 - Setting via ical() | |
448 | # Failed test (- at line 6) | |
449 | # got: '20010814T233649Z' | |
450 | # expected: '20201231Z' | |
451 | # Looks like you failed 1 tests of 1. | |
452 | ||
453 | Whoops! Looks like it's unimplemented. Let's assume we don't have | |
454 | the time to fix this. [11] Normally, you'd just comment out the test | |
455 | and put a note in a todo list somewhere. Instead, we're going to | |
456 | explicitly state "this test will fail" by wrapping it in a C<TODO> block. | |
457 | ||
458 | use Test::More tests => 1; | |
459 | ||
460 | TODO: { | |
461 | local $TODO = 'ical($ical) not yet implemented'; | |
462 | ||
463 | my $ical = Date::ICal->new; | |
464 | $ical->ical('20201231Z'); | |
465 | ||
466 | is( $ical->ical, '20201231Z', 'Setting via ical()' ); | |
467 | } | |
468 | ||
469 | Now when you run, it's a little different: | |
470 | ||
471 | 1..1 | |
472 | not ok 1 - Setting via ical() # TODO ical($ical) not yet implemented | |
473 | # got: '20010822T201551Z' | |
474 | # expected: '20201231Z' | |
475 | ||
476 | Test::More doesn't say "Looks like you failed 1 tests of 1". That '# | |
477 | TODO' tells Test::Harness "this is supposed to fail" and it treats a | |
478 | failure as a successful test. So you can write tests even before | |
479 | you've fixed the underlying code. | |
480 | ||
481 | If a TODO test passes, Test::Harness will report it "UNEXPECTEDLY | |
482 | SUCCEEDED". When that happens, you simply remove the TODO block with | |
483 | C<local $TODO> and turn it into a real test. | |
484 | ||
485 | ||
486 | =head2 Testing with taint mode. | |
487 | ||
488 | Taint mode is a funny thing. It's the globalest of all global | |
489 | features. Once you turn it on it effects I<all> code in your program | |
490 | and I<all> modules used (and all the modules they use). If a single | |
491 | piece of code isn't taint clean, the whole thing explodes. With that | |
492 | in mind, it's very important to ensure your module works under taint | |
493 | mode. | |
494 | ||
495 | It's very simple to have your tests run under taint mode. Just throw | |
496 | a C<-T> into the C<#!> line. Test::Harness will read the switches | |
497 | in C<#!> and use them to run your tests. | |
498 | ||
499 | #!/usr/bin/perl -Tw | |
500 | ||
501 | use Test::More 'no_plan'; | |
502 | ||
503 | ...test normally here... | |
504 | ||
505 | So when you say C<make test> it will be run with taint mode and | |
506 | warnings on. | |
507 | ||
508 | ||
509 | =head1 FOOTNOTES | |
510 | ||
511 | =over 4 | |
512 | ||
513 | =item 1 | |
514 | ||
515 | The first number doesn't really mean anything, but it has to be 1. | |
516 | It's the second number that's important. | |
517 | ||
518 | =item 2 | |
519 | ||
520 | For those following along at home, I'm using version 1.31. It has | |
521 | some bugs, which is good -- we'll uncover them with our tests. | |
522 | ||
523 | =item 3 | |
524 | ||
525 | You can actually take this one step further and test the manual | |
526 | itself. Have a look at B<Test::Inline> (formerly B<Pod::Tests>). | |
527 | ||
528 | =item 4 | |
529 | ||
530 | Yes, there's a mistake in the test suite. What! Me, contrived? | |
531 | ||
532 | =item 5 | |
533 | ||
534 | We'll get to testing the contents of lists later. | |
535 | ||
536 | =item 6 | |
537 | ||
538 | But what happens if your test program dies halfway through?! Since we | |
539 | didn't say how many tests we're going to run, how can we know it | |
540 | failed? No problem, Test::More employs some magic to catch that death | |
541 | and turn the test into a failure, even if every test passed up to that | |
542 | point. | |
543 | ||
544 | =item 7 | |
545 | ||
546 | I cleaned it up a little. | |
547 | ||
548 | =item 8 | |
549 | ||
550 | Most Operating Systems record time as the number of seconds since a | |
551 | certain date. This date is the beginning of the epoch. Unix's starts | |
552 | at midnight January 1st, 1970 GMT. | |
553 | ||
554 | =item 9 | |
555 | ||
556 | MacOS's epoch is midnight January 1st, 1904. VMS's is midnight, | |
557 | November 17th, 1858, but vmsperl emulates the Unix epoch so it's not a | |
558 | problem. | |
559 | ||
560 | =item 10 | |
561 | ||
562 | As long as the code inside the SKIP block at least compiles. Please | |
563 | don't ask how. No, it's not a filter. | |
564 | ||
565 | =item 11 | |
566 | ||
567 | Do NOT be tempted to use TODO tests as a way to avoid fixing simple | |
568 | bugs! | |
569 | ||
570 | =back | |
571 | ||
572 | =head1 AUTHORS | |
573 | ||
574 | Michael G Schwern E<lt>schwern@pobox.comE<gt> and the perl-qa dancers! | |
575 | ||
576 | =head1 COPYRIGHT | |
577 | ||
578 | Copyright 2001 by Michael G Schwern E<lt>schwern@pobox.comE<gt>. | |
579 | ||
580 | This documentation is free; you can redistribute it and/or modify it | |
581 | under the same terms as Perl itself. | |
582 | ||
583 | Irrespective of its distribution, all code examples in these files | |
584 | are hereby placed into the public domain. You are permitted and | |
585 | encouraged to use this code in your own programs for fun | |
586 | or for profit as you see fit. A simple comment in the code giving | |
587 | credit would be courteous but is not required. | |
588 | ||
589 | =cut |