Commit | Line | Data |
---|---|---|
920dae64 AT |
1 | +++++++++++++++++++++++++++++++ |
2 | Writing Python Regression Tests | |
3 | +++++++++++++++++++++++++++++++ | |
4 | ||
5 | :Author: Skip Montanaro | |
6 | :Contact: skip@mojam.com | |
7 | ||
8 | Introduction | |
9 | ============ | |
10 | ||
11 | If you add a new module to Python or modify the functionality of an existing | |
12 | module, you should write one or more test cases to exercise that new | |
13 | functionality. There are different ways to do this within the regression | |
14 | testing facility provided with Python; any particular test should use only | |
15 | one of these options. Each option requires writing a test module using the | |
16 | conventions of the selected option: | |
17 | ||
18 | - PyUnit_ based tests | |
19 | - doctest_ based tests | |
20 | - "traditional" Python test modules | |
21 | ||
22 | Regardless of the mechanics of the testing approach you choose, | |
23 | you will be writing unit tests (isolated tests of functions and objects | |
24 | defined by the module) using white box techniques. Unlike black box | |
25 | testing, where you only have the external interfaces to guide your test case | |
26 | writing, in white box testing you can see the code being tested and tailor | |
27 | your test cases to exercise it more completely. In particular, you will be | |
28 | able to refer to the C and Python code in the CVS repository when writing | |
29 | your regression test cases. | |
30 | ||
31 | .. _PyUnit: | |
32 | .. _unittest: http://www.python.org/doc/current/lib/module-unittest.html | |
33 | .. _doctest: http://www.python.org/doc/current/lib/module-doctest.html | |
34 | ||
35 | PyUnit based tests | |
36 | ------------------ | |
37 | The PyUnit_ framework is based on the ideas of unit testing as espoused | |
38 | by Kent Beck and the `Extreme Programming`_ (XP) movement. The specific | |
39 | interface provided by the framework is tightly based on the JUnit_ | |
40 | Java implementation of Beck's original SmallTalk test framework. Please | |
41 | see the documentation of the unittest_ module for detailed information on | |
42 | the interface and general guidelines on writing PyUnit based tests. | |
43 | ||
44 | The test_support helper module provides two functions for use by | |
45 | PyUnit based tests in the Python regression testing framework: | |
46 | ||
47 | - ``run_unittest()`` takes a ``unittest.TestCase`` derived class as a | |
48 | parameter and runs the tests defined in that class | |
49 | ||
50 | - ``run_suite()`` takes a populated ``TestSuite`` instance and runs the | |
51 | tests | |
52 | ||
53 | ``run_suite()`` is preferred because unittest files typically grow multiple | |
54 | test classes, and you might as well be prepared. | |
55 | ||
56 | All test methods in the Python regression framework have names that | |
57 | start with "``test_``" and use lower-case names with words separated with | |
58 | underscores. | |
59 | ||
60 | Test methods should *not* have docstrings! The unittest module prints | |
61 | the docstring if there is one, but otherwise prints the function name | |
62 | and the full class name. When there's a problem with a test, the | |
63 | latter information makes it easier to find the source for the test | |
64 | than the docstring. | |
65 | ||
66 | All PyUnit-based tests in the Python test suite use boilerplate that | |
67 | looks like this (with minor variations):: | |
68 | ||
69 | import unittest | |
70 | from test import test_support | |
71 | ||
72 | class MyTestCase1(unittest.TestCase): | |
73 | ||
74 | # Define setUp and tearDown only if needed | |
75 | ||
76 | def setUp(self): | |
77 | unittest.TestCase.setUp(self) | |
78 | ... additional initialization... | |
79 | ||
80 | def tearDown(self): | |
81 | ... additional finalization... | |
82 | unittest.TestCase.tearDown(self) | |
83 | ||
84 | def test_feature_one(self): | |
85 | # Testing feature one | |
86 | ...unit test for feature one... | |
87 | ||
88 | def test_feature_two(self): | |
89 | # Testing feature two | |
90 | ...unit test for feature two... | |
91 | ||
92 | ...etc... | |
93 | ||
94 | class MyTestCase2(unittest.TestCase): | |
95 | ...same structure as MyTestCase1... | |
96 | ||
97 | ...etc... | |
98 | ||
99 | def test_main(): | |
100 | suite = unittest.TestSuite() | |
101 | suite.addTest(unittest.makeSuite(MyTestCase1)) | |
102 | suite.addTest(unittest.makeSuite(MyTestCase2)) | |
103 | ...add more suites... | |
104 | test_support.run_suite(suite) | |
105 | ||
106 | if __name__ == "__main__": | |
107 | test_main() | |
108 | ||
109 | This has the advantage that it allows the unittest module to be used | |
110 | as a script to run individual tests as well as working well with the | |
111 | regrtest framework. | |
112 | ||
113 | .. _Extreme Programming: http://www.extremeprogramming.org/ | |
114 | .. _JUnit: http://www.junit.org/ | |
115 | ||
116 | doctest based tests | |
117 | ------------------- | |
118 | Tests written to use doctest_ are actually part of the docstrings for | |
119 | the module being tested. Each test is written as a display of an | |
120 | interactive session, including the Python prompts, statements that would | |
121 | be typed by the user, and the output of those statements (including | |
122 | tracebacks, although only the exception msg needs to be retained then). | |
123 | The module in the test package is simply a wrapper that causes doctest | |
124 | to run over the tests in the module. The test for the difflib module | |
125 | provides a convenient example:: | |
126 | ||
127 | import difflib | |
128 | from test import test_support | |
129 | test_support.run_doctest(difflib) | |
130 | ||
131 | If the test is successful, nothing is written to stdout (so you should not | |
132 | create a corresponding output/test_difflib file), but running regrtest | |
133 | with -v will give a detailed report, the same as if passing -v to doctest. | |
134 | ||
135 | A second argument can be passed to run_doctest to tell doctest to search | |
136 | ``sys.argv`` for -v instead of using test_support's idea of verbosity. This | |
137 | is useful for writing doctest-based tests that aren't simply running a | |
138 | doctest'ed Lib module, but contain the doctests themselves. Then at | |
139 | times you may want to run such a test directly as a doctest, independent | |
140 | of the regrtest framework. The tail end of test_descrtut.py is a good | |
141 | example:: | |
142 | ||
143 | def test_main(verbose=None): | |
144 | from test import test_support, test_descrtut | |
145 | test_support.run_doctest(test_descrtut, verbose) | |
146 | ||
147 | if __name__ == "__main__": | |
148 | test_main(1) | |
149 | ||
150 | If run via regrtest, ``test_main()`` is called (by regrtest) without | |
151 | specifying verbose, and then test_support's idea of verbosity is used. But | |
152 | when run directly, ``test_main(1)`` is called, and then doctest's idea of | |
153 | verbosity is used. | |
154 | ||
155 | See the documentation for the doctest module for information on | |
156 | writing tests using the doctest framework. | |
157 | ||
158 | "traditional" Python test modules | |
159 | --------------------------------- | |
160 | The mechanics of how the "traditional" test system operates are fairly | |
161 | straightforward. When a test case is run, the output is compared with the | |
162 | expected output that is stored in .../Lib/test/output. If the test runs to | |
163 | completion and the actual and expected outputs match, the test succeeds, if | |
164 | not, it fails. If an ``ImportError`` or ``test_support.TestSkipped`` error | |
165 | is raised, the test is not run. | |
166 | ||
167 | Executing Test Cases | |
168 | ==================== | |
169 | If you are writing test cases for module spam, you need to create a file | |
170 | in .../Lib/test named test_spam.py. In addition, if the tests are expected | |
171 | to write to stdout during a successful run, you also need to create an | |
172 | expected output file in .../Lib/test/output named test_spam ("..." | |
173 | represents the top-level directory in the Python source tree, the directory | |
174 | containing the configure script). If needed, generate the initial version | |
175 | of the test output file by executing:: | |
176 | ||
177 | ./python Lib/test/regrtest.py -g test_spam.py | |
178 | ||
179 | from the top-level directory. | |
180 | ||
181 | Any time you modify test_spam.py you need to generate a new expected | |
182 | output file. Don't forget to desk check the generated output to make sure | |
183 | it's really what you expected to find! All in all it's usually better | |
184 | not to have an expected-out file (note that doctest- and unittest-based | |
185 | tests do not). | |
186 | ||
187 | To run a single test after modifying a module, simply run regrtest.py | |
188 | without the -g flag:: | |
189 | ||
190 | ./python Lib/test/regrtest.py test_spam.py | |
191 | ||
192 | While debugging a regression test, you can of course execute it | |
193 | independently of the regression testing framework and see what it prints:: | |
194 | ||
195 | ./python Lib/test/test_spam.py | |
196 | ||
197 | To run the entire test suite: | |
198 | ||
199 | - [UNIX, + other platforms where "make" works] Make the "test" target at the | |
200 | top level:: | |
201 | ||
202 | make test | |
203 | ||
204 | - [WINDOWS] Run rt.bat from your PCBuild directory. Read the comments at | |
205 | the top of rt.bat for the use of special -d, -O and -q options processed | |
206 | by rt.bat. | |
207 | ||
208 | - [OTHER] You can simply execute the two runs of regrtest (optimized and | |
209 | non-optimized) directly:: | |
210 | ||
211 | ./python Lib/test/regrtest.py | |
212 | ./python -O Lib/test/regrtest.py | |
213 | ||
214 | But note that this way picks up whatever .pyc and .pyo files happen to be | |
215 | around. The makefile and rt.bat ways run the tests twice, the first time | |
216 | removing all .pyc and .pyo files from the subtree rooted at Lib/. | |
217 | ||
218 | Test cases generate output based upon values computed by the test code. | |
219 | When executed, regrtest.py compares the actual output generated by executing | |
220 | the test case with the expected output and reports success or failure. It | |
221 | stands to reason that if the actual and expected outputs are to match, they | |
222 | must not contain any machine dependencies. This means your test cases | |
223 | should not print out absolute machine addresses (e.g. the return value of | |
224 | the id() builtin function) or floating point numbers with large numbers of | |
225 | significant digits (unless you understand what you are doing!). | |
226 | ||
227 | ||
228 | Test Case Writing Tips | |
229 | ====================== | |
230 | Writing good test cases is a skilled task and is too complex to discuss in | |
231 | detail in this short document. Many books have been written on the subject. | |
232 | I'll show my age by suggesting that Glenford Myers' `"The Art of Software | |
233 | Testing"`_, published in 1979, is still the best introduction to the subject | |
234 | available. It is short (177 pages), easy to read, and discusses the major | |
235 | elements of software testing, though its publication predates the | |
236 | object-oriented software revolution, so doesn't cover that subject at all. | |
237 | Unfortunately, it is very expensive (about $100 new). If you can borrow it | |
238 | or find it used (around $20), I strongly urge you to pick up a copy. | |
239 | ||
240 | The most important goal when writing test cases is to break things. A test | |
241 | case that doesn't uncover a bug is much less valuable than one that does. | |
242 | In designing test cases you should pay attention to the following: | |
243 | ||
244 | * Your test cases should exercise all the functions and objects defined | |
245 | in the module, not just the ones meant to be called by users of your | |
246 | module. This may require you to write test code that uses the module | |
247 | in ways you don't expect (explicitly calling internal functions, for | |
248 | example - see test_atexit.py). | |
249 | ||
250 | * You should consider any boundary values that may tickle exceptional | |
251 | conditions (e.g. if you were writing regression tests for division, | |
252 | you might well want to generate tests with numerators and denominators | |
253 | at the limits of floating point and integer numbers on the machine | |
254 | performing the tests as well as a denominator of zero). | |
255 | ||
256 | * You should exercise as many paths through the code as possible. This | |
257 | may not always be possible, but is a goal to strive for. In | |
258 | particular, when considering if statements (or their equivalent), you | |
259 | want to create test cases that exercise both the true and false | |
260 | branches. For loops, you should create test cases that exercise the | |
261 | loop zero, one and multiple times. | |
262 | ||
263 | * You should test with obviously invalid input. If you know that a | |
264 | function requires an integer input, try calling it with other types of | |
265 | objects to see how it responds. | |
266 | ||
267 | * You should test with obviously out-of-range input. If the domain of a | |
268 | function is only defined for positive integers, try calling it with a | |
269 | negative integer. | |
270 | ||
271 | * If you are going to fix a bug that wasn't uncovered by an existing | |
272 | test, try to write a test case that exposes the bug (preferably before | |
273 | fixing it). | |
274 | ||
275 | * If you need to create a temporary file, you can use the filename in | |
276 | ``test_support.TESTFN`` to do so. It is important to remove the file | |
277 | when done; other tests should be able to use the name without cleaning | |
278 | up after your test. | |
279 | ||
280 | .. _"The Art of Software Testing": | |
281 | http://www.amazon.com/exec/obidos/ISBN=0471043281 | |
282 | ||
283 | Regression Test Writing Rules | |
284 | ============================= | |
285 | Each test case is different. There is no "standard" form for a Python | |
286 | regression test case, though there are some general rules (note that | |
287 | these mostly apply only to the "classic" tests; unittest_- and doctest_- | |
288 | based tests should follow the conventions natural to those frameworks):: | |
289 | ||
290 | * If your test case detects a failure, raise ``TestFailed`` (found in | |
291 | ``test.test_support``). | |
292 | ||
293 | * Import everything you'll need as early as possible. | |
294 | ||
295 | * If you'll be importing objects from a module that is at least | |
296 | partially platform-dependent, only import those objects you need for | |
297 | the current test case to avoid spurious ``ImportError`` exceptions | |
298 | that prevent the test from running to completion. | |
299 | ||
300 | * Print all your test case results using the ``print`` statement. For | |
301 | non-fatal errors, print an error message (or omit a successful | |
302 | completion print) to indicate the failure, but proceed instead of | |
303 | raising ``TestFailed``. | |
304 | ||
305 | * Use ``assert`` sparingly, if at all. It's usually better to just print | |
306 | what you got, and rely on regrtest's got-vs-expected comparison to | |
307 | catch deviations from what you expect. ``assert`` statements aren't | |
308 | executed at all when regrtest is run in -O mode; and, because they | |
309 | cause the test to stop immediately, can lead to a long & tedious | |
310 | test-fix, test-fix, test-fix, ... cycle when things are badly broken | |
311 | (and note that "badly broken" often includes running the test suite | |
312 | for the first time on new platforms or under new implementations of | |
313 | the language). | |
314 | ||
315 | Miscellaneous | |
316 | ============= | |
317 | There is a test_support module in the test package you can import for | |
318 | your test case. Import this module using either:: | |
319 | ||
320 | import test.test_support | |
321 | ||
322 | or:: | |
323 | ||
324 | from test import test_support | |
325 | ||
326 | test_support provides the following useful objects: | |
327 | ||
328 | * ``TestFailed`` - raise this exception when your regression test detects | |
329 | a failure. | |
330 | ||
331 | * ``TestSkipped`` - raise this if the test could not be run because the | |
332 | platform doesn't offer all the required facilities (like large | |
333 | file support), even if all the required modules are available. | |
334 | ||
335 | * ``ResourceDenied`` - this is raised when a test requires a resource that | |
336 | is not available. Primarily used by 'requires'. | |
337 | ||
338 | * ``verbose`` - you can use this variable to control print output. Many | |
339 | modules use it. Search for "verbose" in the test_*.py files to see | |
340 | lots of examples. | |
341 | ||
342 | * ``forget(module_name)`` - attempts to cause Python to "forget" that it | |
343 | loaded a module and erase any PYC files. | |
344 | ||
345 | * ``is_resource_enabled(resource)`` - Returns a boolean based on whether | |
346 | the resource is enabled or not. | |
347 | ||
348 | * ``requires(resource [, msg])`` - if the required resource is not | |
349 | available the ResourceDenied exception is raised. | |
350 | ||
351 | * ``verify(condition, reason='test failed')``. Use this instead of:: | |
352 | ||
353 | assert condition[, reason] | |
354 | ||
355 | ``verify()`` has two advantages over ``assert``: it works even in -O | |
356 | mode, and it raises ``TestFailed`` on failure instead of | |
357 | ``AssertionError``. | |
358 | ||
359 | * ``have_unicode`` - true if Unicode is available, false otherwise. | |
360 | ||
361 | * ``is_jython`` - true if the interpreter is Jython, false otherwise. | |
362 | ||
363 | * ``TESTFN`` - a string that should always be used as the filename when | |
364 | you need to create a temp file. Also use ``try``/``finally`` to | |
365 | ensure that your temp files are deleted before your test completes. | |
366 | Note that you cannot unlink an open file on all operating systems, so | |
367 | also be sure to close temp files before trying to unlink them. | |
368 | ||
369 | * ``sortdict(dict)`` - acts like ``repr(dict.items())``, but sorts the | |
370 | items first. This is important when printing a dict value, because | |
371 | the order of items produced by ``dict.items()`` is not defined by the | |
372 | language. | |
373 | ||
374 | * ``findfile(file)`` - you can call this function to locate a file | |
375 | somewhere along sys.path or in the Lib/test tree - see | |
376 | test_linuxaudiodev.py for an example of its use. | |
377 | ||
378 | * ``fcmp(x,y)`` - you can call this function to compare two floating | |
379 | point numbers when you expect them to only be approximately equal | |
380 | withing a fuzz factor (``test_support.FUZZ``, which defaults to 1e-6). | |
381 | ||
382 | * ``check_syntax(statement)`` - make sure that the statement is *not* | |
383 | correct Python syntax. | |
384 | ||
385 | ||
386 | Python and C statement coverage results are currently available at | |
387 | ||
388 | http://www.musi-cal.com/~skip/python/Python/dist/src/ | |
389 | ||
390 | As of this writing (July, 2000) these results are being generated nightly. | |
391 | You can refer to the summaries and the test coverage output files to see | |
392 | where coverage is adequate or lacking and write test cases to beef up the | |
393 | coverage. | |
394 | ||
395 | Some Non-Obvious regrtest Features | |
396 | ================================== | |
397 | * Automagic test detection: When you create a new test file | |
398 | test_spam.py, you do not need to modify regrtest (or anything else) | |
399 | to advertise its existence. regrtest searches for and runs all | |
400 | modules in the test directory with names of the form test_xxx.py. | |
401 | ||
402 | * Miranda output: If, when running test_spam.py, regrtest does not | |
403 | find an expected-output file test/output/test_spam, regrtest | |
404 | pretends that it did find one, containing the single line | |
405 | ||
406 | test_spam | |
407 | ||
408 | This allows new tests that don't expect to print anything to stdout | |
409 | to not bother creating expected-output files. | |
410 | ||
411 | * Two-stage testing: To run test_spam.py, regrtest imports test_spam | |
412 | as a module. Most tests run to completion as a side-effect of | |
413 | getting imported. After importing test_spam, regrtest also executes | |
414 | ``test_spam.test_main()``, if test_spam has a ``test_main`` attribute. | |
415 | This is rarely required with the "traditional" Python tests, and | |
416 | you shouldn't create a module global with name test_main unless | |
417 | you're specifically exploiting this gimmick. This usage does | |
418 | prove useful with PyUnit-based tests as well, however; defining | |
419 | a ``test_main()`` which is run by regrtest and a script-stub in the | |
420 | test module ("``if __name__ == '__main__': test_main()``") allows | |
421 | the test to be used like any other Python test and also work | |
422 | with the unittest.py-as-a-script approach, allowing a developer | |
423 | to run specific tests from the command line. |