Initial commit of OpenSPARC T2 architecture model.

[OpenSPARC-T2-SAM] / sam-t2 / devtools / amd64 / man / man3 / Test::Harness::TAP.3

.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sh \" Subsection heading
.br
.if t .Sp
.ne 5
.PP
\fB\\$1\fR
.PP
..
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  | will give a
.\" real vertical bar.  \*(C+ will give a nicer C++.  Capital omega is used to
.\" do unbreakable dashes and therefore won't be available.  \*(C` and \*(C'
.\" expand to `' in nroff, nothing in troff, for use with C<>.
.tr \(*W-|\(bv\*(Tr
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
'br\}
.\"
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.if \nF \{\
.    de IX
.    tm Index:\\$1\t\\n%\t"\\$2"
..
.    nr % 0
.    rr F
.\}
.\"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.hy 0
.if n .na
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "Test::Harness::TAP 3"
.TH Test::Harness::TAP 3 "2001-09-21" "perl v5.8.8" "Perl Programmers Reference Guide"
.SH "NAME"
Test::Harness::TAP \- Documentation for the TAP format
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
\&\s-1TAP\s0, the Test Anything Protocol, is Perl's simple text-based interface
between testing modules such as Test::More and the test harness
Test::Harness.
.SH "TODO"
.IX Header "TODO"
Exit code of the process.
.SH "THE TAP FORMAT"
.IX Header "THE TAP FORMAT"
\&\s-1TAP\s0's general format is:
.PP
.Vb 7
\&    1..N
\&    ok 1 Description # Directive
\&    # Diagnostic
\&    ....
\&    ok 47 Description
\&    ok 48 Description
\&    more tests....
.Ve
.PP
For example, a test file's output might look like:
.PP
.Vb 5
\&    1..4
\&    ok 1 - Input file opened
\&    not ok 2 - First line of the input valid
\&    ok 3 - Read the rest of the file
\&    not ok 4 - Summarized correctly # TODO Not written yet
.Ve
.SH "HARNESS BEHAVIOR"
.IX Header "HARNESS BEHAVIOR"
In this document, the \*(L"harness\*(R" is any program analyzing \s-1TAP\s0 output.
Typically this will be Perl's \fIprove\fR program, or the underlying
\&\f(CW\*(C`Test::Harness::runtests\*(C'\fR subroutine.
.PP
A harness must only read \s-1TAP\s0 output from standard output and not
from standard error.  Lines written to standard output matching
\&\f(CW\*(C`/^(not )?ok\eb/\*(C'\fR must be interpreted as test lines.  All other
lines must not be considered test output.
.SH "TESTS LINES AND THE PLAN"
.IX Header "TESTS LINES AND THE PLAN"
.Sh "The plan"
.IX Subsection "The plan"
The plan tells how many tests will be run, or how many tests have
run.  It's a check that the test file hasn't stopped prematurely.
It must appear once, whether at the beginning or end of the output.
.PP
The plan is usually the first line of \s-1TAP\s0 output and it specifies how
many test points are to follow. For example,
.PP
.Vb 1
\&    1..10
.Ve
.PP
means you plan on running 10 tests. This is a safeguard in case your test
file dies silently in the middle of its run.  The plan is optional but if
there is a plan before the test points it must be the first non-diagnostic
line output by the test file.
.PP
In certain instances a test file may not know how many test points
it will ultimately be running. In this case the plan can be the last
non-diagnostic line in the output.
.PP
The plan cannot appear in the middle of the output, nor can it appear more
than once.
.Sh "The test line"
.IX Subsection "The test line"
The core of \s-1TAP\s0 is the test line.  A test file prints one test line test
point executed. There must be at least one test line in \s-1TAP\s0 output. Each
test line comprises the following elements:
.ie n .IP "* ""ok""\fR or \f(CW""not ok""" 4
.el .IP "* \f(CWok\fR or \f(CWnot ok\fR" 4
.IX Item "ok or not ok"
This tells whether the test point passed or failed. It must be
at the beginning of the line. \f(CW\*(C`/^not ok/\*(C'\fR indicates a failed test
point. \f(CW\*(C`/^ok/\*(C'\fR is a successful test point. This is the only mandatory
part of the line.
.Sp
Note that unlike the Directives below, \f(CW\*(C`ok\*(C'\fR and \f(CW\*(C`not ok\*(C'\fR are
case\-sensitive.
.IP "* Test number" 4
.IX Item "Test number"
\&\s-1TAP\s0 expects the \f(CW\*(C`ok\*(C'\fR or \f(CW\*(C`not ok\*(C'\fR to be followed by a test point
number. If there is no number the harness must maintain
its own counter until the script supplies test numbers again. So
the following test output
.Sp
.Vb 6
\&    1..6
\&    not ok
\&    ok
\&    not ok
\&    ok
\&    ok
.Ve
.Sp
has five tests.  The sixth is missing.  Test::Harness will generate
.Sp
.Vb 2
\&    FAILED tests 1, 3, 6
\&    Failed 3/6 tests, 50.00% okay
.Ve
.IP "* Description" 4
.IX Item "Description"
Any text after the test number but before a \f(CW\*(C`#\*(C'\fR is the description of
the test point.
.Sp
.Vb 1
\&    ok 42 this is the description of the test
.Ve
.Sp
Descriptions should not begin with a digit so that they are not confused
with the test point number.
.Sp
The harness may do whatever it wants with the description.
.IP "* Directive" 4
.IX Item "Directive"
The test point may include a directive, following a hash on the
test line.  There are currently two directives allowed: \f(CW\*(C`TODO\*(C'\fR and
\&\f(CW\*(C`SKIP\*(C'\fR.  These are discussed below.
.PP
To summarize:
.IP "* ok/not ok (required)" 4
.IX Item "ok/not ok (required)"
.PD 0
.IP "* Test number (recommended)" 4
.IX Item "Test number (recommended)"
.IP "* Description (recommended)" 4
.IX Item "Description (recommended)"
.IP "* Directive (only when necessary)" 4
.IX Item "Directive (only when necessary)"
.PD
.SH "DIRECTIVES"
.IX Header "DIRECTIVES"
Directives are special notes that follow a \f(CW\*(C`#\*(C'\fR on the test line.
Only two are currently defined: \f(CW\*(C`TODO\*(C'\fR and \f(CW\*(C`SKIP\*(C'\fR.  Note that
these two keywords are not case\-sensitive.
.Sh "\s-1TODO\s0 tests"
.IX Subsection "TODO tests"
If the directive starts with \f(CW\*(C`# TODO\*(C'\fR, the test is counted as a
todo test, and the text after \f(CW\*(C`TODO\*(C'\fR is the explanation.
.PP
.Vb 1
\&    not ok 13 # TODO bend space and time
.Ve
.PP
Note that if the \s-1TODO\s0 has an explanation it must be separated from
\&\f(CW\*(C`TODO\*(C'\fR by a space.
.PP
These tests represent a feature to be implemented or a bug to be fixed
and act as something of an executable \*(L"things to do\*(R" list.  They are
\&\fBnot\fR expected to succeed.  Should a todo test point begin succeeding,
the harness should report it as a bonus.  This indicates that whatever
you were supposed to do has been done and you should promote this to a
normal test point.
.Sh "Skipping tests"
.IX Subsection "Skipping tests"
If the directive starts with \f(CW\*(C`# SKIP\*(C'\fR, the test is counted as having
been skipped.  If the whole test file succeeds, the count of skipped
tests is included in the generated output.  The harness should report
the text after \f(CW\*(C` # SKIP\eS*\es+\*(C'\fR as a reason for skipping.
.PP
.Vb 1
\&    ok 23 # skip Insufficient flogiston pressure.
.Ve
.PP
Similarly, one can include an explanation in a plan line,
emitted if the test file is skipped completely:
.PP
.Vb 1
\&    1..0 # Skipped: WWW::Mechanize not installed
.Ve
.SH "OTHER LINES"
.IX Header "OTHER LINES"
.Sh "Bail out!"
.IX Subsection "Bail out!"
As an emergency measure a test script can decide that further tests
are useless (e.g. missing dependencies) and testing should stop
immediately. In that case the test script prints the magic words
.PP
.Vb 1
\&    Bail out!
.Ve
.PP
to standard output. Any message after these words must be displayed
by the interpreter as the reason why testing must be stopped, as
in
.PP
.Vb 1
\&    Bail out! MySQL is not running.
.Ve
.Sh "Diagnostics"
.IX Subsection "Diagnostics"
Additional information may be put into the testing output on separate
lines.  Diagnostic lines should begin with a \f(CW\*(C`#\*(C'\fR, which the harness must
ignore, at least as far as analyzing the test results.  The harness is
free, however, to display the diagnostics.  Typically diagnostics are
used to provide information about the environment in which test file is
running, or to delineate a group of tests.
.PP
.Vb 9
\&    ...
\&    ok 18 - Closed database connection
\&    # End of database section.
\&    # This starts the network part of the test.
\&    # Daemon started on port 2112
\&    ok 19 - Opened socket
\&    ...
\&    ok 47 - Closed socket
\&    # End of network tests
.Ve
.Sh "Anything else"
.IX Subsection "Anything else"
Any output line that is not a plan, a test line or a diagnostic is
incorrect.  How a harness handles the incorrect line is undefined.
Test::Harness silently ignores incorrect lines, but will become more
stringent in the future.
.SH "EXAMPLES"
.IX Header "EXAMPLES"
All names, places, and events depicted in any example are wholly
fictitious and bear no resemblance to, connection with, or relation to any
real entity. Any such similarity is purely coincidental, unintentional,
and unintended.
.Sh "Common with explanation"
.IX Subsection "Common with explanation"
The following \s-1TAP\s0 listing declares that six tests follow as well as
provides handy feedback as to what the test is about to do. All six
tests pass.
.PP
.Vb 11
\&    1..6
\&    #
\&    # Create a new Board and Tile, then place
\&    # the Tile onto the board.
\&    #
\&    ok 1 - The object isa Board
\&    ok 2 - Board size is zero
\&    ok 3 - The object isa Tile
\&    ok 4 - Get possible places to put the Tile
\&    ok 5 - Placing the tile produces no error
\&    ok 6 - Board size is 1
.Ve
.Sh "Unknown amount and failures"
.IX Subsection "Unknown amount and failures"
This hypothetical test program ensures that a handful of servers are
online and network\-accessible. Because it retrieves the hypothetical
servers from a database, it doesn't know exactly how many servers it
will need to ping. Thus, the test count is declared at the bottom after
all the test points have run. Also, two of the tests fail.
.PP
.Vb 9
\&    ok 1 - retrieving servers from the database
\&    # need to ping 6 servers
\&    ok 2 - pinged diamond
\&    ok 3 - pinged ruby
\&    not ok 4 - pinged saphire
\&    ok 5 - pinged onyx
\&    not ok 6 - pinged quartz
\&    ok 7 - pinged gold
\&    1..7
.Ve
.Sh "Giving up"
.IX Subsection "Giving up"
This listing reports that a pile of tests are going to be run. However,
the first test fails, reportedly because a connection to the database
could not be established. The program decided that continuing was
pointless and exited.
.PP
.Vb 3
\&    1..573
\&    not ok 1 - database handle
\&    Bail out! Couldn't connect to database.
.Ve
.Sh "Skipping a few"
.IX Subsection "Skipping a few"
The following listing plans on running 5 tests. However, our program
decided to not run tests 2 thru 5 at all. To properly report this,
the tests are marked as being skipped.
.PP
.Vb 7
\&    1..5
\&    ok 1 - approved operating system
\&    # $^0 is solaris
\&    ok 2 - # SKIP no /sys directory
\&    ok 3 - # SKIP no /sys directory
\&    ok 4 - # SKIP no /sys directory
\&    ok 5 - # SKIP no /sys directory
.Ve
.Sh "Skipping everything"
.IX Subsection "Skipping everything"
This listing shows that the entire listing is a skip. No tests were run.
.PP
.Vb 1
\&    1..0 # skip because English-to-French translator isn't installed
.Ve
.Sh "Got spare tuits?"
.IX Subsection "Got spare tuits?"
The following example reports that four tests are run and the last two
tests failed. However, because the failing tests are marked as things
to do later, they are considered successes. Thus, a harness should report
this entire listing as a success.
.PP
.Vb 5
\&    1..4
\&    ok 1 - Creating test program
\&    ok 2 - Test program runs, no error
\&    not ok 3 - infinite loop # TODO halting problem unsolved
\&    not ok 4 - infinite loop 2 # TODO halting problem unsolved
.Ve
.Sh "Creative liberties"
.IX Subsection "Creative liberties"
This listing shows an alternate output where the test numbers aren't
provided. The test also reports the state of a ficticious board game in
diagnostic form. Finally, the test count is reported at the end.
.PP
.Vb 23
\&    ok - created Board
\&    ok
\&    ok
\&    ok
\&    ok
\&    ok
\&    ok
\&    ok
\&    # +------+------+------+------+
\&    # |      |16G   |      |05C   |
\&    # |      |G N C |      |C C G |
\&    # |      |  G   |      |  C  +|
\&    # +------+------+------+------+
\&    # |10C   |01G   |      |03C   |
\&    # |R N G |G A G |      |C C C |
\&    # |  R   |  G   |      |  C  +|
\&    # +------+------+------+------+
\&    # |      |01G   |17C   |00C   |
\&    # |      |G A G |G N R |R N R |
\&    # |      |  G   |  R   |  G   |
\&    # +------+------+------+------+
\&    ok - board has 7 tiles + starter tile
\&    1..9
.Ve
.SH "AUTHORS"
.IX Header "AUTHORS"
Andy Lester, based on the original Test::Harness documentation by Michael Schwern.
.SH "ACKNOWLEDGEMENTS"
.IX Header "ACKNOWLEDGEMENTS"
Thanks to
Pete Krawczyk,
Paul Johnson,
Ian Langworth
and Nik Clayton
for help and contributions on this document.
.PP
The basis for the \s-1TAP\s0 format was created by Larry Wall in the
original test script for Perl 1.  Tim Bunce and Andreas Koenig
developed it further with their modifications to Test::Harness.
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 2003\-2005 by
Michael G Schwern \f(CW\*(C`<schwern@pobox.com>\*(C'\fR,
Andy Lester \f(CW\*(C`<andy@petdance.com>\*(C'\fR.
.PP
This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.
.PP
See <http://www.perl.com/perl/misc/Artistic.html>.