.\" Automatically generated by Pod::Man v1.34, Pod::Parser v1.13
.\" ========================================================================
.de Sh \" Subsection heading
.de Sp \" Vertical space (when we can't use .PP)
.de Vb \" Begin verbatim text
.de Ve \" End verbatim text
.\" Set up some character translations and predefined strings. \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote. | will give a
.\" real vertical bar. \*(C+ will give a nicer C++. Capital omega is used to
.\" do unbreakable dashes and therefore won't be available. \*(C` and \*(C'
.\" expand to `' in nroff, nothing in troff, for use with C<>.
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
. if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
. if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index
.\" entries marked with X<> in POD. Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
. tm Index:\\$1\t\\n%\t"\\$2"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear. Run. Save yourself. No user-serviceable parts.
. \" fudge factors for nroff and troff
. ds #H ((1u-(\\\\n(.fu%2u))*.13m)
. \" simple accents for nroff and troff
. ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
. ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
. ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
. ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
. ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
. ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
. \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
. \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
. \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
.\" ========================================================================
.IX Title "HTML::TableExtract 3"
.TH HTML::TableExtract 3 "2002-04-04" "perl v5.8.0" "User Contributed Perl Documentation"
HTML::TableExtract \- Perl extension for extracting the text contained in tables within an HTML document.
\& # Matched tables are returned as "table state" objects; tables can be
\& # matched using column headers, depth, count within a depth, or some
\& # combination of the three.
\& # Using column header information. Assume an HTML document with
\& # tables that have "Date", "Price", and "Cost" somewhere in a
\& # row. The columns beneath those headings are what you want to
\& # extract. They will be returned in the same order as you specified
\& # the headers since 'automap' is enabled by default.
\& use HTML::TableExtract;
\& $te = new HTML::TableExtract( headers => [qw(Date Price Cost)] );
\& $te->parse($html_string);
\& # Examine all matching tables
\& foreach $ts ($te->table_states) {
\& print "Table (", join(',', $ts->coords), "):\en";
\& foreach $row ($ts->rows) {
\& print join(',', @$row), "\en";
\& # Old style, using top level methods rather than table state objects.
\& foreach $table ($te->tables) {
\& print "Table (", join(',', $te->table_coords($table)), "):\en";
\& foreach $row ($te->rows($table)) {
\& print join(',', @$row), "\en";
\& # Shorthand...top level rows() method assumes the first table found
\& # in the document if no arguments are supplied.
\& foreach $row ($te->rows) {
\& print join(',', @$row), "\en";
\& # Using depth and count information. Every table in the document has
\& # a unique depth and count tuple, so when both are specified it is a
\& # unique table. Depth and count both begin with 0, so in this case we
\& # are looking for a table (depth 2) within a table (depth 1) within a
\& # table (depth 0, which is the top level HTML document). In addition,
\& # it must be the third (count 2) such instance of a table at that
\& $te = new HTML::TableExtract( depth => 2, count => 2 );
\& $te->parse($html_string);
\& foreach $ts ($te->table_states) {
\& print "Table found at ", join(',', $ts->coords), ":\en";
\& foreach $row ($ts->rows) {
\& print " ", join(',', @$row), "\en";
HTML::TableExtract is a subclass of HTML::Parser that serves to
extract the textual information from tables of interest contained
within an \s-1HTML\s0 document. The text from each extracted table is stored
in tabe state objects which hold the information as an array of arrays
that represent the rows and cells of that table.
There are three constraints available to specify which tables you
would like to extract from a document: \fIHeaders\fR, \fIDepth\fR, and
\&\fIHeaders\fR, the most flexible and adaptive of the techniques, involves
specifying text in an array that you expect to appear above the data
in the tables of interest. Once all headers have been located in a row
of that table, all further cells beneath the columns that matched your
headers are extracted. All other columns are ignored: think of it as
vertical slices through a table. In addition, TableExtract
automatically rearranges each row in the same order as the headers you
provided. If you would like to disable this, set \fIautomap\fR to 0
during object creation, and instead rely on the \fIcolumn_map()\fR method to
find out the order in which the headers were found. Furthermore,
TableExtract will automatically compensate for cell span issues so
that columns are really the same columns as you would visually see in
a browser. This behavior can be disabled by setting the \fIgridmap\fR
parameter to 0. \s-1HTML\s0 is stripped from the entire textual content of a
cell before header matches are attempted \*(-- unless the \fIkeep_html\fR
\&\fIDepth\fR and \fICount\fR are more specific ways to specify tables in
relation to one another. \fIDepth\fR represents how deeply a table
resides in other tables. The depth of a top-level table in the
document is 0. A table within a top-level table has a depth of 1, and
so on. Each depth can be thought of as a layer; tables sharing the
same depth are on the same layer. Within each of these layers,
\&\fICount\fR represents the order in which a table was seen at that depth,
starting with 0. Providing both a \fIdepth\fR and a \fIcount\fR will
uniquely specify a table within a document.
Each of the \fIHeaders\fR, \fIDepth\fR, and \fICount\fR specifications are
cumulative in their effect on the overall extraction. For instance, if
you specify only a \fIDepth\fR, then you get all tables at that depth
(note that these could very well reside in separate higher-level
tables throughout the document since depth extends across tables). If
you specify only a \fICount\fR, then the tables at that \fICount\fR from all
depths are returned (i.e., the \fIn\fRth occurrence of a table at each
depth). If you only specify \fIHeaders\fR, then you get all tables in the
document containing those column headers. If you have specified
multiple constraints of \fIHeaders\fR, \fIDepth\fR, and \fICount\fR, then each
constraint has veto power over whether a particular table is
If no \fIHeaders\fR, \fIDepth\fR, or \fICount\fR are specified, then all
Text that is gathered from the tables is decoded with HTML::Entities
by default; this can be disabled by setting the \fIdecode\fR parameter to
Make sure you fully understand the notions of \fIdepth\fR and \fIcount\fR
before proceeding, because it is about to become a bit more involved.
Table matches using \fIHeaders\fR, \fIDepth\fR, or \fICount\fR can be chained
together in order to further specify tables relative to one
another. Links in chains are successively applied to tables within
tables. Top level constraints (i.e., \fIheader\fR, \fIdepth\fR, and \fIcount\fR
parameters for the TableExtract object) behave as the first link in
the chain. Additional links are specified using the \fIchain\fR
parameter. Each link in the chain has its own set of constraints. For
\& $te = new HTML::TableExtract
\& headers => [qw(Summary Region)],
\& { depth => 0, count => 2 },
\& { headers => [qw(Part Qty Cost)] }
The matching process in this case will start with \fBall\fR tables in the
document that have \*(L"Summary\*(R" and \*(L"Region\*(R" in their headers. For now,
assume that there was only one table that matched these headers. Each
table contained within that table will be compared to the first link
in the chain. Depth 0 means that a matching table must be immediately
contained within the current table; count 2 means that the matching
table must also be the third at that depth (counts and depths start at
0). In other words, the next link of the chain will match on the
third table immediately contained within our first matched table. Once
this link matches, then \fBall\fR further tables beneath that table that
have \*(L"Part\*(R", \*(L"Qty\*(R", and \*(L"Cost\*(R" in their headers will match. By
default, it is only tables at the end of the chains that are returned
to the application, so these tables are returned.
Each time a link in a chain matches a table, an additional context for
\&\fIdepth\fR and \fIcount\fR is established. It is perhaps easiest to
visualize a \fIcontext\fR as a brand-new \s-1HTML\s0 document, with new depths
and counts to compare to the remaining links in the chain. The top
level \s-1HTML\s0 document is the first context. Each table in the document
establishes a new context. \fIDepth\fR in a chain link is relative to the
context that the matching table creates (i.e., a link depth of 0 would
be a table immediately contained within the table that matched the
prior link in the chain). Likewise, that same context keeps track of
\&\fIcounts\fR within the new depth scheme for comparison to the remaining
links in the chain. Headers still apply if they are present in a link,
but they are always independent of context.
As it turns out, specifying a depth and count provides a unique
address for a table within a context. For non-unique constraints, such
as just a depth, or headers, there can be multiple matches for a given
link. In these cases the chain \*(L"forks\*(R" and attempts to make further
matches within each of these tables.
By default, chains are \fIelastic\fR. This means that when a particular
link does not match on a table, it is passed down to subtables
\& $te = new HTML::TableExtract
\& headers => [qw(Summary Region)],
\& { headers => [qw(Part Qty Cost)] }
If there are intervening tables between the two header queries, they
will be ignored; this query will extract all tables with \*(L"Part\*(R",
\&\*(L"Qty\*(R", and \*(L"Cost\*(R" in the headers that are contained in any table with
\&\*(L"Summary\*(R" and \*(L"Region\*(R" in its headers, regardless of how embedded the
inner tables are. If you want a chain to be inelastic, you can set the
\&\fIelastic\fR parameter to 0 for the whole TableExtract object. Using the
\& $te = new HTML::TableExtract
\& headers => [qw(Summary Region)],
\& { headers => [qw(Part Qty Cost)] }
In this case, the inner table (Part, Qty, Cost) must be \fBimmediately\fR
contained within the outer table (Summary, Region) in order for the
match to take place. This is equivalent to specifying a depth of 0 for
each link in the chain; if you only want particular links to be
inelastic, then simply set their depths to 0.
By default, only tables that match at the end of the chains are
retained. The intermediate matches along the chain are referred to as
\&\fIwaypoints\fR, and are not extracted by default. A waypoint may be
retained, however, by specifiying the \fIkeep\fR parameter in that link
of the chain. This parameter may be specified at the top level as well
if you want to keep tables that match the first set of constraints in
the object. If you want to keep all tables that match along the chain,
the specify the \fIkeepall\fR parameter at the top level.
Are chains overkill? Probably. In reality, nested \s-1HTML\s0 tables tend not
to be very deep, so there will usually not be much need for lots of
links in a chain. Theoretically, however, chains offer precise
targeting of tables relative to one another, no matter how deeply
.IX Subsection "Pop Quiz"
What happens with the following table extraction?
\& $te = new HTML::TableExtract(
\& chain => [ { depth => 0 } ],
Answer: All tables that are contained in another table are extracted
from the document. In this case, there were no top-level constraints
specified, which if you recall means that \fBall\fR tables match the
first set of constraints (or non\-constraints, in this case!). A depth
of 0 in the next link of the chain means that the matching table must
be immediately contained within the table from a prior match.
The following is equivalent:
\& $te = new HTML::TableExtract(
The \fIsubtables\fR parameter tells TableExtract to scoop up all tables
contained within the matching tables. In conjunction with a depth of
1, this has the affect of discarding all top-level tables in the
document, which is exactly what occurred in the prior example.
The main point of this module was to provide a flexible method of
extracting tabular information from \s-1HTML\s0 documents without relying to
heavily on the document layout. For that reason, I suggest using
\&\fIHeaders\fR whenever possible \*(-- that way, you are anchoring your
extraction on what the document is trying to communicate rather than
some feature of the \s-1HTML\s0 comprising the document (other than the fact
that the data is contained in a table).
HTML::TableExtract is a subclass of HTML::Parser, and as such inherits
all of its basic methods. In particular, \f(CW\*(C`start()\*(C'\fR, \f(CW\*(C`end()\*(C'\fR, and
\&\f(CW\*(C`text()\*(C'\fR are utilized. Feel free to override them, but if you do not
eventually invoke them in the \s-1SUPER\s0 class with some content, results
The following are the top-level methods of the HTML::TableExtract
object. Tables that have matched a query are actually returned as
separate objects of type HTML::TableExtract::TableState. These table
state objects have their own methods, documented further below. There
are some top-level methods that are present for convenience and
backwards compatibility that are nothing more than front-ends for
equivalent table state methods.
.IX Subsection "Constructor"
Return a new HTML::TableExtract object. Valid attributes are:
Passed as an array reference, headers specify strings of interest at
the top of columns within targeted tables. These header strings will
eventually be passed through a non\-anchored, case-insensitive regular
expression, so regexp special characters are allowed. The table row
containing the headers is \fBnot\fR returned. Columns that are not
beneath one of the provided headers will be ignored. Columns will, by
default, be rearranged into the same order as the headers you provide
(see the \fIautomap\fR parameter for more information). Additionally, by
default columns are considered what you would see visually beneath
that header when the table is rendered in a browser. See the
\&\fIgridmap\fR parameter for more information. \s-1HTML\s0 within a header is
stripped before the match is attempted, unless the \fBkeep_html\fR
Specify how embedded in other tables your tables of interest should
be. Top-level tables in the \s-1HTML\s0 document have a depth of 0, tables
within top-level tables have a depth of 1, and so on.
Specify which table within each depth you are interested in, beginning
List of additional constraints to be matched sequentially from the top
level constraints. This is a reference to an array of hash
references. Each hash is a link in the chain, and can be specified in
terms of \fIdepth\fR, \fIcount\fR, and \fIheaders\fR. Further modifiers include
\&\fIkeep\fR, which means to retain the table if it would normally be
Automatically applies the ordering reported by \fIcolumn_map()\fR to the
rows returned by \fIrows()\fR. This only makes a difference if you have
specified \fIHeaders\fR and they turn out to be in a different order in
the table than what you specified. Automap will rearrange the columns
in the same order as the headers appear. To get the original ordering,
you will need to take another slice of each row using
\&\fIcolumn_map()\fR. \fIautomap\fR is enabled by default.
Controls whether the table contents are returned as a grid or a
tree. \s-1ROWSPAN\s0 and \s-1COLSPAN\s0 issues are compensated for, and columns
really are columns. Empty phantom cells are created where they would
have been obscured by \s-1ROWSPAN\s0 or \s-1COLSPAN\s0 settings. This really becomes
an issue when extracting columns beneath headers. Enabled by default.
Keep all tables that matched along a chain, including tables matched
by top level contraints. By default, waypoints are dropped and only
the matches at the end of the chain are retained. To retain a
particular waypoint along a chain, use the \fIkeep\fR parameter in that
When set to 0, all links in chains will be treated as though they had
a depth of 0 specified, which means there can be no intervening
unmatched tables between matches on links.
Extract all tables within matched tables.
Automatically decode retrieved text with
\&\fIHTML::Entities::decode_entities()\fR. Enabled by default.
Translate <br> tags into newlines. Sometimes the remaining text can be
hard to parse if the <br> tag is simply dropped. Enabled by default.
Has no effect if \fIkeep_html\fR is enabled.
Return the raw \s-1HTML\s0 contained in the cell, rather than just the
visible text. Embedded tables are \fBnot\fR retained in the \s-1HTML\s0
extracted from a cell. Patterns for header matches must take into
account \s-1HTML\s0 in the string if this option is enabled.
Prints some debugging information to \s-1STDOUT\s0, more for higher values.
.IX Subsection "Regular Methods"
Returns all depths that contained matched tables in the document.
.IX Item "counts($depth)"
For a particular depth, returns all counts that contained matched
.ie n .IP "table_state($depth, $count)" 4
.el .IP "table_state($depth, \f(CW$count\fR)" 4
.IX Item "table_state($depth, $count)"
For a particular depth and count, return the table state object for
.IP "\fItable_states()\fR" 4
.IX Item "table_states()"
Return table state objects for all tables that matched.
.IP "\fIfirst_table_state_found()\fR" 4
.IX Item "first_table_state_found()"
Return the table state object for the first table matched in the
.Sh "\s-1TABLE\s0 \s-1STATE\s0 \s-1METHODS\s0"
.IX Subsection "TABLE STATE METHODS"
The following methods are invoked from an
HTML::TableExtract::TableState object, such as those returned from the
\&\f(CW\*(C`table_states()\*(C'\fR method.
Return all rows within a matched table. Each row returned is a
reference to an array containing the text of each cell.
Return the (absolute) depth at which this table was found.
Return the count for this table within the depth it was found.
Return depth and count in a list.
.IP "\fIcolumn_map()\fR" 4
Return the order (via indices) in which the provided headers were
found. These indices can be used as slices on rows to either order the
rows in the same order as headers or restore the rows to their natural
order, depending on whether the rows have been pre-adjusted using the
\&\fIautomap\fR parameter.
Returns the path of matched tables that led to matching this
table. Lineage only makes sense if chains were used. Tables that were
not matched by a link in the chain are not included in lineage. The
lineage path is a list of array refs containing depth and count values
.IX Subsection "Procedural Methods"
The following top level methods are alternatives to invoking methods
in a table state object. If you do not want to deal with table state
objects, then these methods are for you. The \*(L"tables\*(R" they deal in are
actually just arrays of arrays, which happen to be the current
internal data structure of the table state objects. They are here for
.ie n .IP "table($depth, $count)" 4
.el .IP "table($depth, \f(CW$count\fR)" 4
.IX Item "table($depth, $count)"
Same as \f(CW\*(C`table_state()\*(C'\fR, but returns the internal data structure
rather than the table state object.
Same as \f(CW\*(C`table_states()\*(C'\fR, but returns the data structures rather than
.IP "\fIfirst_table_found()\fR" 4
.IX Item "first_table_found()"
Same as \f(CW\*(C`first_table_state_found()\*(C'\fR, except returns the data
structure for first table that matched.
.IP "table_coords($table)" 4
.IX Item "table_coords($table)"
Returns the depth and count for a particular table data structure. See
the \f(CW\*(C`coords()\*(C'\fR method provided by table state objects.
Return a lsit of the rows for a particular table data structure (first
table found by default). See the \f(CW\*(C`rows()\*(C'\fR method provided by table
.IP "\fIcolumn_map()\fR" 4
.IP "column_map($table)" 4
.IX Item "column_map($table)"
Return the column map for a particular table data structure (first
found by default). See the \f(CW\*(C`column_map()\*(C'\fR method provided by table
\&\fIHTML::Parser\fR\|(3), \fIHTML::Entities\fR\|(3)
Matthew P. Sisk, <\fIsisk@mojotoad.com\fR>
Copyright (c) 2000\-2002 Matthew P. Sisk.
All rights reserved. All wrongs revenged. This program is free
software; you can redistribute it and/or modify it under the
same terms as Perl itself.
\&\fIHTML::Parser\fR\|(3), \fIperl\fR\|(1).
projects / OpenSPARC-T2-DV / blob