.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32
.\" ========================================================================
.de Sh \" Subsection heading
.de Sp \" Vertical space (when we can't use .PP)
.de Vb \" Begin verbatim text
.de Ve \" End verbatim text
.\" Set up some character translations and predefined strings. \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote. | will give a
.\" real vertical bar. \*(C+ will give a nicer C++. Capital omega is used to
.\" do unbreakable dashes and therefore won't be available. \*(C` and \*(C'
.\" expand to `' in nroff, nothing in troff, for use with C<>.
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
. if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
. if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index
.\" entries marked with X<> in POD. Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
. tm Index:\\$1\t\\n%\t"\\$2"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear. Run. Save yourself. No user-serviceable parts.
. \" fudge factors for nroff and troff
. ds #H ((1u-(\\\\n(.fu%2u))*.13m)
. \" simple accents for nroff and troff
. ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
. ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
. ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
. ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
. ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
. ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
. \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
. \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
. \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
.\" ========================================================================
.IX Title "I18N::LangTags 3"
.TH I18N::LangTags 3 "2001-09-21" "perl v5.8.8" "Perl Programmers Reference Guide"
I18N::LangTags \- functions for dealing with RFC3066\-style language tags
\&...or specify whichever of those functions you want to import, like so:
\& use I18N::LangTags qw(implicate_supers similarity_language_tag);
All the exportable functions are listed below \*(-- you're free to import
only some, or none at all. By default, none are imported. If you
\& use I18N::LangTags qw(:ALL)
\&...then all are exported. (This saves you from having to use
something less obvious like \f(CW\*(C`use I18N::LangTags qw(/./)\*(C'\fR.)
If you don't import any of these functions, assume a \f(CW&I18N::LangTags::\fR
in front of all the function names in the following examples.
Language tags are a formalism, described in \s-1RFC\s0 3066 (obsoleting
1766), for declaring what language form (language and possibly
dialect) a given chunk of information is in.
This library provides functions for common tasks involving language
tags as they are needed in a variety of protocols and applications.
Please see the \*(L"See Also\*(R" references for a thorough explanation
of how to correctly use language tags.
.IP "* the function is_language_tag($lang1)" 4
.IX Item "the function is_language_tag($lang1)"
Returns true iff \f(CW$lang1\fR is a formally valid language tag.
\& is_language_tag("fr") is TRUE
\& is_language_tag("x-jicarilla") is FALSE
\& (Subtags can be 8 chars long at most -- 'jicarilla' is 9)
\& is_language_tag("sgn-US") is TRUE
\& (That's American Sign Language)
\& is_language_tag("i-Klikitat") is TRUE
\& (True without regard to the fact noone has actually
\& registered Klikitat -- it's a formally valid tag)
\& is_language_tag("fr-patois") is TRUE
\& (Formally valid -- altho descriptively weak!)
\& is_language_tag("Spanish") is FALSE
\& is_language_tag("french-patois") is FALSE
\& (No good -- first subtag has to match
\& /^([xXiI]|[a-zA-Z]{2,3})$/ -- see RFC3066)
\& is_language_tag("x-borg-prot2532") is TRUE
\& (Yes, subtags can contain digits, as of RFC3066)
.IP "* the function extract_language_tags($whatever)" 4
.IX Item "the function extract_language_tags($whatever)"
Returns a list of whatever looks like formally valid language tags
in \f(CW$whatever\fR. Not very smart, so don't get too creative with
what you want to feed it.
\& extract_language_tags("fr, fr-ca, i-mingo")
\& returns: ('fr', 'fr-ca', 'i-mingo')
\& extract_language_tags("It's like this: I'm in fr -- French!")
\& returns: ('It', 'in', 'fr')
\& (So don't just feed it any old thing.)
The output is untainted. If you don't know what tainting is,
.ie n .IP "* the function same_language_tag($lang1, $lang2)" 4
.el .IP "* the function same_language_tag($lang1, \f(CW$lang2\fR)" 4
.IX Item "the function same_language_tag($lang1, $lang2)"
Returns true iff \f(CW$lang1\fR and \f(CW$lang2\fR are acceptable variant tags
representing the same language\-form.
\& same_language_tag('x-kadara', 'i-kadara') is TRUE
\& (The x/i- alternation doesn't matter)
\& same_language_tag('X-KADARA', 'i-kadara') is TRUE
\& (...and neither does case)
\& same_language_tag('en', 'en-US') is FALSE
\& (all-English is not the SAME as US English)
\& same_language_tag('x-kadara', 'x-kadar') is FALSE
\& (these are totally unrelated tags)
\& same_language_tag('no-bok', 'nb') is TRUE
\& (no-bok is a legacy tag for nb (Norwegian Bokmal))
\&\f(CW\*(C`same_language_tag\*(C'\fR works by just seeing whether
\&\f(CW\*(C`encode_language_tag($lang1)\*(C'\fR is the same as
\&\f(CW\*(C`encode_language_tag($lang2)\*(C'\fR.
(Yes, I know this function is named a bit oddly. Call it historic
.ie n .IP "* the function similarity_language_tag($lang1, $lang2)" 4
.el .IP "* the function similarity_language_tag($lang1, \f(CW$lang2\fR)" 4
.IX Item "the function similarity_language_tag($lang1, $lang2)"
Returns an integer representing the degree of similarity between
tags \f(CW$lang1\fR and \f(CW$lang2\fR (the order of which does not matter), where
similarity is the number of common elements on the left,
without regard to case and to x/i\- alternation.
\& similarity_language_tag('fr', 'fr-ca') is 1
\& (one element in common)
\& similarity_language_tag('fr-ca', 'fr-FR') is 1
\& (one element in common)
\& similarity_language_tag('fr-CA-joual',
\& similarity_language_tag('fr-CA-joual', 'fr-CA') is 2
\& (two elements in common)
\& similarity_language_tag('x-kadara', 'i-kadara') is 1
\& similarity_language_tag('en', 'x-kadar') is 0
\& similarity_language_tag('x-kadara', 'x-kadar') is 0
\& (unrelated tags -- no similarity)
\& similarity_language_tag('i-cree-syllabic',
\& 'i-cherokee-syllabic') is 0
\& (no B<leftmost> elements in common!)
.ie n .IP "* the function is_dialect_of($lang1, $lang2)" 4
.el .IP "* the function is_dialect_of($lang1, \f(CW$lang2\fR)" 4
.IX Item "the function is_dialect_of($lang1, $lang2)"
Returns true iff language tag \f(CW$lang1\fR represents a subform of
language tag \f(CW$lang2\fR.
\&\fBGet the order right! It doesn't work the other way around!\fR
\& is_dialect_of('en-US', 'en') is TRUE
\& (American English IS a dialect of all-English)
\& is_dialect_of('fr-CA-joual', 'fr-CA') is TRUE
\& is_dialect_of('fr-CA-joual', 'fr') is TRUE
\& (Joual is a dialect of (a dialect of) French)
\& is_dialect_of('en', 'en-US') is FALSE
\& (all-English is a NOT dialect of American English)
\& is_dialect_of('fr', 'en-CA') is FALSE
\& is_dialect_of('en', 'en' ) is TRUE
\& is_dialect_of('en-US', 'en-US') is TRUE
\& (B<Note:> these are degenerate cases)
\& is_dialect_of('i-mingo-tom', 'x-Mingo') is TRUE
\& (the x/i thing doesn't matter, nor does case)
\& is_dialect_of('nn', 'no') is TRUE
\& (because 'nn' (New Norse) is aliased to 'no-nyn',
\& as a special legacy case, and 'no-nyn' is a
\& subform of 'no' (Norwegian))
.IP "* the function super_languages($lang1)" 4
.IX Item "the function super_languages($lang1)"
Returns a list of language tags that are superordinate tags to \f(CW$lang1\fR
\&\*(-- it gets this by removing subtags from the end of \f(CW$lang1\fR until
nothing (or just \*(L"i\*(R" or \*(L"x\*(R") is left.
\& super_languages("fr-CA-joual") is ("fr-CA", "fr")
\& super_languages("en-AU") is ("en")
\& super_languages("en") is empty-list, ()
\& super_languages("i-cherokee") is empty-list, ()
\& ...not ("i"), which would be illegal as well as pointless.
If \f(CW$lang1\fR is not a valid language tag, returns empty-list in
a list context, undef in a scalar context.
A notable and rather unavoidable problem with this method:
\&\*(L"x\-mingo\-tom\*(R" has an \*(L"x\*(R" because the whole tag isn't an
IANA-registered tag \*(-- but super_languages('x\-mingo\-tom') is
('x\-mingo') \*(-- which isn't really right, since 'i\-mingo' is
registered. But this module has no way of knowing that. (But note
that same_language_tag('x\-mingo', 'i\-mingo') is \s-1TRUE\s0.)
More importantly, you assume \fIat your peril\fR that superordinates of
\&\f(CW$lang1\fR are mutually intelligible with \f(CW$lang1\fR. Consider this
.IP "* the function locale2language_tag($locale_identifier)" 4
.IX Item "the function locale2language_tag($locale_identifier)"
This takes a locale name (like \*(L"en\*(R", \*(L"en_US\*(R", or \*(L"en_US.ISO8859\-1\*(R")
and maps it to a language tag. If it's not mappable (as with,
notably, \*(L"C\*(R" and \*(L"\s-1POSIX\s0\*(R"), this returns empty-list in a list context,
or undef in a scalar context.
\& locale2language_tag("en") is "en"
\& locale2language_tag("en_US") is "en-US"
\& locale2language_tag("en_US.ISO8859-1") is "en-US"
\& locale2language_tag("C") is undef or ()
\& locale2language_tag("POSIX") is undef or ()
\& locale2language_tag("POSIX") is undef or ()
I'm not totally sure that locale names map satisfactorily to language
tags. Think \s-1REAL\s0 hard about how you use this. \s-1YOU\s0 \s-1HAVE\s0 \s-1BEEN\s0 \s-1WARNED\s0.
The output is untainted. If you don't know what tainting is,
.IP "* the function encode_language_tag($lang1)" 4
.IX Item "the function encode_language_tag($lang1)"
This function, if given a language tag, returns an encoding of it such
* tags representing different languages never get the same encoding.
* tags representing the same language always get the same encoding.
* an encoding of a formally valid language tag always is a string
value that is defined, has length, and is true if considered as a
Note that the encoding itself is \fBnot\fR a formally valid language tag.
Note also that you cannot, currently, go from an encoding back to a
language tag that it's an encoding of.
Note also that you \fBmust\fR consider the encoded value as atomic; i.e.,
you should not consider it as anything but an opaque, unanalysable
string value. (The internals of the encoding method may change in
future versions, as the language tagging standard changes over time.)
\&\f(CW\*(C`encode_language_tag\*(C'\fR returns undef if given anything other than a
formally valid language tag.
The reason \f(CW\*(C`encode_language_tag\*(C'\fR exists is because different language
tags may represent the same language; this is normally treatable with
\&\f(CW\*(C`same_language_tag\*(C'\fR, but consider this situation:
You have a data file that expresses greetings in different languages.
Its format is \*(L"[language tag]=[how to say 'Hello']\*(R", like:
And suppose you write a program that reads that file and then runs as
a daemon, answering client requests that specify a language tag and
then expect the string that says how to greet in that language. So an
\& greeting-client asks: fr
\& greeting-server answers: Bonjour
So far so good. But suppose the way you're implementing this is:
\& die unless open(IN, "<in.dat");
\& next unless /^([^=]+)=(.+)/s;
\& my($lang, $expr) = ($1, $2);
\& $greetings{$lang} = $expr;
at which point \f(CW%greetings\fR has the contents:
And suppose then that you answer client requests for language \f(CW$wanted\fR
by just looking up \f(CW$greetings\fR{$wanted}.
If the client asks for \*(L"fr\*(R", that will look up successfully in
\&\f(CW%greetings\fR, to the value \*(L"Bonjour\*(R". And if the client asks for
\&\*(L"i\-mingo\*(R", that will look up successfully in \f(CW%greetings\fR, to the value
But if the client asks for \*(L"i\-Mingo\*(R" or \*(L"x\-mingo\*(R", or \*(L"Fr\*(R", then the
lookup in \f(CW%greetings\fR fails. That's the Wrong Thing.
You could instead do lookups on \f(CW$wanted\fR with:
\& use I18N::LangTags qw(same_language_tag);
\& foreach my $l2 (keys %greetings) {
\& if(same_language_tag($wanted, $l2)) {
\& $response = $greetings{$l2};
But that's rather inefficient. A better way to do it is to start your
\& use I18N::LangTags qw(encode_language_tag);
\& die unless open(IN, "<in.dat");
\& next unless /^([^=]+)=(.+)/s;
\& my($lang, $expr) = ($1, $2);
\& encode_language_tag($lang)
and then just answer client requests for language \f(CW$wanted\fR by just
\& $greetings{encode_language_tag($wanted)}
And that does the Right Thing.
.IP "* the function alternate_language_tags($lang1)" 4
.IX Item "the function alternate_language_tags($lang1)"
This function, if given a language tag, returns all language tags that
are alternate forms of this language tag. (I.e., tags which refer to
the same language.) This is meant to handle legacy tags caused by
the minor changes in language tag standards over the years; and
the x\-/i\- alternation is also dealt with.
Note that this function does \fInot\fR try to equate new (and never\-used,
\&\s-1ISO639\-2\s0 three-letter tags to old (and still in use) \s-1ISO639\-1\s0
two-letter equivalents \*(-- like \*(L"ara\*(R" \-> \*(L"ar\*(R" \*(-- because
\&\*(L"ara\*(R" has \fInever\fR been in use as an Internet language tag,
and \s-1RFC\s0 3066 stipulates that it never should be, since a shorter
tag (\*(L"ar\*(R") exists.
\& alternate_language_tags('no-bok') is ('nb')
\& alternate_language_tags('nb') is ('no-bok')
\& alternate_language_tags('he') is ('iw')
\& alternate_language_tags('iw') is ('he')
\& alternate_language_tags('i-hakka') is ('zh-hakka', 'x-hakka')
\& alternate_language_tags('zh-hakka') is ('i-hakka', 'x-hakka')
\& alternate_language_tags('en') is ()
\& alternate_language_tags('x-mingo-tom') is ('i-mingo-tom')
\& alternate_language_tags('x-klikitat') is ('i-klikitat')
\& alternate_language_tags('i-klikitat') is ('x-klikitat')
This function returns empty-list if given anything other than a formally
.ie n .IP "* the function @langs = panic_languages(@accept_languages)" 4
.el .IP "* the function \f(CW@langs\fR = panic_languages(@accept_languages)" 4
.IX Item "the function @langs = panic_languages(@accept_languages)"
This function takes a list of 0 or more language
tags that constitute a given user's Accept-Language list, and
returns a list of tags for \fIother\fR (non\-super)
languages that are probably acceptable to the user, to be
used \fIif all else fails\fR.
For example, if a user accepts only 'ca' (Catalan) and
\&'es' (Spanish), and the documents/interfaces you have
available are just in German, Italian, and Chinese, then
the user will most likely want the Italian one (and not
the Chinese or German one!), instead of getting
nothing. So \f(CW\*(C`panic_languages('ca', 'es')\*(C'\fR returns
a list containing 'it' (Italian).
English ('en') is \fIalways\fR in the return list, but
whether it's at the very end or not depends
on the input languages. This function works by consulting
an internal table that stipulates what common
languages are \*(L"close\*(R" to each other.
A useful construct you might consider using is:
\& @fallbacks = super_languages(@accept_languages);
\& push @fallbacks, panic_languages(
\& @accept_languages, @fallbacks,
.IP "* the function implicate_supers( ...languages... )" 4
.IX Item "the function implicate_supers( ...languages... )"
This takes a list of strings (which are presumed to be language\-tags;
strings that aren't, are ignored); and after each one, this function
inserts super-ordinate forms that don't already appear in the list.
The original list, plus these insertions, is returned.
In other words, it takes this:
\& pt-br de-DE en-US fr pt-br-janeiro
\& pt-br pt de-DE de en-US en fr pt-br-janeiro
This function is most useful in the idiom
\& implicate_supers( I18N::LangTags::Detect::detect() );
(See I18N::LangTags::Detect.)
.IP "* the function implicate_supers_strictly( ...languages... )" 4
.IX Item "the function implicate_supers_strictly( ...languages... )"
This works like \f(CW\*(C`implicate_supers\*(C'\fR except that the implicated
forms are added to the end of the return list.
In other words, implicate_supers_strictly takes a list of strings
(which are presumed to be language\-tags; strings that aren't, are
ignored) and after the whole given list, it inserts the super-ordinate forms
of all given tags, minus any tags that already appear in the input list.
In other words, it takes this:
\& pt-br de-DE en-US fr pt-br-janeiro
\& pt-br de-DE en-US fr pt-br-janeiro pt de en
The reason this function has \*(L"_strictly\*(R" in its name is that when
you're processing an Accept-Language list according to the RFCs, if
you interpret the RFCs quite strictly, then you would use
implicate_supers_strictly, but for normal use (i.e., common-sense use,
as far as I'm concerned) you'd use implicate_supers.
.IX Header "ABOUT LOWERCASING"
I've considered making all the above functions that output language
tags return all those tags strictly in lowercase. Having all your
language tags in lowercase does make some things easier. But you
might as well just lowercase as you like, or call
\&\f(CW\*(C`encode_language_tag($lang1)\*(C'\fR where appropriate.
.SH "ABOUT UNICODE PLAINTEXT LANGUAGE TAGS"
.IX Header "ABOUT UNICODE PLAINTEXT LANGUAGE TAGS"
In some future version of I18N::LangTags, I plan to include support
for RFC2482\-style language tags \*(-- which are basically just normal
language tags with their \s-1ASCII\s0 characters shifted into Plane 14.
* \s-1RFC\s0 3066, \f(CW\*(C`ftp://ftp.isi.edu/in\-notes/rfc3066.txt\*(C'\fR, \*(L"Tags for the
Identification of Languages\*(R". (Obsoletes \s-1RFC\s0 1766)
* \s-1RFC\s0 2277, \f(CW\*(C`ftp://ftp.isi.edu/in\-notes/rfc2277.txt\*(C'\fR, \*(L"\s-1IETF\s0 Policy on
Character Sets and Languages\*(R".
* \s-1RFC\s0 2231, \f(CW\*(C`ftp://ftp.isi.edu/in\-notes/rfc2231.txt\*(C'\fR, \*(L"\s-1MIME\s0 Parameter
Value and Encoded Word Extensions: Character Sets, Languages, and
* \s-1RFC\s0 2482, \f(CW\*(C`ftp://ftp.isi.edu/in\-notes/rfc2482.txt\*(C'\fR,
\&\*(L"Language Tagging in Unicode Plain Text\*(R".
\&\f(CW\*(C`http://www.perl.com/CPAN/modules/by\-module/Locale/\*(C'\fR
* \s-1ISO\s0 639\-2, \*(L"Codes for the representation of names of languages\*(R",
including two-letter and three-letter codes,
\&\f(CW\*(C`http://www.loc.gov/standards/iso639\-2/langcodes.html\*(C'\fR
* The \s-1IANA\s0 list of registered languages (hopefully up\-to\-date),
\&\f(CW\*(C`http://www.iana.org/assignments/language\-tags\*(C'\fR
Copyright (c) 1998+ Sean M. Burke. All rights reserved.
This library is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.
The programs and documentation in this dist are distributed in
the hope that they will be useful, but without any warranty; without
even the implied warranty of merchantability or fitness for a
Sean M. Burke \f(CW\*(C`sburke@cpan.org\*(C'\fR
projects / OpenSPARC-T2-SAM / blob