|
The following is a long section of miscellaneous requirements and suggestions to do with Pod
processing.
- Pod formatters should tolerate lines in verbatim blocks that are of any length, even if
that means having to break them (possibly several times, for very long lines) to avoid text
running off the side of the page. Pod formatters may warn of such line-breaking. Such
warnings are particularly appropriate for lines are over 100 characters long, which are
usually not intentional.
- Pod parsers must recognize all of the three well-known newline formats: CR, LF, and
CRLF. See perlport.
- Pod parsers should accept input lines that are of any length.
-
Since Perl recognizes a Unicode Byte Order Mark at the start of files as signaling that
the file is Unicode encoded as in UTF-16 (whether big-endian or little-endian) or UTF-8, Pod
parsers should do the same. Otherwise, the character encoding should be understood as being
UTF-8 if the first highbit byte sequence in the file seems valid as a UTF-8 sequence, or
otherwise as Latin-1.
Future versions of this specification may specify how Pod can accept other encodings.
Presumably treatment of other encodings in Pod parsing would be as in XML parsing: whatever
the encoding declared by a particular Pod file, content is to be stored in memory as Unicode
characters.
-
The well known Unicode Byte Order Marks are as follows: if the file begins with the two
literal byte values 0xFE 0xFF, this is the BOM for big-endian UTF-16. If the file begins
with the two literal byte value 0xFF 0xFE, this is the BOM for little-endian UTF-16. If the
file begins with the three literal byte values 0xEF 0xBB 0xBF, this is the BOM for UTF-8.
-
A naive but sufficient heuristic for testing the first highbit byte-sequence in a BOM-less
file (whether in code or in Pod!), to see whether that sequence is valid as UTF-8 (RFC 2279)
is to check whether that the first byte in the sequence is in the range 0xC0 - 0xFD and
whether the next byte is in the range 0x80 - 0xBF. If so, the parser may conclude that this
file is in UTF-8, and all highbit sequences in the file should be assumed to be UTF-8.
Otherwise the parser should treat the file as being in Latin-1. In the unlikely circumstance
that the first highbit sequence in a truly non-UTF-8 file happens to appear to be UTF-8, one
can cater to our heuristic (as well as any more intelligent heuristic) by prefacing that
line with a comment line containing a highbit sequence that is clearly not valid as
UTF-8. A line consisting of simply "#", an e-acute, and any non-highbit byte, is
sufficient to establish this file's encoding.
- This document's requirements and suggestions about encodings do not apply to Pod
processors running on non-ASCII platforms, notably EBCDIC platforms.
- Pod processors must treat a "=for [label] [content...]" paragraph as meaning the
same thing as a "=begin [label]" paragraph, content, and an "=end
[label]" paragraph. (The parser may conflate these two constructs, or may leave them
distinct, in the expectation that the formatter will nevertheless treat them the same.)
-
When rendering Pod to a format that allows comments (i.e., to nearly any format other
than plaintext), a Pod formatter must insert comment text identifying its name and version
number, and the name and version numbers of any modules it might be using to process the
Pod. Minimal examples:
%% POD::Pod2PS v3.14159, using POD::Parser v1.92
<!-- Pod::HTML v3.14159, using POD::Parser v1.92 -->
{\doccomm generated by Pod::Tree::RTF 3.14159 using Pod::Tree 1.08}
.\" Pod::Man version 3.14159, using POD::Parser version 1.92
|
|
Formatters may also insert additional comments, including: the release date of the Pod
formatter program, the contact address for the author(s) of the formatter, the current time,
the name of input file, the formatting options in effect, version of Perl used, etc.
Formatters may also choose to note errors/warnings as comments, besides or instead of
emitting them otherwise (as in messages to STDERR, or dieing).
- Pod parsers may emit warnings or error messages ("Unknown E code E<zslig>!")
to STDERR (whether through printing to STDERR, or
warning/carping,
or dieing/croaking), but must allow suppressing all such
STDERR output, and instead allow an option for reporting errors/warnings in some other way,
whether by triggering a callback, or noting errors in some attribute of the document object,
or some similarly unobtrusive mechanism -- or even by appending a "Pod Errors"
section to the end of the parsed form of the document.
- In cases of exceptionally aberrant documents, Pod parsers may abort the parse. Even then,
using
dieing/croaking is to be avoided; where possible, the parser
library may simply close the input file and add text like "*** Formatting Aborted
***" to the end of the (partial) in-memory document.
- In paragraphs where formatting codes (like E<...>, B<...>) are understood
(i.e., not verbatim paragraphs, but including ordinary paragraphs, and command
paragraphs that produce renderable text, like "=head1"), literal whitespace should
generally be considered "insignificant", in that one literal space has the same
meaning as any (nonzero) number of literal spaces, literal newlines, and literal tabs (as
long as this produces no blank lines, since those would terminate the paragraph). Pod
parsers should compact literal whitespace in each processed paragraph, but may provide an
option for overriding this (since some processing tasks do not require it), or may follow
additional special rules (for example, specially treating period-space-space or period-newline
sequences).
- Pod parsers should not, by default, try to coerce apostrophe (') and quote (") into
smart quotes (little 9's, 66's, 99's, etc), nor try to turn backtick (`) into anything else
but a single backtick character (distinct from an openquote character!), nor "--"
into anything but two minus signs. They must never do any of those things to text in
C<...> formatting codes, and never ever to text in verbatim paragraphs.
- When rendering Pod to a format that has two kinds of hyphens (-), one that's a nonbreaking
hyphen, and another that's a breakable hyphen (as in "object-oriented", which can
be split across lines as "object-", newline, "oriented"), formatters are
encouraged to generally translate "-" to nonbreaking hyphen, but may apply
heuristics to convert some of these to breaking hyphens.
- Pod formatters should make reasonable efforts to keep words of Perl code from being broken
across lines. For example, "Foo::Bar" in some formatting systems is seen as
eligible for being broken across lines as "Foo::" newline "Bar" or even
"Foo::-" newline "Bar". This should be avoided where possible, either by
disabling all line-breaking in mid-word, or by wrapping particular words with internal
punctuation in "don't break this across lines" codes (which in some formats may
not be a single code, but might be a matter of inserting non-breaking zero-width spaces
between every pair of characters in a word.)
- Pod parsers should, by default, expand tabs in verbatim paragraphs as they are processed,
before passing them to the formatter or other processor. Parsers may also allow an option
for overriding this.
- Pod parsers should, by default, remove newlines from the end of ordinary and verbatim
paragraphs before passing them to the formatter. For example, while the paragraph you're
reading now could be considered, in Pod source, to end with (and contain) the newline(s)
that end it, it should be processed as ending with (and containing) the period character
that ends this sentence.
- Pod parsers, when reporting errors, should make some effort to report an approximate line
number ("Nested E<>'s in Paragraph #52, near line 633 of Thing/Foo.pm!"),
instead of merely noting the paragraph number ("Nested E<>'s in Paragraph #52 of
Thing/Foo.pm!"). Where this is problematic, the paragraph number should at least be
accompanied by an excerpt from the paragraph ("Nested E<>'s in Paragraph #52 of
Thing/Foo.pm, which begins 'Read/write accessor for the C<interest rate>
attribute...'").
-
Pod parsers, when processing a series of verbatim paragraphs one after another, should
consider them to be one large verbatim paragraph that happens to contain blank lines. I.e.,
these two lines, which have a blank line between them:
use Foo;
print Foo->VERSION
|
|
should be unified into one paragraph ("\tuse Foo;\n\n\tprint Foo->VERSION")
before being passed to the formatter or other processor. Parsers may also allow an option
for overriding this.
While this might be too cumbersome to implement in event-based Pod parsers, it is
straightforward for parsers that return parse trees.
- Pod formatters, where feasible, are advised to avoid splitting short verbatim paragraphs
(under twelve lines, say) across pages.
- Pod parsers must treat a line with only spaces and/or tabs on it as a "blank
line" such as separates paragraphs. (Some older parsers recognized only two adjacent
newlines as a "blank line" but would not recognize a newline, a space, and a
newline, as a blank line. This is noncompliant behavior.)
- Authors of Pod formatters/processors should make every effort to avoid writing their own
Pod parser. There are already several in CPAN, with a wide range of interface styles -- and
one of them, Pod::Parser, comes with modern versions of Perl.
-
Characters in Pod documents may be conveyed either as literals, or by number in
E<n> codes, or by an equivalent mnemonic, as in E<eacute> which is exactly
equivalent to E<233>.
Characters in the range 32-126 refer to those well known US-ASCII characters (also
defined there by Unicode, with the same meaning), which all Pod formatters must render
faithfully. Characters in the ranges 0-31 and 127-159 should not be used (neither as
literals, nor as E<number> codes), except for the literal byte-sequences for newline
(13, 13 10, or 10), and tab (9).
Characters in the range 160-255 refer to Latin-1 characters (also defined there by
Unicode, with the same meaning). Characters above 255 should be understood to refer to
Unicode characters.
- Be warned that some formatters cannot reliably render characters outside 32-126; and many
are able to handle 32-126 and 160-255, but nothing above 255.
- Besides the well-known "E<lt>" and "E<gt>" codes for
less-than and greater-than, Pod parsers must understand "E<sol>" for
"/" (solidus, slash), and "E<verbar>" for "|" (vertical
bar, pipe). Pod parsers should also understand "E<lchevron>" and "E<rchevron>"
as legacy codes for characters 171 and 187, i.e., "left-pointing double angle quotation
mark" = "left pointing guillemet" and "right-pointing double angle
quotation mark" = "right pointing guillemet". (These look like little
"<<" and ">>", and they are now preferably expressed with
the HTML/XHTML codes "E<laquo>" and "E<raquo>".)
- Pod parsers should understand all "E<html>" codes as defined in the entity
declarations in the most recent XHTML specification at
www.W3.org. Pod parsers
must understand at least the entities that define characters in the range 160-255 (Latin-1).
Pod parsers, when faced with some unknown "E<identifier>" code,
shouldn't simply replace it with nullstring (by default, at least), but may pass it through
as a string consisting of the literal characters E, less-than, identifier,
greater-than. Or Pod parsers may offer the alternative option of processing such unknown
"E<identifier>" codes by firing an event especially for such codes,
or by adding a special node-type to the in-memory document tree. Such "E<identifier>"
may have special meaning to some processors, or some processors may choose to add them to a
special error report.
- Pod parsers must also support the XHTML codes "E<quot>" for character 34 (doublequote,
"), "E<amp>" for character 38 (ampersand, &), and "E<apos>"
for character 39 (apostrophe, ').
- Note that in all cases of "E<whatever>", whatever (whether an
htmlname, or a number in any base) must consist only of alphanumeric characters -- that is, whatever
must watch
m/\A\w+\z/. So "E< 0 1 2 3 >" is invalid, because it
contains spaces, which aren't alphanumeric characters. This presumably does not need
special treatment by a Pod processor; " 0 1 2 3 " doesn't look like a number in
any base, so it would presumably be looked up in the table of HTML-like names. Since there
isn't (and cannot be) an HTML-like entity called " 0 1 2 3 ", this will be treated
as an error. However, Pod processors may treat "E< 0 1 2 3 >" or
"E<e-acute>" as syntactically invalid, potentially earning a
different error message than the error message (or warning, or event) generated by a merely
unknown (but theoretically valid) htmlname, as in "E<qacute>" [sic].
However, Pod parsers are not required to make this distinction.
-
Note that E<number> must not be interpreted as simply "codepoint number
in the current/native character set". It always means only "the character
represented by codepoint number in Unicode." (This is identical to the semantics
of &#number; in XML.)
This will likely require many formatters to have tables mapping from treatable Unicode
codepoints (such as the "\xE9" for the e-acute character) to the escape sequences
or codes necessary for conveying such sequences in the target output format. A converter to
*roff would, for example know that "\xE9" (whether conveyed literally, or via a
E<...> sequence) is to be conveyed as "e\\*'". Similarly, a program
rendering Pod in a Mac OS application window, would presumably need to know that
"\xE9" maps to codepoint 142 in MacRoman encoding that (at time of writing) is
native for Mac OS. Such Unicode2whatever mappings are presumably already widely available
for common output formats. (Such mappings may be incomplete! Implementers are not expected
to bend over backwards in an attempt to render Cherokee syllabics, Etruscan runes, Byzantine
musical symbols, or any of the other weird things that Unicode can encode.) And if a Pod
document uses a character not found in such a mapping, the formatter should consider it an
unrenderable character.
-
If, surprisingly, the implementor of a Pod formatter can't find a satisfactory
pre-existing table mapping from Unicode characters to escapes in the target format (e.g., a
decent table of Unicode characters to *roff escapes), it will be necessary to build such a
table. If you are in this circumstance, you should begin with the characters in the range
0x00A0 - 0x00FF, which is mostly the heavily used accented characters. Then proceed (as
patience permits and fastidiousness compels) through the characters that the (X)HTML
standards groups judged important enough to merit mnemonics for. These are declared in the (X)HTML
specifications at the www.W3.org site. At time of writing (September 2001), the most recent
entity declaration files are:
http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent
http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent
http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent
|
|
Then you can progress through any remaining notable Unicode characters in the range
0x2000-0x204D (consult the character tables at www.unicode.org), and whatever else strikes
your fancy. For example, in xhtml-symbol.ent, there is the entry:
<!ENTITY infin "∞"> <!-- infinity, U+221E ISOtech -->
|
|
While the mapping "infin" to the character "\x{221E}" will
(hopefully) have been already handled by the Pod parser, the presence of the character in
this file means that it's reasonably important enough to include in a formatter's table that
maps from notable Unicode characters to the codes necessary for rendering them. So for a
Unicode-to-*roff mapping, for example, this would merit the entry:
It is eagerly hoped that in the future, increasing numbers of formats (and formatters)
will support Unicode characters directly (as (X)HTML does with ∞, ∞,
or ∞), reducing the need for idiosyncratic mappings of Unicode-to-my_escapes.
-
It is up to individual Pod formatter to display good judgment when confronted with an
unrenderable character (which is distinct from an unknown E<thing> sequence that the
parser couldn't resolve to anything, renderable or not). It is good practice to map Latin
letters with diacritics (like "E<eacute>"/"E<233>") to the
corresponding unaccented US-ASCII letters (like a simple character 101, "e"), but
clearly this is often not feasible, and an unrenderable character may be represented as
"?", or the like. In attempting a sane fallback (as from E<233> to
"e"), Pod formatters may use the %Latin1Code_to_fallback table in Pod::Escapes, or Text::Unidecode, if available.
For example, this Pod text:
magic is enabled if you set C<$Currency> to 'E<euro>'.
|
|
may be rendered as: "magic is enabled if you set $Currency to '?'"
or as "magic is enabled if you set $Currency to '[euro]'", or
as "magic is enabled if you set $Currency to '[x20AC]', etc.
A Pod formatter may also note, in a comment or warning, a list of what unrenderable
characters were encountered.
- E<...> may freely appear in any formatting code (other than in another E<...>
or in an Z<>). That is, "X<The E<euro>1,000,000 Solution>" is
valid, as is "L<The E<euro>1,000,000 Solution|Million::Euros>".
- Some Pod formatters output to formats that implement nonbreaking spaces as an individual
character (which I'll call "NBSP"), and others output to formats that implement
nonbreaking spaces just as spaces wrapped in a "don't break this across lines"
code. Note that at the level of Pod, both sorts of codes can occur: Pod can contain a NBSP
character (whether as a literal, or as a "E<160>" or "E<nbsp>"
code); and Pod can contain "S<foo I<bar> baz>" codes, where "mere
spaces" (character 32) in such codes are taken to represent nonbreaking spaces. Pod
parsers should consider supporting the optional parsing of "S<foo I<bar> baz>"
as if it were "fooNBSPI<bar>NBSPbaz", and, going the other
way, the optional parsing of groups of words joined by NBSP's as if each group were in a
S<...> code, so that formatters may use the representation that maps best to what the
output format demands.
-
Some processors may find that the S<...> code is easiest to implement
by replacing each space in the parse tree under the content of the S, with an NBSP. But
note: the replacement should apply not to spaces in all text, but only
to spaces in printable text. (This distinction may or may not be evident in the
particular tree/event model implemented by the Pod parser.) For example, consider this
unusual case:
S<L</Autoloaded Functions>>
|
|
This means that the space in the middle of the visible link text must not be broken
across lines. In other words, it's the same as this:
L<"AutoloadedE<160>Functions"/Autoloaded Functions>
|
|
However, a misapplied space-to-NBSP replacement could (wrongly) produce something
equivalent to this:
L<"AutoloadedE<160>Functions"/AutoloadedE<160>Functions>
|
|
...which is almost definitely not going to work as a hyperlink (assuming this formatter
outputs a format supporting hypertext).
Formatters may choose to just not support the S format code, especially in cases where
the output format simply has no NBSP character/code and no code for "don't break this
stuff across lines".
-
Besides the NBSP character discussed above, implementors are reminded of the existence of
the other "special" character in Latin-1, the "soft hyphen" character,
also known as "discretionary hyphen", i.e. E<173> = E<0xAD>
= E<shy>). This character expresses an optional hyphenation point. That
is, it normally renders as nothing, but may render as a "-" if a formatter breaks
the word at that point. Pod formatters should, as appropriate, do one of the following: 1)
render this with a code with the same meaning (e.g., "\-" in RTF), 2) pass it
through in the expectation that the formatter understands this character as such, or 3)
delete it.
For example:
sigE<shy>action
manuE<shy>script
JarkE<shy>ko HieE<shy>taE<shy>nieE<shy>mi
|
|
These signal to a formatter that if it is to hyphenate "sigaction" or
"manuscript", then it should be done as "sig-[linebreak]action"
or "manu-[linebreak]script" (and if it doesn't hyphenate it, then the E<shy>
doesn't show up at all). And if it is to hyphenate "Jarkko" and/or "Hietaniemi",
it can do so only at the points where there is a E<shy> code.
In practice, it is anticipated that this character will not be used often, but formatters
should either support it, or delete it.
- If you think that you want to add a new command to Pod (like, say, a "=biblio"
command), consider whether you could get the same effect with a for or begin/end sequence:
"=for biblio ..." or "=begin biblio" ... "=end biblio". Pod
processors that don't understand "=for biblio", etc, will simply ignore it,
whereas they may complain loudly if they see "=biblio".
- Throughout this document, "Pod" has been the preferred spelling for the name of
the documentation format. One may also use "POD" or "pod". For the
documentation that is (typically) in the Pod format, you may use "pod", or
"Pod", or "POD". Understanding these distinctions is useful; but
obsessing over how to spell them, usually is not.
|
|
