- bytes_from_utf8
-
Converts a string s of length len from UTF8 into byte
encoding. Unlike <utf8_to_bytes> but like bytes_to_utf8, returns a
pointer to the newly-created string, and updates len to contain the new
length. Returns the original string if no conversion occurs, len is
unchanged. Do nothing if is_utf8 points to 0. Sets is_utf8 to 0
if s is converted or contains all 7bit characters.
NOTE: this function is experimental and may change or be removed without notice.
U8* bytes_from_utf8(U8 *s, STRLEN *len, bool *is_utf8)
|
|
- bytes_to_utf8
-
Converts a string s of length len from ASCII into UTF8
encoding. Returns a pointer to the newly-created string, and sets len to
reflect the new length.
NOTE: this function is experimental and may change or be removed without notice.
U8* bytes_to_utf8(U8 *s, STRLEN *len)
|
|
- ibcmp_utf8
-
Return true if the strings s1 and s2 differ case-insensitively, false if not (if they
are equal case-insensitively). If u1 is true, the string s1 is assumed to be in
UTF-8-encoded Unicode. If u2 is true, the string s2 is assumed to be in UTF-8-encoded
Unicode. If u1 or u2 are false, the respective string is assumed to be in native 8-bit
encoding.
If the pe1 and pe2 are non-NULL, the scanning pointers will be copied in there (they
will point at the beginning of the next character). If the pointers behind pe1 or
pe2 are non-NULL, they are the end pointers beyond which scanning will not continue under
any circustances. If the byte lengths l1 and l2 are non-zero, s1+l1 and s2+l2 will be used
as goal end pointers that will also stop the scan, and which qualify towards defining a
successful match: all the scans that define an explicit length must reach their goal
pointers for a match to succeed).
For case-insensitiveness, the "casefolding" of Unicode is used instead of
upper/lowercasing both the characters, see http://www.unicode.org/unicode/reports/tr21/
(Case Mappings).
I32 ibcmp_utf8(const char* a, char **pe1, UV l1, bool u1, const char* b, char **pe2, UV l2, bool u2)
|
|
- is_utf8_char
-
Tests if some arbitrary number of bytes begins in a valid UTF-8 character. Note that an
INVARIANT (i.e. ASCII) character is a valid UTF-8 character. The actual number of bytes in
the UTF-8 character will be returned if it is valid, otherwise 0.
STRLEN is_utf8_char(U8 *p)
|
|
- is_utf8_string
-
Returns true if first len bytes of the given string form a valid UTF8
string, false otherwise. Note that 'a valid UTF8 string' does not mean 'a string that
contains UTF8' because a valid ASCII string is a valid UTF8 string.
bool is_utf8_string(U8 *s, STRLEN len)
|
|
- pv_uni_display
-
Build to the scalar dsv a displayable version of the string spv, length len, the
displayable version being at most pvlim bytes long (if longer, the rest is truncated and
"..." will be appended).
The flags argument can have UNI_DISPLAY_ISPRINT set to display isPRINT()able characters
as themselves, UNI_DISPLAY_BACKSLASH to display the \\[nrfta\\] as the backslashed
versions (like '\n') (UNI_DISPLAY_BACKSLASH is preferred over UNI_DISPLAY_ISPRINT for \\).
UNI_DISPLAY_QQ (and its alias UNI_DISPLAY_REGEX) have both UNI_DISPLAY_BACKSLASH and
UNI_DISPLAY_ISPRINT turned on.
The pointer to the PV of the dsv is returned.
char* pv_uni_display(SV *dsv, U8 *spv, STRLEN len, STRLEN pvlim, UV flags)
|
|
- sv_recode_to_utf8
-
The encoding is assumed to be an Encode object, on entry the PV of the sv is assumed to
be octets in that encoding, and the sv will be converted into Unicode (and UTF-8).
If the sv already is UTF-8 (or if it is not POK), or if the encoding is not a
reference, nothing is done to the sv. If the encoding is not an Encode::XS
Encoding object, bad things will happen. (See lib/encoding.pm and Encode).
The PV of the sv is returned.
char* sv_recode_to_utf8(SV* sv, SV *encoding)
|
|
- sv_uni_display
-
Build to the scalar dsv a displayable version of the scalar sv, the displayable version
being at most pvlim bytes long (if longer, the rest is truncated and "..." will
be appended).
The flags argument is as in pv_uni_display().
The pointer to the PV of the dsv is returned.
char* sv_uni_display(SV *dsv, SV *ssv, STRLEN pvlim, UV flags)
|
|
- to_utf8_case
-
The "p" contains the pointer to the UTF-8 string encoding the character that
is being converted.
The "ustrp" is a pointer to the character buffer to put the conversion result
to. The "lenp" is a pointer to the length of the result.
The "swashp" is a pointer to the swash to use.
Both the special and normal mappings are stored lib/unicore/To/Foo.pl, and loaded by
SWASHGET, using lib/utf8_heavy.pl. The special (usually, but not always, a multicharacter
mapping), is tried first.
The "special" is a string like "utf8::ToSpecLower", which means the
hash %utf8::ToSpecLower. The access to the hash is through Perl_to_utf8_case().
The "normal" is a string like "ToLower" which means the swash
%utf8::ToLower.
UV to_utf8_case(U8 *p, U8* ustrp, STRLEN *lenp, SV **swash, char *normal, char *special)
|
|
- to_utf8_fold
-
Convert the UTF-8 encoded character at p to its foldcase version and store that in
UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least
UTF8_MAXLEN_FOLD+1 bytes since the foldcase version may be longer than the original
character (up to three characters).
The first character of the foldcased version is returned (but note, as explained above,
that there may be more.)
UV to_utf8_fold(U8 *p, U8* ustrp, STRLEN *lenp)
|
|
- to_utf8_lower
-
Convert the UTF-8 encoded character at p to its lowercase version and store that in
UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least
UTF8_MAXLEN_UCLC+1 bytes since the lowercase version may be longer than the original
character (up to two characters).
The first character of the lowercased version is returned (but note, as explained
above, that there may be more.)
UV to_utf8_lower(U8 *p, U8* ustrp, STRLEN *lenp)
|
|
- to_utf8_title
-
Convert the UTF-8 encoded character at p to its titlecase version and store that in
UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least
UTF8_MAXLEN_UCLC+1 bytes since the titlecase version may be longer than the original
character (up to two characters).
The first character of the titlecased version is returned (but note, as explained
above, that there may be more.)
UV to_utf8_title(U8 *p, U8* ustrp, STRLEN *lenp)
|
|
- to_utf8_upper
-
Convert the UTF-8 encoded character at p to its uppercase version and store that in
UTF-8 in ustrp and its length in bytes in lenp. Note that the ustrp needs to be at least
UTF8_MAXLEN_UCLC+1 bytes since the uppercase version may be longer than the original
character (up to two characters).
The first character of the uppercased version is returned (but note, as explained
above, that there may be more.)
UV to_utf8_upper(U8 *p, U8* ustrp, STRLEN *lenp)
|
|
- utf8n_to_uvchr
-
Returns the native character value of the first character in the string s
which is assumed to be in UTF8 encoding; retlen will be set to the length, in
bytes, of that character.
Allows length and flags to be passed to low level routine.
UV utf8n_to_uvchr(U8 *s, STRLEN curlen, STRLEN* retlen, U32 flags)
|
|
- utf8n_to_uvuni
-
Bottom level UTF-8 decode routine. Returns the unicode code point value of the first
character in the string s which is assumed to be in UTF8 encoding and no
longer than curlen; retlen will be set to the length, in bytes,
of that character.
If s does not point to a well-formed UTF8 character, the behaviour is
dependent on the value of flags: if it contains UTF8_CHECK_ONLY, it is
assumed that the caller will raise a warning, and this function will silently just set retlen
to -1 and return zero. If the flags does not contain
UTF8_CHECK_ONLY, warnings about malformations will be given, retlen will be
set to the expected length of the UTF-8 character in bytes, and zero will be returned.
The flags can also contain various flags to allow deviations from the
strict UTF-8 encoding (see utf8.h).
Most code should use utf8_to_uvchr() rather than call this directly.
UV utf8n_to_uvuni(U8 *s, STRLEN curlen, STRLEN* retlen, U32 flags)
|
|
- utf8_distance
-
Returns the number of UTF8 characters between the UTF-8 pointers a and b.
WARNING: use only if you *know* that the pointers point inside the same UTF-8 buffer.
IV utf8_distance(U8 *a, U8 *b)
|
|
- utf8_hop
-
Return the UTF-8 pointer s displaced by off characters,
either forward or backward.
WARNING: do not use the following unless you *know* off is within the
UTF-8 data pointed to by s *and* that on entry s is aligned on
the first byte of character or just after the last byte of a character.
U8* utf8_hop(U8 *s, I32 off)
|
|
- utf8_length
-
Return the length of the UTF-8 char encoded string s in characters. Stops
at e (inclusive). If e < s or if the scan would end up past e,
croaks.
STRLEN utf8_length(U8* s, U8 *e)
|
|
- utf8_to_bytes
-
Converts a string s of length len from UTF8 into byte
encoding. Unlike bytes_to_utf8, this over-writes the original string, and
updates len to contain the new length. Returns zero on failure, setting len
to -1.
NOTE: this function is experimental and may change or be removed without notice.
U8* utf8_to_bytes(U8 *s, STRLEN *len)
|
|
- utf8_to_uvchr
-
Returns the native character value of the first character in the string s
which is assumed to be in UTF8 encoding; retlen will be set to the length, in
bytes, of that character.
If s does not point to a well-formed UTF8 character, zero is returned and
retlen is set, if possible, to -1.
UV utf8_to_uvchr(U8 *s, STRLEN* retlen)
|
|
- utf8_to_uvuni
-
Returns the Unicode code point of the first character in the string s
which is assumed to be in UTF8 encoding; retlen will be set to the length, in
bytes, of that character.
This function should only be used when returned UV is considered an index into the
Unicode semantic tables (e.g. swashes).
If s does not point to a well-formed UTF8 character, zero is returned and
retlen is set, if possible, to -1.
UV utf8_to_uvuni(U8 *s, STRLEN* retlen)
|
|
- uvchr_to_utf8
-
Adds the UTF8 representation of the Native codepoint uv to the end of the
string d; d should be have at least UTF8_MAXLEN+1
free bytes available. The return value is the pointer to the byte after the end of the new
character. In other words,
d = uvchr_to_utf8(d, uv);
|
|
is the recommended wide native character-aware way of saying
*(d++) = uv;
U8* uvchr_to_utf8(U8 *d, UV uv)
|
|
- uvuni_to_utf8_flags
-
Adds the UTF8 representation of the Unicode codepoint uv to the end of the
string d; d should be have at least UTF8_MAXLEN+1
free bytes available. The return value is the pointer to the byte after the end of the new
character. In other words,
d = uvuni_to_utf8_flags(d, uv, flags);
|
|
or, in most cases,
d = uvuni_to_utf8(d, uv);
|
|
(which is equivalent to)
d = uvuni_to_utf8_flags(d, uv, 0);
|
|
is the recommended Unicode-aware way of saying
*(d++) = uv;
U8* uvuni_to_utf8_flags(U8 *d, UV uv, UV flags)
|
|
|
|
