ASCII character utilities

Contents

1. Revision history

1.1. Changes since R4

• Rename is_ascii_* functions to ascii_is_* to increase consensus in SG16
• Replace incorrect "C" locale with C locale in §2. Introduction
• Add §3.8. Naming
• Remove stray default argument for ascii_is_digit in <ascii> synopsis

1.2. Changes since R3

• Fix a bug in the get_hex_digit_value example in §3.2. ascii_is_any
• Split ascii_is_digit (formerly know as is_ascii_digit) into two overloads
• Add design discussion in §3.13. namespace ascii

1.3. Changes since R2

• Expand §3.6. Why no function objects? based on feedback from BSI (British Standards Institution); no changes to design are made.
• Rebase §5. Wording on [N5014].

1.4. Changes since R1

• In §5. Wording, fix incorrect return type for ascii_case_insensitive_compare in synopsis.
• In §5. Wording, fix superfluous std:: prefix for strong_ordering in definition of ascii_case_insensitive_compare.

1.5. Changes since R0

• In §3.3. base parameter in ascii_is_digit, explain why the precondition is not hardened.
• In §5. Wording, fix a missing addition to [tab:headers.cpp].
• Minor editorial changes.

2. Introduction

Testing whether a character falls into a specific subset of ASCII characters or performing some simple transformations are common tasks in text processing. For example, applications may need to check if identifiers are comprised of alphanumeric ASCII characters or underscores; Unicode properties are not relevant to this task, and usually, neither are locales.

Unfortunately, these common and simple tasks are only supported through functions in the <cctype> and <locale> headers, such as:

Especially the <cctype> functions are ridden with problems:

1. There is no support for Unicode character types (char8_t, char16_t, and char32_t).
2. These functions are not constexpr, but performing basic characters tests would be useful at compile time.
3. There are distinct function names for char and wchar_t such as std::isalnum and std::iswalnum, making generic programming more difficult.
4. If char is signed, these functions can easily result in undefined behavior because the input must be representable as unsigned char or be EOF. If char represents a UTF-8 code unit, passing any non-ASCII code unit into these functions has undefined behavior.
5. These functions violate the zero-overhead principle by also handling an EOF input, and in many use cases, EOF will never be passed into these functions anyway. The caller can easily deal with EOF themselves.
6. The return type of character tests is int, where a nonzero return value indicates that a test succeeded. This is very unnatural in C++, where bool is more idiomatic.
7. Some functions use the currently installed C locale, which makes their use questionable for high-performance tasks because each invocation is typically an opaque call that checks the current locale.

We propose lightweight replacement functions which address all these problems.

2.1. Can't you implement this trivially yourself?

It is worth noting that some of the functions can be implemented very easily by the user. For example, existing code may already use a check like c >= '0' && c <= '9' to test for ASCII digits, and our proposed ascii_is_digit does just that.

However, not all of the proposed functions are this simple. For example, checking whether a char is an ASCII punctuation character ('#', '?', etc.) would require lots of separate checks done naively. In the standard library, it can be efficiently implemented using a 128-bit or 256-bit bitset.

Even if all proposed functions were trivial to implement, working with ASCII characters is such an overwhelmingly common use case that it's worth supporting in the standard library.

3. Design

All proposed functions are constexpr, locale-independent, overloaded (i.e. no separate name for separate input types), and accept any character type (char, wchar_t, char8_t, char16_t, and char32_t). Furthermore, all function names contain ascii to raise awareness for the fact that these functions do not handle Unicode characters. A user would expect is_upper(U'Ä') to be true, but ascii_is_upper(U'Ä') to be false.

character-type means that there exists an overload set where this placeholder is replaced with each of the character types. This design is more consistent with std::from_chars and <cmath> functions than say, template<class Char>. Equivalent functions could also be added to C, if there is interest. This signature also allows the use with types that are convertible to a specific character type.

3.1. List of proposed functions

Find below a list of proposed functions. Note that the character set notation [...] is taken from RegEx.

3.2. ascii_is_any

This additional function is mainly useful for checking if a character "is ASCII", i.e. falls into the basic latin block, before performing an ASCII-only evaluation.

3.3. base parameter in ascii_is_digit

Similar to std::to_chars, std::ascii_is_digit can also take a base parameter. There are two overloads:

If base ≤ 10, the range of valid ASCII digit character is simply limited. For greater base, a subset of alphabetic characters is also accepted, starting with 'a' or 'A'. Such a function is useful when parsing numbers with a base of choice, which is what std::to_chars does, for example.

Similar to std::from_chars and std::to_chars, the given base has to be between 2 and 36 (inclusive). This is a non-hardened precondition because all functions in <ascii> are low-level, high-performance, and spiritually numeric. Hardened preconditions are not used within that context.

3.4. ascii_is_bit and ascii_is_octal_digit

C++ and various other programming languages support binary and octal literals, so it seems like an arbitrary choice to only have dedicated overloads for (hexa)decimal digits. ascii_is_bit may be especially useful, such as when dealing with bit-strings like one of the std::bitset constructors.

In conclusion, we may as well have functions for bases 2, 8, 10, and 16; they're not doing much harm, they're trivial to implement, and some users may find them useful.

3.5. Case-insensitive comparison functions

As shown in the table above, we also propose the case-insensitive comparison functions.

3.6. Why no function objects?

For case-insensitive comparisons and for character tests in general, function objects may be convenient because they can be more easily used in algorithms:

However, there is no reason why ascii_is_digit needs to be a function object. It is not a customization point, but a simple utility, and the established practice is to make these utilities free functions.

There are countless functions in the standard library that the user may desire to put directly into an algorithm. For example, a user may want to put std::sqrt or std::abs in ranges::transform, or use them as a projection in an algorithm. Cherry-picking the functions in <ascii> to be function objects is far from solving the general problem; in fact, encouraging direct use of <ascii> function objects in algorithms could mislead the user into attempting the same with various non-addressable functions, such as std::abs, making the program IFNDR.

This problem should be solved generally, such as:

• Overhauling the standard library to convert most functions into function objects.
• Standardizing a LIFT macro that wraps an overload set in a lambda, or some other means of wrapping, possibly as a core language feature similar to the one proposed in [P3312R1] "Overload Set Types".

Any solution to the general problem far exceeds the scope of this paper. Perhaps one will emerge via C++26 reflection with further C++29 additions.

3.7. What to do for ASCII-incompatible char and wchar_t

Not every ordinary and wide character encoding is ASCII-compatible, such as EBCDIC, Shift-JIS, and (defunct) ISO-646, i.e. code units ≤ 0x7f do not represent the same characters as ASCII.

This begs the question: what should ascii_is_digit('0') do on an EBCDIC platform, where this call is ascii_is_digit(char(0xf0)) ? We have three options, discussed below.

3.7.1. Conditionally supported char overloads

We could mandate that the ordinary literal encoding is an ASCII superset for the char overload to exist. This would force a cast (to char8_t) to use the functions on EBCDIC platforms. It is not clear how implementations would treat Shift-JIS; GCC assumes '\\' == '¥' to be true, so this option may not be enough to alleviate the awkwardness of ascii_is_punctuation('¥').

Also, this option is not very useful. It is reasonable to have UTF-8 data stored in a char[] on EBCDIC platforms, and having to perform casts to char8_t would be awkward.

3.7.2. Transcode char to ASCII

We could transcode from the ordinary literal encoding to ASCII and produce an answer for the result of that transcoding. This would be a greater burden for implementations, especially on EBCDIC platforms. The benefit is that ascii_is_digit('0') is always true, although ascii_is_digit(char(0x30)) may not be. However, ascii_is_digit(char8_t(0x30)) is always true.

It probably does not solve the ascii_is_punctuation('¥') case, as implementers may keep transcoding '¥' and '\\' in the same way. It would also give incorrect answers for stateful encodings. There are EBCDIC control characters that do not have an ASCII equivalent, so if we were to do conversions, we would have to decide what, for example, ascii_is_control('\u008B') should produce.

3.7.3. Treat the input as ASCII, regardless of the literal encoding

This is our proposed behavior.

The most simple option is to ignore literal encoding entirely, and assume that char inputs are ASCII-encoded. The greatest downside is that depending on encoding, ascii_is_digit('0') may be false, which may be surprising to the user. However, the main purpose of these functions is to be called with characters taken from ASCII text, so what results they yield when passing literals is not so important.

There are use cases for this behavior on EBCDIC platforms. A lot of protocols (HTTP, POP) and file formats (JSON, XML) are ASCII/UTF-8-based and need to be supported on EBCDIC systems, making these functions universally useful, especially as <cctype> functions cannot easily be used to deal with ASCII on these platforms.

Ultimately, do we want functions to deal with ASCII or the literal encoding? If we want them to be a general way to query the ordinary literal encoding, ascii_is_* is a terrible name, and finding a more general name would prove difficult.

3.8. Naming

3.8.1. Why to include ascii in each name

All proposed functions should have ascii somewhere in their name. This emphasizes that rather than using the literal encoding or execution encoding, these functions operate only on ASCII characters (or Basic Latin code points, depending on how you think of it).

ascii also makes it obvious that no Unicode property tests are performed. Names such as std::is_digit are worth reserving for more general tests.

3.8.2. ascii_is_* vs. is_ascii_*

R4 and previous revisions of this paper used the naming scheme is_ascii_* rather than is_ascii_* for character rests. This was chosen because the name is more natural for English readers. After discussion of R4 in SG16, the scheme was changed to use the ascii_ prefix in all cases because:

• is_ascii_* could be understood as asking: Is this thing I have, whatever it is, considered an ASCII digit? This is a problematic question when considering the is_ascii_digit('0') problem described in §3.7. What to do for ASCII-incompatible char and wchar_t. The ascii_is_* scheme more clearly signals that all functions assume a domain of ASCII, as an encoding assumption.
• The is_ascii_* scheme can be applied consistently to all functions, including ascii_case_insensitive_equal.

3.9. What if the input is a non-ASCII code unit?

Text input is rarely guaranteed to be pure ASCII, i.e. some code units may be > 0x7f. However, we're still interested in ASCII characters within that input. For example, we may

• parse pure ASCII numbers like 123 in a UTF-8 JSON (or other config) file,
• trim ASCII whitespace in HTTP headers, which are encoded with ISO-8859-1,
• parse ASCII-alphanumeric variable names in Lua scripts, where non-ASCII characters can appear (comments, string),
• ...

It is possible (and expected) that the user calls say, ascii_is_digit(U'ö'), at least indirectly. For the sake of convenience, all proposed functions should handle such inputs by

• returning false in the case of all testing functions, and
• applying an identity transformation in transformation/case-insensitive comparison functions.

The proposed behavior also works excellently with any ASCII-compatible encoding, such as UTF-8. Surrogate code units in UTF-8 are all greater than 0x7F, so if we implement say, ascii_is_digit naively by checking c >= '0' && c <= '9', it "just works".

3.10. Why not accept any integer type?

Some people argue that a test like ascii_is_digit('0') is a purely numerical test using the ASCII table, and so passing ascii_is_digit(0x30) should also be valid.

However, this permissive interface would invite bugs. For example, c - '0' is the difference between ASCII characters, not an ASCII character, so passing it into ascii_is_digit would be nonsensical. Static type systems exist for a reason: to protect us from stupid mistakes. While char, char32_t etc. are not required to be ASCII-encoded, they are at least characters, so passing them into our functions is likely something the user intended to do, which we cannot say with confidence about int, unsigned int, etc.

Additionally, if we allowed passing signed integers, we may want to make the behavior erroneous or undefined for negative inputs because ascii_is_digit(-1'000'000) is most likely a developer mistake. Our interface is very simple: it has a wide contract and almost all functions are noexcept. Let's keep it that way!

Lastly, even proponents of passing integer types would not want ascii_is_digit(true) to be valid.

3.11. ASCII case-insensitive views and case transformation algorithms

Ignoring or transforming ASCII case in algorithms is a fairly common problem. Therefore, it may be useful to provide views such as std::views::ascii_lower, algorithms like std::ranges::equal_ascii_case_insensitive, etc.

While case transformations can be implemented naively using std::transform, dedicated functions would allow an efficient vectorized implementation for contiguous ranges, which can be many times faster ([AvoidCharByChar], [AVX-512CaseConv]) Similarly, a case-insensitive comparison function can be vectorized. In fact, POSIX's strncasecmp has been heavily optimized in glibc ([AVX2strncasecmp]), and providing range-based interfaces would allow delegating to these heavily optimized functions.

We intend to propose such utilities in a future paper or revision of this paper. Currently, this proposal is focused exclusively on operations involving character types.

3.12. Why just ASCII?

It may be tempting to generalize the proposed utilities beyond ASCII, e.g. to UTF-8. However, this is not proposed for multiple reasons:

• You cannot pass char8_t into a UTF-8 is_upper function and expect meaningful results. In general, operations on variable-length encodings require sequences of code units. The interface we propose only makes sense for ASCII.
• Unicode utilities are tremendously more complex than ASCII utilities. Some Unicode case conversions even require multi-code-point changes.

3.13. namespace ascii

Instead of "pseudo-namespacing" the proposed functions by including ascii in the name, it would also be possible to create a new namespace ascii which houses all functions. This would have the notable benefit of letting the user opt into shorter functions like:

However, we chose not to do this because it is unusual to create distinct namespace for these small sets of utilities. Furthermore, it may be desirable to have SIMD overloads of these utilities in the future. This begs the question: would it be std::simd::ascii::is_lower or std::ascii::simd::is_lower? Would it be std::views::ascii::lower or std::ascii::views::lower? The simpler option is to avoid namespaces.

4. Implementation experience

A naive implementation of all proposed functions can be found at [CompilerExplorer], although these are implemented as function templates, not as overload sets (as proposed).

A more advanced implementation of some functions can be found in [µlight]. Character tests can be optimized using 128-bit or 256-bit bitsets.

5. Wording

The wording changes are relative to [N5014].

In [tab:headers.cpp], add a new element to C++ library headers table:

In subclause [version.syn], update the synopsis as follows:

In Clause [text], append a new subclause:

6. Acknowledgements

Thanks to Joe Gottman for spotting a mistake in get_hex_digit_value.

7. References