C++ String Benchmark: STL vs. ATL vs. Custom Pool Allocator

Let’s see how STL strings, ATL CString and strings coming from a custom pool allocator perform in a couple of interesting contexts (string vector creation and sorting).

“Looking-glass upon the wall, who is fastest of us all?” — A C++ string class would ask, paraphrasing Snow White.

As you probably already know, there are quite a few different kinds of strings available in C++. I was curious to compare the performance of Microsoft Visual Studio C++ STL string implementation versus ATL CString versus a custom string pool allocator.

Basically, the pool allocator maintains a singly-linked list of chunks, and string memory is carved from each chunk just increasing a string character pointer. When there isn’t enough memory available in the current chunk to serve the requested allocation, a new chunk is allocated. The new chunk is safely linked to the previous chunk list maintained by the allocator object, and the memory for the requested string is carved from this new chunk. At destruction time, the linked list of chunks is traversed to properly release all the allocated memory blocks.

Schema of a custom memory pool allocator.
Custom pool allocator

You can find the complete compilable C++ implementation code on GitHub.

I measured the execution times to create and fill string vectors, and the execution times to sort the same vectors.

TL;DR: The STL string performance is great! However, you can improve creation times with a custom pool allocator.

The execution times are measured for vectors storing each kind of strings: STL’s wstring, ATL’s CString (i.e. CStringW in Unicode builds), and the strings created using the custom string pool allocator.

This is a sample run (executed on a Windows 11 PC with an AMD Ryzen 7 9700X CPU, with code compiled using Visual Studio 2022 in 64-bit Release build mode):

String benchmark showing creation times.
String benchmark: STL vs. ATL’s CString vs. custom string pool allocator

As you can note, the best creation times are obtained with the custom string pool allocator (see the POL1, POL2 and POL3 times in the “Creation” section).

For example:

String typeRun time (ms)
ATL CStringW647
STL std::wstring622
Pool-allocated strings475
Sample execution times for creating and filling string vectors

In the above sample run, the pool-allocated strings are about 36% faster than ATL’s CString, and about 31% faster than STL’s wstring.

This was expected, as the allocation strategy of carving string memory from pre-allocated blocks is very efficient.

I also wanted to take a look at sorting times.

String benchmark showing creation and sorting times.
Sample execution times for creating and sorting string vectors

In this case, the pool-allocated strings are about 9% faster than STL’s std::wstring, and about 87% faster than ATL’s CString.

As you can see, ATL’s CString shows the worst execution times for both creation and sorting. Probably this is caused by CString implementation lacking optimizations like move semantics, and using _InterlockedIncrement/_InterlockedDecrement to atomically update the reference count used in their CoW (Copy on Write) implementation. Moreover, managing the shared control block for CString instances could cause an additional overhead, too.

Historical Note: I recall that I performed a similar benchmark some years ago with Visual Studio 2008 SP1, and in that case the performance of ATL’s CString was much better than std::wstring. I think that move semantics introduced with C++11 and initially implemented in VS 2010 and then refined in the following versions of the VC++ compiler, and more “programming love” given to the MSVC’s STL implementation, have shown their results here in the much improved STL string performance.

Benchmark Variation: Tiny Strings (and SSO)

It’s also possible to run the benchmark with short strings, triggering the SSO (to enable this, compile the benchmark code #define’ing TEST_TINY_STRINGS).

Here’s a sample run:

Benchmark results for tiny strings.
String benchmark with tiny strings: STL vs. ATL vs. custom string pool allocator

As you can see in this case, thanks to the SSO, STL strings win by an important margin in both creation and sorting times vs. ATL. On the other hand, the pool-allocated strings are still the best performer, even in this SSO STL-friendly context.

Where to Put [[nodiscard]]: A Silent Trap with MSVC!

An example of silent sneaky bug, with MSVC giving a *false* sense of security with [[nodiscard]].

Consider the following C++ code snippet:

static [[nodiscard]] int GetSomething()
{
    return 64;
}

If you invoke this function like this:

int main()
{
    GetSomething();
}

the code compiles successfully with MSVC!

Even at warning level 4 (/W4, which I use on almost all my VC++ projects)!

Why am I upset? Because in the above code I do discard the return value of GetSomething(), and even with the [[nodiscard]] attribute applied to the function, and compiling at /W4, the MSVC C++ compiler does not complain. I do want the compiler to complain in this case! 🙂

Dear MSVC C++ compiler, speak up! Say something like: “Hey, you discarded the return value of GetSomething(), which is marked with the [[nodiscard]] attribute. Please fix your code.”

The interesting thing is that, if you change the position of the [[nodiscard]] attribute, putting it before “static”, you get the expected warning:

// Was: 
//   static [[nodiscard]] int GetSomething()
//
// Now put [[nodiscard]] at the beginning of the line,
// before "static":
//
[[nodiscard]] static int GetSomething()
{
    return 64;
}

Now the C++ compiler complains when trying to compile the above code snippet inside main(), with: “warning C4834: discarding return value of function with [[nodiscard]] attribute”. Great!

The problem is that, a wrong behavior is being silently accepted, even at warning level 4! The MSVC compiler gave me a false sense of security!

I compiled the above code in C++20 mode at /W4 with Visual Studio 2022.

Interestingly, both Clang and GCC refuse to compile the code with [[nodiscard]] in the wrong position, while unfortunately MSVC does silently swallow that.

For example, Clang emits the following clear error message:

Clang correctly emits an error when [[nodiscard]] is written in the wrong position.

On Windows, my favorite IDE for C++ (and some C#) programming is Visual Studio, and I use the Microsoft Visual C++ compiler (MSVC) to compile my C++ code. I hope that this silent sneaky compiler bug is fixed ASAP.

The C++20’s u8/char8_t Backward-Compatibility Fiasco

There is a myth according to which every new C++ version is backward compatible with the older versions. While there is a common general trend to try and keep backward compatibility across different C++ versions, that’s not always the case.

A “recent” example is provided by C++20’s u8 and char8_t.

Consider the following C++ code snippet:

// This function takes a UTF-8 string as input,
// and does something with it.
void DoSomething(const char* utf8Text);

...

int main()
{
    // A UTF-8 string literal
    auto s = u8"Connie plus some UTF-8 stuff...";

    DoSomething(s);
}

The above code compiles successfully in C++17 mode.

Now you want to be cool and update your language standard to C++20, because, you know, we are in 2026 😉

Well, surprise, surprise…the same code will not compile in C++20 mode!

MSVC compilation error involving u8 and char8_t when switching to C++20 mode.
Microsoft VC++ compilation error when switching to C++20 mode.

The MSVC compiler complains with the following error message:

‘void DoSomething(const char *)’: cannot convert argument 1 from ‘const char8_t *’ to ‘const char *’

The reason for that is that until C++20 a u8 string literal was interpreted as a const char array; on the other hand, C++20 did introduce a breaking change, and the same u8 string literal is now a const char8_t array in the new standard!

u8″…” SyntaxTypeEncoding
Until C++20
(C++11, 14, 17)
const char[N]UTF-8
Since C++20const char8_t[N]UTF-8

The DoSomething function expects a char-based string, not a char8_t-based string, and so the C++ compiler complains in C++20 mode.

Of course, this was a simple example to illustrate the nature of the problem. But suppose that you are working with large libraries and existing codebases that used to compile just fine in previous C++ standards (e.g. C++17), and now the same code break all of a sudden when you switch to C++20 mode!

To mitigate the problem, an option could be to avoid the use of the u8 prefix when possible.

For example, this is what Google’s C++ Style Guide currently suggests:

When possible, avoid the u8 prefix. It has significantly different semantics starting in C++20 than in C++17, producing arrays of char8_t rather than char, and will change again in C++23.

What C++ Version Should You Target?

What C++ version should you target in your codebase? There’s no easy answer, except maybe “it depends”…

This is an interesting question, with not an easy answer.

I would say that the correct answer is: “It depends.”

Modern C++ started with C++11. I think C++11 is the most impactful version of C++ to date, as it introduced very important and convenient features like: range-for loops, lambdas, move semantics, standard smart pointers like std::unique_ptr and std::shared_ptr, etc.

C++14 was kind of a small improvement for things that didn’t make it into C++11. For example, C++11 had std::make_shared for std::shared_ptr, but it clearly lacked the corresponding std::make_unique for std::unique_ptr. Well, std::make_unique (and other small features, like supporting auto in lambda parameters) were added in C++14.

Then it came C++17. Again, no big new features here, at least not at the C++11 impact level, but some convenient stuff, like: attributes (e.g. [[nodiscard]], [[fallthrough]], or [[maybe_unused]]); static_assert with no messages (a.k.a. terse static_assert); structured bindings, which come in handy in some cases like when iterating through containers like std::map; new standard library components that can be useful, like std::optional, std::variant, etc.

Some library features that were previously deprecated were definitively removed in C++17; so that could break some existing legacy C++ code that previously compiled just fine in your projects.

And this brings an important point when picking the C++ version for your codebase: Does using C++ version X > Y break existing code? Is it compatible with older legacy libraries that you don’t want to or cannot upgrade or fix? Does it break some binary compatibility interface (ABI)1? Is it supported by your toolchain(s)?

For example, if you want to target Windows XP with Microsoft Visual C++ compiler, the latest toolset that has Windows XP support is v141_xp, which is a toolset from Visual Studio 20172, having full C++14 support, and partial C++17 support (for example: you can use C++17 features like terse static_assert, std::optional, and std::variant; but not other features like std::shared_mutex).3

In general, I would say that these days targeting C++17 is fine, and if your toolset supports C++20, and if it doesn’t break any existing library or legacy component you depend on (and you don’t want to/are unable to fix), then C++20 can be a good choice, too.

Keep in mind that today in 2026 (August 18th, 2026), Google’s C++ Style Guide still targets C++204, and LLVM C++17.

Google's C++ Style Guide targets C++20 (August 18th, 2026).
Google’s C++ Style Guide targets C++20 as of today (August 18th, 2026).

LLVM Coding Standard targets C++17 (August 18th, 2026).
LLVM Coding Style targets C++17 (August 18th, 2026).

Note also that targeting a given C++ version, like C++20, does not mean that you have to use all the features available in that language version!

For example, talking about the C++20 adoption by Google’s C++ Style Guide, it’s worth noting that, the main “Big Three Features” of C++20, i.e. concepts, modules, and coroutines, are used sparingly, or not used at all!

For example, on concepts:

Use concepts sparingly.

And on coroutines:

Use only coroutine libraries that have been approved for project-wide use by your project leads. Do not roll your own promise or awaitable types.

Moreover, on C++20 modules:

Do not use C++20 Modules.


Remember that, at the end of the day, the key is to write code that is clear, easy to read, understand and maintain, and not some mess of esotheric advanced features that make it impossible or very hard to understand what’s going on, or unstable features not well supported by the available toolsets.

  1. See for example the C++20 [[no_unique_address]] problem with MSVC. ↩︎
  2. It comes with Visual Studio 2019, too. ↩︎
  3. You can read more about that in this StackOverflow answer. ↩︎
  4. Despite reading: “The C++ version targeted by this guide will advance (aggressively) over time.” (emphasis mine), they are not even targeting C++23 in (at least mid August) 2026. ↩︎

Why We Should Double-Check the AI Output: Claude Lies (AKA AI Hallucinations)

Another example showing how AI output should be carefully reviewed, and not accepted passively.

I was “discussing” with Claude about using #define guards vs. #pragma once for C++ header files. At some point, Claude wrote that:

If your library is only ever going to target modern mainstream compilers (which, realistically, is true for almost everyone today), plain #pragma once is fine and is what most modern C++ style guides (including Google’s and LLVM’s) now recommend by default.

I did remember that, last time I checked, Google’s C++ Style Guide required #define guards (not #pragma once) for header files. That is still true as of today (August 18th, 2026); quoting Google’s C++ Style Guide on this point:

All header files should have #define guards to prevent multiple inclusion. The format of the symbol name should be <PROJECT>_<PATH>_<FILE>_H_.

So I made my point to Claude, and it replied apologizing for its error:

You’re right, and I apologize for the error. Looking at the actual page, Google’s C++ Style Guide explicitly requires #define guards, not #pragma once

This is another example showing how AI output cannot be 100% trusted, and should always be carefully reviewed.

Google's C++ Style Guide point on using #define guards for header files.
Google’s C++ Style Guide requiring #define guards for header files.

Claude AI wrongly stating that Google's C++ Style Guide recommends #pragma once, and its apologies after my correction.
An excerpt of my “discussion” with Claude AI on #define guards vs. #pragma once.

On AI Hallucinations

I briefly asked another AI (Gemini) about AI hallucinations (like the previous example with Claude), and it interestingly pointed out that:

AI models predict the next most likely word based on patterns in data.
They do not have real-world awareness or a true concept of “truth.”

That is something to keep in mind when dealing with AI.

Google Gemini AI explaining AI hallucinations.
Google Gemini AI on AI hallucinations.

Why We Should Double-Check the AI Output: A Bug Which Wasn’t

This is a concrete real-world example showing how AI-generated results should not be 100% completely trusted.

Recently I asked Claude to review the code of my WinReg C++ library (which is a C++ high-level wrapper around the low-level C-interface Windows Registry API).

As a result of its analysis, Claude reported that there were zero-length bugs in my code, in particular Claude stated that zero-length REG_SZ/REG_EXPAND_SZ values crash the GetStringValue, GetExpandStringValue, TryGetStringValue and TryGetExpandStringValue methods of the RegKey class.

In particular, Claude noted that I correctly guarded against dataSize == 0 in the binary-returning getters (like RegKey::GetBinaryValue), but the string getters do not have such guard; they unconditionally do:

result.resize((dataSize / sizeof(wchar_t)) - 1);

Claude specified that REG_SZ and REG_EXPAND_SZ values can be legitimately stored with cbData == 0 (zero bytes, no NUL at all; different from an empty string made by a single NUL ‘\0’).

If you substitute zero for the dataSize variable in the above statement, you end up with:

result.resize(SIZE_MAX);

which would throw a std::length_error exception.

An alleged bug reported by Claude AI on a zero-length edge case in my WinReg C++ library.

Claude proposed to fix the above code using the same edge-case check logic I had already implemented in the binary getters to guard against the zero-length case:

if (dataSize == 0)
{
    result.clear();
}
else
{
    result.resize((dataSize / sizeof(wchar_t)) - 1);
}

My WinReg library is quite battle-tested, and there were bugs related to some edge cases that I had already fixed, so I was curious, and tried writing a zero-length string value in the registry, and read it back with my existing code. And I noted that (at least in Windows 11 where I tested my code) the dataSize == 0 condition was not hit at run-time.

That is because in my C++ code I invoke the RegGetValue(W) API, which by contract guarantees to return a NUL-terminated string, even if the string stored in the registry doesn’t have a NUL-terminator. (This is not the case for older APIs like RegQueryValueEx.)

I replied to Claude pointing that out, and Claude corrected itself:

Correcting Claude.
Claude self-correction after I pointed out its bug analysis was probably wrong.

You’re right, and thanks for the pushback — I should have accounted for RegGetValueW‘s null-termination guarantee before flagging that as a bug.

Those AI tools can be very powerful, but the key takeway here is that we should not forget that they are just tools, and we should not trust AI-generated output and code 100%, because bugs and wrong assumptions can be hidden in that AI code, too.

How to Declare a C++ Function that Takes a Blob of Memory?

Discussing several options, starting from the good old C-style void* pointer.

An interesting question you may ask in C++ is: “How would you declare a function that takes a blob of memory as input?”

For example, think of a function that hashes some input data (using SHA-256, or whatever hash algorithm), or a function that takes some binary data and writes that to disk.

Coming from my C background, an option that came to mind would certainly be:

void DoSomething(const void* p, size_t numBytes)

You simply pass a const void* pointer to the beginning of the input memory block, and the total size of the memory block, expressed in bytes.

Then, some C++ programmer could start complaining: “Hey, why do you use the unsafe old C-style void* pointer? Use some safe explicit type like uint8_t, which clearly represents an 8-bit byte!”.

So, they propose to “step up” to the following prototype:

void DoSomething(const uint8_t* p, size_t numBytes)

Now, suppose that you want to pass to this function a custom structure, like this:

struct MyCustomData {
    ...
};

MyCustomData data;

With the original void* version, you can invoke the function simply and clearly like this:

DoSomething(&data, sizeof(data));

The code is very clear and straightforward: you pass a pointer to the custom data structure, and its size in bytes. That’s it. Simple and clear.

On the other hand, with the “safe and modern” uint8_t prototype, the function call gets more complicated, as you need to add a type cast:

// void DoSomething(const uint8_t* p, size_t numBytes)
//
// DoSomething(&data, sizeof(data));
//
// This gives a compiler error when the function expects 
// a const uint8_t* instead of const void*, something like:
//
// Error: cannot convert 'MyCustomData*' to 'const uint8_t*'
//
// You need an explicit cast in this case!
DoSomething(
    reinterpret_cast<const uint8_t*>(&data), 
    sizeof(data)
);

Why should people complexify and uglify their C++ code with the uint8_t pointer (or std::byte), when void* works just fine??

Moreover, someone could even say: “Hey, in modern C++20, we have std::span! Use it!”

Well, congratulations for rising the complexity and noise of the code even further!

In fact, std::span is a class template, and somebody would suggest to make the function that processes the generic memory blob a function template! Really? Something like this??

template <typename T>
void DoSomething(std::span<T> data)

Or maybe something even more complicated, like this?

template <typename T, std::size_t N>
void DoSomething(std::span<T, N> data)

// Or this?
template <typename T, std::size_t N>
void DoSomething(std::span<const T, N> data)

Wow. With std::span the complexity-meter bumps in the red zone and goes even higher!

Someone may suggest something like a std::span<const uint8_t>? But that’s still more complex than the initial void* signature.

Do you want a pointer to a generic memory blob? C++ has already if from C: it’s called void*! Use it and enjoy.

I really dislike this attitude of some “modern” C++ programmers, that make choices that have the effect of making the code more complex, uglier and harder to write and understand.

It seems that some people are really losing the taste for good readable code.

Some good old habit from C can still be positively used in C++, like the void* pointer and the size parameters.


BTW: As a nice addition, if you use SAL annotations, the function could be decorated a bit to help code analyzers detecting memory bugs:

void DoSomething(
  _In_reads_bytes_(numBytes) const void * p,
  _In_ size_t numBytes
);

The _In_reads_bytes_ annotation applied to the pointer parameter explicitly states that the pointer points to input read-only memory (_In_reads_), and the size of this input buffer expressed in bytes (_bytes_) is represented by the numBytes parameter.

In this way, we still keep the clarity and simplicity of the function invocation:

DoSomething(&data, sizeof(data));

while also adding pieces of information that are helpful to spot memory bugs with code analyzers and other tools.

If you want to learn more about SAL annotations, you can start reading this MSDN documentation: Using SAL Annotations to Reduce C/C++ Code Defects.

The char-TCHAR-wchar_t Pendulum in Windows API Native C/C++ Programming

A trip down memory lane for Windows C/C++ text-related coding patterns: from char, to TCHAR, to wchar_t… and back to char?

I started learning Windows Win32 API programming in C and C++ on Windows 95 (I believe it was Windows 95 OSR 2, in about late 1996 or early 1997, with Visual C++ 4). Back then, the common coding pattern was to use char for string characters (as in Amiga and MS-DOS C programming). For example, the following is a code snippet extracted from the HELLOWIN.C source code from the “Programming Windows 95” book by Charles Petzold:

static char szAppName[] = "HelloWin";

// ...

hwnd = CreateWindow(szAppName,
                    "The Hello Program", 
                    ... 

After some time, I learned about the TCHAR model, and the wchar_t-based Unicode versions of Windows APIs, and the option to compile the same C/C++ source code in ANSI (char) or Unicode (wchar_t) mode using TCHAR instead of char.

In fact, the next edition of the aforementioned Petzold’s book (i.e. the fifth edition, in which the title went back to the original “Programming Windows”, without explicit reference to a specific Windows version) embraced the TCHAR model, and used TCHAR instead of char.

Using the TCHAR model, the above code would look like this, with char replaced by TCHAR:

static TCHAR szAppName[] = TEXT("HelloWin");

// ...

hwnd = CreateWindow(szAppName,
                    TEXT("The Hello Program"), 
                    ...

Note that TCHAR is used instead of char, and the string literals are enclosed or “decorated” with the TEXT(“…”) preprocessor macro. Note however that, in both cases, the same CreateWindow name is used as the API identifier.

Note that Visual C++ 4, 5, 6 and .NET 2003 all defaulted to ANSI/MBCS (i.e. 8-bit char strings, with TCHAR expanded to char).

When I moved to Windows XP, and was still using the great Visual C++ 6 (with Service Pack 6), the common “modern” pattern for international software was to just drop ANSI/MBCS 8-bit char strings, and use Unicode (UTF-16) with wchar_t at the Windows API boundary. The new Unicode-only version of the above code snippet became something like this:

static wchar_t szAppName[] = L"HelloWin";

// ...

hwnd = CreateWindow(szAppName,
                    L"The Hello Program", 
                    ...

Note that wchar_t is used this time instead of TCHAR, and string literals are decorated with L”…” instead of TEXT(“…”). The same CreateWindow API name is used. Note that this kind of code compiles just fine in Unicode (UTF-16) builds, but will fail to compile in ANSI/MBCS builds. That is because in ANSI/MBCS builds, CreateWindow, which is a preprocessor macro, will be expanded to CreateWindowA (the real API name), and CreateWindowA expects 8-bit char strings, not wchar_t strings.

On the other hand, in Unicode (UTF-16) builds, CreateWindow is expanded to CreateWindowW, which expects wchar_t strings, as provided in the above code snippet.

One of the problems with “ANSI/MBCS” (as they are identified in Visual Studio IDE) 8-bit char strings for international software was that “ANSI” was just insufficient for representing characters like Japanese kanjis or Chinese characters, just to name a few. While you may not care about those if you are only interested in writing programs for English-speaking customers, things become very different if you want to develop software for an international market.

I have to say that “ANSI” was a bit ambigous as a code page term. To be more precise, one of the most popular encoding for 8-bit char strings on Windows was Windows code page 1252, a.k.a. CP-1252 or Windows-1252. If you take a look at the representable characters in CP-1252, you’ll see that it is fine for English and Western Europe languages (like Italian), but it is insufficient for Japanese or Chinese, as their “characters” are not represented in there.

Note that CP-1252 is not even sufficient for some Eastern Europe languages, which are better covered by another code page: Windows-1250.

Another problem that arises with these 8-bit char encodings is ambiguity. For example, the same byte 0xC8 represents È (upper case E grave) in Windows-1252, but it maps to this completely different grapheme Č in Windows-1250.

So, moving to Unicode UTF-16 and wchar_t in Windows native API programming solved these problems.

Note that, starting with Visual C++ 2005 (that came with Visual Studio 2005), the default setting for C/C++ code was using Unicode (UTF-16) and wchar_t, instead of ANSI/MBCS as in previous versions.


More recently, starting with some edition of Windows 10 (version 1903, May 2019 Update), there is an option to set the default “code page” for a process to Unicode UTF-8. In other words, the 8-bit -A versions of the Windows APIs can default to Unicode UTF-8, instead of some other code page.

So, for some Windows programmers, the pendulum is swinging back to char!

The IsoCpp.org Process for Suggesting Articles Is Broken and Should Be Fixed

The process of submitting article suggestions to IsoCpp.org can be kind of “frustrating”, with inconsistencies in acceptance timing and a lack of communication. Making suggestions requires some effort, yet the outcomes feel random. I propose some improvements.

I have suggested several articles to the IsoCpp.org Web site. Some article suggestions were published just a few hours after sending them; others the next day or two, others after a week or two, while other suggestions seemed like lost by anonymous persons in a “black hole”. Who processed those suggestions? Why were those rejected?

This process seems kind of random and unprofessional, and not respectful for the time we put in suggesting articles.

In fact, for suggesting an article, it’s not sufficient to copy-and-paste a link to the content and just click a “Suggest” button. You have to prepare a little document, following a pattern and some editorial guides made available from the IsoCpp Web site. It does take some time.

Then, you click the button to make the suggestion… and it’s like a random coin toss! Will the suggestion be accepted? Will the suggestion be discarded? When? Why? By whom?

The process is clearly broken, and should be fixed, out of respect for the time of the people who made a suggestion, and for what should be a quality Web site that lists links to relevant content.

A possible fix to the process could be like this:

Once you make a suggestion, an email is sent to you, saying that the IsoCpp editorial team has received the suggestion, and will reply in a maximum given period of time: one week, two weeks, whatever. But do give a time limit, and don’t just disappear! I think a 15-day time limit for a reply would be acceptable.

Then, do send a reply to the person suggesting the article, be it positive or negative. But do send a reply! If the suggestion is accepted, say thank you and give a link to the Web page containing the suggestion.

On the other hand, if the suggestion is not accepted, say thank you again, and do give a reason for the refusal. And also give the person suggesting the article an option to further discuss that via email with the editor who refused the suggestion, with the option to discuss that with other editors, too.

Moreover, once a person has a certain number of approved suggestions, let the system automatically approve their suggestions by default. This “privilege level” could be revoked if a certain number of unworthy suggestions or suggestions not relevant for the IsoCpp topics are made.

Finding the Next Unicode Code Point in Strings: UTF-8 vs. UTF-16

How does the simple ASCII “pch++” map to Unicode? How can we find the next Unicode code point in text that uses variable-length encodings like UTF-16 and UTF-8? And, very importantly: Which one is *simpler*?

When working with ASCII strings, finding the next character is really easy: if p is a const char* pointer pointing to the current char, you can simply advance it to point to the next ASCII character with a simple p++.

What happens when the text is encoded in Unicode? Let’s consider both cases of the UTF-16 and UTF-8 encodings.

According to the official “What is Unicode?” web page of the Unicode consortium’s Web site:

The Unicode Standard provides a unique number for every character, no matter what platform, device, application or language.

This unique number is called code point.

In the UTF-16 encoding, a Unicode code point is represented using 16-bit code units. In the UTF-8 encoding, a Unicode code point is represented using 8-bit code units.

Both UTF-16 and UTF-8 are variable-length encodings. In particular, UTF-8 encodes each valid Unicode code point using one to four 8-bit byte units. On the other hand, UTF-16 is somewhat simpler: In fact, Unicode code points are encoded in UTF-16 using just one or two 16-bit code units.

EncodingSize of a code unitNumber of code units for encoding a single code point
UTF-1616 bits1 or 2
UTF-88 bits1, 2, 3, 4

I used the help of AI to generate C++ code that finds the next code point, in both cases of UTF-8 and UTF-16.

The functions have the following prototypes:

// Returns the next Unicode code point and number of bytes consumed.
// Throws std::out_of_range if index is out of bounds or string ends prematurely.
// Throws std::invalid_argument on invalid UTF-8 sequence.
[[nodiscard]] std::pair<char32_t, size_t> NextCodePointUtf8(
    const std::string& str, 
    size_t index
);

// Returns the next Unicode code point and the number of UTF-16 code units consumed.
// Throws std::out_of_range if index is out of bounds or string ends prematurely.
// Throws std::invalid_argument on invalid UTF-16 sequence.
[[nodiscard]] std::pair<char32_t, size_t> NextCodePointUtf16(
    const std::wstring& input, 
    size_t index
);

If you take a look at the implementation code, the code for UTF-16 is much simpler than the code for UTF-8. Even just in term of lines of code, the UTF-16 version is 34 LOC, vs. the UTF-8 version which is 84 LOC! So, the UTF-8 version takes more than 2X LOC than UTF-16! In addition, the code of the UTF-8 version (which I generated with the help of AI) is also much more complex in its logic.

For more details, you can take a look at this GitHub repo of mine. In particular, the implementation code for these functions is located inside the NextCodePoint.cpp source file.

Now, I’d like to ask: Does it really make sense to use UTF-8 to process Unicode text inside our C++ code? Is the higher complexity of processing UTF-8 really worth it? Wouldn’t it be better to use UTF-16 for Unicode string processing, and just use UTF-8 outside of application boundaries?