My Favorite Bugs: Invalid Surrogate Pairs

TL;DR

Prime Big Deal Days · Oct 6–7Offer from Amazon

Get the latest gadgets delivered free — and shop member deals

• Fast, free delivery on millions of items
• Access to Prime Big Deal Days deals on October 6–7
• Prime Video, Amazon Music and more included

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

A rare bug was discovered in a collaborative editing tool where inserting certain emojis caused the sync process to silently fail. The issue stems from how JavaScript handles surrogate pairs in Unicode. This highlights challenges in handling complex characters in web apps.

Developers have confirmed a bug in a collaborative editing platform where inserting certain emojis causes the sync process to silently stop working. The issue is linked to how JavaScript handles complex Unicode characters, affecting real-time editing features and highlighting underlying technical challenges.

The problem was traced to surrogate pairs used to encode emojis above U+FFFF in Unicode. When a user inserts or modifies emojis like 🤠 (U+1F920) adjacent to other multi-byte characters, the underlying CRDT library’s splice operation can split a surrogate pair. This results in invalid strings with orphaned surrogates, which then cause errors during URI encoding, leading to a silent failure in the sync process.

Specifically, the bug only manifests when edits occur at precise byte offsets where surrogate pairs are split. Not all emoji insertions trigger the problem, only those involving characters requiring surrogate pairs. The developers confirmed that the error was uncaught, causing the sync to halt without visible errors or crash logs.

Why It Matters

This bug underscores the complex challenges in handling Unicode characters in web applications, especially those involving real-time collaboration and CRDTs. It reveals how subtle issues with surrogate pairs can lead to data loss or silent failures, emphasizing the need for robust Unicode handling and error management in modern web development.

Background

Prior to this discovery, developers suspected network issues or websocket instability caused sync failures, but no concrete cause was identified. The bug was finally isolated through detailed investigation of emoji insertion patterns, particularly involving characters above U+FFFF that require surrogate pairs in UTF-16 encoding. The problem was exacerbated by the use of JavaScript’s string methods, which operate at the code unit level, not code points or grapheme clusters.

“This bug shows how intricate Unicode handling can have real consequences in web apps, especially with collaborative features relying on precise string manipulations.”

— Lead Developer

“Inserting specific emojis triggered a silent failure in our sync process, revealing a subtle but critical flaw in how we process Unicode characters.”

— Product Manager

What Remains Unclear

It remains unclear how widespread this issue might be across other applications or libraries that handle Unicode and CRDTs. The full scope of affected characters or scenarios is still being evaluated, and the team is exploring more robust handling strategies.

What’s Next

The developers plan to implement stricter validation of surrogate pairs during string operations and improve error handling to prevent silent failures. They will also release patches to address the bug and conduct further testing with complex Unicode characters.

Key Questions

What are surrogate pairs in Unicode?

Surrogate pairs are a way to encode characters outside the Basic Multilingual Plane (above U+FFFF) in UTF-16. They consist of a high surrogate and a low surrogate, each a 16-bit code unit, combined to represent a single character like emojis.

Why did this bug cause the sync process to silently fail?

When surrogate pairs are split during string operations, they produce invalid strings with orphaned surrogates. Passing these to URI encoding functions causes errors that are not caught, halting the sync without visible errors.

Is this issue limited to emojis?

Primarily, yes. The bug manifests with characters that require surrogate pairs, such as many emojis above U+FFFF. Simple ASCII characters are unaffected.

How are developers addressing this problem?

Developers plan to add validation to detect and handle surrogate pairs correctly during string manipulations and improve error handling to prevent silent failures in real-time sync systems.