The Language Server Protocol was announced on 27 June 2016. On 28 June 2016 — the next day — Raph Levien opened issue #19 against the spec repository, titled “Clarify units of ‘character offset’ in Position.” The answer, once it arrived, was UTF-16 position encoding: the character field of every position in the protocol counts 16-bit code units, because the reference client is written in TypeScript and JavaScript strings are UTF-16 internally.
That issue was closed in December 2016. The underlying problem was not addressed until LSP 3.17, dated 10 May 2022 in the spec’s own changelog — five years and ten months later — and even then only by adding a negotiated capability on top of the original behaviour. The 3.18 specification, which describes itself as “the current 3.18.x version of the language server protocol” and is still under development, carries a changelog entry dated 4 June 2026. Checked against its raw source on 2026-09-04: UTF-16 is still the default, and is still, in that document’s own words, “a mandatory encoding.”
This is the cleanest case study I know of an implementation detail of one client becoming a permanent obligation for every implementer of a standard. It is worth walking through precisely, because the mechanism that produced it is not exotic and the shape of it recurs everywhere.
The Question Nobody Thought Was a Question
Position is the most-used type in LSP. Completion, hover, go-to-definition, rename, diagnostics, semantic tokens, inlay hints — every one of them either sends or receives one. The definition is unchanged between 3.17 and the current 3.18 document:
“Position in a text document expressed as zero-based line and zero-based character offset. A position is between two characters like an ‘insert’ cursor in an editor.”
interface Position { line: uinteger; character: uinteger;}line is unambiguous. character is not, and the 2016 spec did not say what it counted. In a protocol that carries UTF-8 JSON between a TypeScript editor and a compiler written in C++, Rust, Go, Java or Haskell, “character offset” is three different numbers depending on who is asked.
Levien’s issue was not pedantry. It was the observation that a field named after a unit that does not exist will be implemented as whatever the reference implementation happens to do — and that once several implementations exist, that becomes the definition.
Code Unit, Code Point, Grapheme Cluster
Three distinct things get called “a character,” and the distinction is the whole article.
A code point is a Unicode scalar value — an integer between U+0000 and U+10FFFF assigned to an abstract character. é is U+00E9. 漢 is U+6F22. 🎉 is U+1F389.
A code unit is a fixed-width piece of a particular encoding. UTF-8 uses 8-bit units and spends one to four of them per code point. UTF-32 uses 32-bit units and spends exactly one. UTF-16 uses 16-bit units and spends one for anything in the Basic Multilingual Plane and two — a surrogate pair — for anything above U+FFFF.
A grapheme cluster is what a person means by “one character”: the smallest unit a text cursor should move over. Unicode defines it in Annex #29. A ZWJ emoji sequence like 👩💻 is one grapheme cluster made of three code points, joined by U+200D.
Here is what those sequences actually cost:
| Sequence | Code point(s) | UTF-8 bytes | UTF-16 units | UTF-32 units | Grapheme clusters |
|---|---|---|---|---|---|
a | U+0061 | 1 | 1 | 1 | 1 |
é | U+00E9 | 2 | 1 | 1 | 1 |
漢 | U+6F22 | 3 | 1 | 1 | 1 |
🎉 | U+1F389 | 4 | 2 | 1 | 1 |
👩💻 | 3 code points | 11 | 5 | 3 | 1 |
Three observations fall straight out of that table. ASCII hides the entire problem: every column agrees. A Latin-1 or CJK character separates UTF-8 from the other two but leaves UTF-16 and UTF-32 identical. Only a character outside the BMP — an emoji, a rarer CJK extension ideograph, a mathematical alphanumeric symbol — makes UTF-16 and UTF-32 diverge. And nothing but grapheme segmentation gets you what the user thinks is happening.
That is why the bug survived. For most of a corpus, most of the time, the encodings agree, and the disagreement surfaces as an off-by-a-few error in a rename or a squiggly underline drawn two characters to the left.
The Same Cursor, Three Different Numbers
Take one line of source and put the cursor immediately after the emoji:
let x = "aé漢🎉";The prefix let x = " is nine ASCII characters, so all three counting schemes agree up to that point. After that they separate:
| Cursor sits after | UTF-8 byte offset | UTF-16 code-unit offset | UTF-32 code-point offset |
|---|---|---|---|
| the opening quote | 9 | 9 | 9 |
a | 10 | 10 | 10 |
é | 12 | 11 | 11 |
漢 | 15 | 12 | 12 |
🎉 | 19 | 14 | 13 |
| end of line | 21 | 16 | 15 |
One cursor position, three correct answers: 19, 14, 13. A client that computes 13 and a server that reads 14 will disagree about where a rename starts, and neither is wrong about anything except which document they thought they were sharing.
There is a fourth case that only UTF-16 produces. 🎉 occupies UTF-16 units 12 and 13. Offset 13 is a legal uinteger that points at the low half of a surrogate pair — a position that names no character at all. In UTF-32 no such offset exists; in UTF-8 the analogous mid-sequence byte offset does exist but is at least detectable as a continuation byte. clangd carries an explicit guard for this case, and only on the UTF-16 branch of its offset conversion.
Why UTF-16 Won: It Was Never Chosen
The reason is in ECMA-262, §6.1.4, “The String Type” (checked 2026-09-04):
“The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value.”
and, two sentences later:
“The length of a String is the number of elements (i.e., 16-bit values) within it.”
So in JavaScript, "🎉".length is 2, and "aé漢🎉".length is 5. Every string index in every JavaScript program is a UTF-16 code-unit offset, whether the author knows it or not.
VS Code is a TypeScript program. Its editor model, its text buffer, its selection arithmetic and its extension API all speak that index. When the same team specified a wire type called character, they did not decide to use UTF-16 code units — they wrote down the number their language already had. There is no design memo to argue with, because there was no design decision. The unit was inherited.
That is the whole failure mode: a protocol specified alongside one dominant client, in one language, will silently adopt that language’s primitives wherever the spec text is imprecise. The gap between “character offset” and “UTF-16 code-unit offset” is the size of the hole an entire implementation detail fits through. LSP’s broader design — capability negotiation, ignore-unknown-fields forward compatibility — is unusually careful about precisely this class of problem. It was careful about features and casual about units.
What a Rust, C++ or Go Server Has to Build
None of the compilers behind the major language servers store text as UTF-16. Rust String is UTF-8. Go string is a UTF-8 byte slice. Clang works in byte offsets. Every one of them therefore maintains a second coordinate system that exists solely to answer LSP position arithmetic — arithmetic that has nothing to do with the language being analysed.
The spec is explicit about where the cost lands:
“Implementation considerations: since the conversion from one encoding into another requires the content of the file / line the conversion is best done where the file is read which is usually on the server side.”
rust-analyzer ships a dedicated crate for it, lib/line-index. Its LineIndex struct has three fields: the byte offset of each line start, the total length, and line_wide_chars, documented as “List of non-ASCII characters on each line.” That third field is a per-line side table of every multi-byte character’s byte extent, built when the document is indexed and kept in memory for its lifetime. The crate also carries two parallel position types — LineCol, “(line, column) information in the native, UTF-8 encoding,” and WideLineCol, the same thing under WideEncoding::Utf16 or WideEncoding::Utf32.
clangd converts on demand. Its measureUnits helper is commented, disarmingly: “Conceptually, this converts to the encoding, truncates to CodeUnits, converts back to UTF-8, and returns the length in bytes.” Its lspLength walks code points one at a time. And its fallback, when nothing has been negotiated, is hard-coded: return Enc ? *Enc : OffsetEncoding::UTF16;. clangd also exposes an --offset-encoding command-line flag whose help text reads “Force the offsetEncoding used for character positions. This bypasses negotiation via client capabilities” — a flag that exists because people needed to override the negotiation from outside the protocol entirely.
gopls is the most explicit about the tax. The header comment of protocol/mapper.go states that “gopls uses four main representations of position” — byte offsets, go/token positions, the CLI’s line/column/offset triple, and the LSP one, where “Characters (columns) are measured in UTF-16 codes.” Its Mapper type keeps a lazily-built line-start table plus a nonASCII boolean, so the UTF-16 conversion can be skipped entirely on pure-ASCII files — an optimisation that only exists because the conversion is otherwise on the hot path.
| Server | Native text unit | What it maintains for LSP |
|---|---|---|
| rust-analyzer | UTF-8 byte offsets (TextSize) | A line-index crate: line starts plus a per-line table of non-ASCII characters; separate LineCol and WideLineCol types |
| clangd | UTF-8 byte offsets | On-demand code-point iteration in lspLength and measureUnits; a hard-coded UTF-16 fallback; an --offset-encoding escape hatch |
| gopls | UTF-8 byte offsets and token.Pos | A fourth position representation purely for the protocol, with an ASCII fast path guarded by a nonASCII flag |
Three unrelated projects, three languages, one shared obligation — and none of it is about Rust, C++ or Go.
The Survey That Made the Cost Legible
The complaint that finally moved things was issue #872, opened 30 November 2019 by Fangrui Song (MaskRay), an LLVM contributor. Its first line is blunt:
“a lot of language servers and non-JavaScript based clients have been inconvenienced due to the UTF-16 position counting.”
What makes #872 useful rather than merely irritated is that it carried data. Song surveyed what LSP clients were actually counting in:
| Language client | Counting method for Position.character | Note |
|---|---|---|
| eglot | UTF-32 code points | UTF-32 by default, optionally UTF-16 |
| LanguageServer-neovim | UTF-32 code points | — |
| lsp-mode | UTF-32 code points | — |
| vim-lsp | UTF-32 code points | — |
| ycm | UTF-16 code units | Would prefer UTF-8, but only if standardized |
| VSCode | UTF-16 code units | — |
Four of six clients had independently converged on code points and were converting at the boundary. The protocol’s mandatory default was the internal representation of exactly one participant in its own ecosystem — plus one holdout that would have preferred UTF-8 and said so on the record.
That was the fourth documented attempt to move it. Issue #376, “Change character units from UTF-16 code unit to Unicode codepoint,” had been opened by the same author on 13 January 2018. Pull request #709, “Change character offsets from UTF-16 codeunits to Unicode codepoints,” was opened 29 March 2019 and closed unmerged in May 2021. Outside the spec repo, rust-lang/rls issue #630, “Character offsets from the client are utf-16 code units,” dates to 16 December 2017 — an implementer hitting it independently, from the far side of the boundary.
Issue #872 itself is still open as of 4 September 2026.
Six Years, and the Fix Was a Second Code Path
| Date | Event |
|---|---|
| 27 Jun 2016 | LSP announced by Microsoft, Red Hat and Codenvy |
| 28 Jun 2016 | Issue #19 — clarify the units of character |
| 16 Dec 2017 | rust-lang/rls #630 — an implementer hits it independently |
| 13 Jan 2018 | Issue #376 — proposal to switch to code points |
| 29 Mar 2019 | PR #709 — the code-point change, later closed unmerged |
| 30 Nov 2019 | Issue #872 — the client survey |
| 10 May 2022 | LSP 3.17 ships PositionEncodingKind; #376 closed the same day |
| 4 Jun 2026 | 3.18 changelog; UTF-16 still the default and still mandatory |
LSP 3.17 added a negotiated type, and 3.18 carries it forward unchanged. The client advertises general.positionEncodings, an ordered array; the server picks one and echoes it back in capabilities.positionEncoding. The three legal values are documented in the spec source as:
utf-8— “Character offsets count UTF-8 code units (i.e. bytes).”utf-16— “Character offsets count UTF-16 code units. This is the default and must always be supported by servers.”utf-32— “Character offsets count UTF-32 code units.”
Note what the fix did not do. It did not change the default, and it could not:
“To keep the protocol backwards compatible the following applies: if the value ‘utf-16’ is missing from the array of position encodings servers can assume that the client supports UTF-16. UTF-16 is therefore a mandatory encoding.”
And on the server side: “If the client didn’t provide any position encodings the only valid value that a server can return is ‘utf-16’. If omitted it defaults to ‘utf-16’.”
So the 2022 fix is strictly additive. A server that wants UTF-8 must still implement UTF-16, because a 2019-vintage client may connect tomorrow and say nothing at all. Negotiation did not remove the obligation; it added a branch to it. Every server now maintains the conversion machinery and a code path deciding which conversion to run, and every client must decide what to advertise. The 2016 behaviour is not deprecated. It is load-bearing, permanently, and the 3.18 draft does not touch it.
That is what fixing an encoding late actually costs. Fixing it in 2016 would have cost one sentence of spec text.
What June 2016 Could Have Done Instead
Three options were available, and one of them is much better than the other two.
Pick UTF-8 byte offsets. The JSON payload is already UTF-8 — the base protocol says the content part defaults to utf-8, “which is the only encoding supported right now.” Nearly every compiler front end works in byte offsets natively. This would have concentrated the conversion cost in exactly one implementation instead of spreading it across all the others. It would also have been the least comfortable choice for the team writing that one implementation, which is worth stating plainly rather than pretending the incentives were neutral.
Pick code points. Encoding-agnostic, conceptually clean, and where four of six surveyed clients ended up anyway. It still costs everybody a conversion, but it costs everybody the same conversion, and it has no surrogate-pair hazard.
Name the unit and negotiate it from version 1. This is the cheap one, and it is the actual lesson. LSP already had capability negotiation on day one — the mechanism that eventually carried the 3.17 fix existed the entire time. The failure was not choosing UTF-16 over UTF-8. The failure was shipping a field called character without saying what it counted, which let the reference implementation’s internals become the specification by default, and then become unchangeable by compatibility.
Generalise it: any protocol primitive that counts or identifies something needs its unit written into the type, not into the reference implementation. A timestamp without a timezone. A size field that might be bytes or records. An ID without a namespace. A path without a root. An offset without an encoding. Each one is a place where whoever ships first defines the semantics for everyone who ships after — and only the ones written down can be argued with later.
The uncomfortable corollary for anyone specifying a protocol next to a dominant reference implementation: your spec is not neutral just because your prose is. Wherever the prose is silent, the implementation is speaking. The only defence is to name the unit — every unit — before the second implementer arrives, because after that the answer stops being a decision and becomes a compatibility constraint.
❓ Frequently Asked Questions
Does UTF-16 position encoding still matter if my source files are all ASCII? Functionally, no — every encoding produces identical offsets for ASCII, which is exactly why the problem stayed invisible for years. Structurally, yes: your server still has to implement the conversion, because it cannot know in advance that a file has no non-ASCII content. gopls’s nonASCII flag is precisely that bet — a cheap runtime check that lets it skip the conversion once it knows the answer.
Can a modern language server just use UTF-8 and ignore the default? Only if the client advertises utf-8 in general.positionEncodings. If the client says nothing, the spec is unambiguous that the only valid value the server may return is utf-16. Supporting UTF-16 is not optional for a conformant server in 3.17 or in the 3.18 draft.
Why not use grapheme clusters, since that is what users actually see? Grapheme segmentation is defined in Unicode Annex #29 and its boundary rules change between Unicode versions, so two implementations built against different releases could legitimately disagree about where one cluster ends. A wire protocol needs an offset both sides compute identically forever. Code units and code points are stable in a way grapheme clusters deliberately are not — which is why clusters are the right unit for cursor movement inside a single editor and the wrong one for a message format.
Is this causing real bugs, or is it a theoretical complaint? Both, and the real ones are unglamorous: a diagnostic underline offset by one on a line containing an emoji, a rename that clips a character in a comment with a CJK ideograph, a code action inserting text in the middle of a surrogate pair. They are hard to reproduce because they need a non-BMP character at a specific column, and they get reported as “the squiggle is in the wrong place” rather than as an encoding fault.
Does the Debug Adapter Protocol inherit the same problem? DAP is a separate specification with its own column semantics, and its relationship to LSP is a matter of shared shape rather than shared types — a companion piece on those two protocols covers that comparison directly.
What is the single transferable rule here? Write the unit into the type. If a field counts something, its name or its documentation must say what a count of one means, before the second implementation exists. After that, the answer belongs to backwards compatibility rather than to you.
📦 Recommended Resources
Some links below are Amazon affiliate links. If you buy through them this site earns a small commission at no extra cost to you.
Start with the free primary sources, because on this specific subject they are better than any book: the LSP 3.17 specification (the Position and initialize sections are short and settle every question about what is actually mandated), Unicode Annex #29 on grapheme cluster boundaries, and the four GitHub issues cited above — #19, #376, #872 and rust-lang/rls #630 — which together are a better case study in protocol evolution than most textbook chapters. Read those first. The books below are for the layer underneath: the encoding model itself, and the general problem of designing a data format that has to survive its first implementation.
- Unicode Explained by Jukka K. Korpela (O’Reilly, 2006) — the book that makes the code unit / code point / grapheme cluster distinction stop being three words you can recite and start being three things you can reason about. Its treatment of UTF-8 versus UTF-16 versus UTF-32, and of what “a character” means at each level, is exactly the model this article assumes. Dated in its coverage of the standard’s later versions — Unicode has moved a long way since 2006 — but the conceptual apparatus has not changed and is still the clearest presentation of it.
- Unicode Demystified: A Practical Programmer’s Guide to the Encoding Standard by Richard Gillam (Addison-Wesley, 2003) — the implementer’s counterpart, written by someone who built this machinery. Where Korpela explains the model, Gillam works through what you actually have to store, index and convert, including the surrogate-pair mechanics that produce the mid-pair offset problem. Read it if you are the person who has to build the wide-encoding index rather than the person consuming it. Same caveat on age, same reason it survives: the encoding forms are stable by design.
- Designing Data-Intensive Applications by Martin Kleppmann (O’Reilly, 2017) — not a Unicode book, and the reason it is here is Chapter 4, “Encoding and Evolution.” It is the best available treatment of the actual argument in this article: what happens to a data format once independent parties depend on it, why forwards and backwards compatibility are separate properties, and why a schema decision made before the second implementation exists is qualitatively different from one made after. The LSP position-encoding story is a textbook instance of that chapter, and reading the chapter makes the story generalise.
✅ Pro tip: Before buying anything, open a REPL in whatever language you ship in and evaluate the length of a string containing one emoji. In JavaScript you get 2, in Python 1, in Go 4 with len() and 1 with utf8.RuneCountInString. That single experiment is the whole article, and it takes ten seconds — the interesting question afterwards is which of those numbers your public interfaces have been quietly exporting.