For context, please see Character Encoding Menu in 2014, Text Encoding Menu in 2021, and A Look at Encoding Detection and Encoding Menu Telemetry from Firefox 86.

Firefox 91 was released two weeks ago. This is the first release that does not have a Text Encoding submenu. Instead, the submenu has been replaced with a single menu item called to Repair Text Encoding. It performs the action that was previously performed by the item Automatic in the Text Encoding submenu: It runs chardetng with UTF-8 as a permitted outcome and ignoring the top-level domain.

The Repair Text Encoding menu item is in the View menu, which is hidden by default on Windows and Linux. The action is also available as an optional toolbar button (invoke the context menu on empty space in the toolbar and choose Customize Toolbar…). On Windows and Linux, you can invoke the menu item from the keyboard by pressing the v key while holding the alt key and then pressing the c key. (The keys may vary with the localization.)

What Problem Does “Repair Text Encoding” Solve?

1. Sometimes the declared encoding is wrong, and the Web Platform would become more brittle if we started second-guessing the declared encoding automatically without user action.

   The typical case is that university faculty have created content over the years that is still worthwhile to read, and the old content is in a legacy encoding. However, independently of the faculty, the administrator has either explicitly or as a side effect of server software update caused the server configuration to claim UTF-8 server-wide even though this is wrong for old content. When the context is in the Latin script, the result is still readable. When the content is in a non-Latin script, the result is completely unreadable (without this feature).

2. For non-Latin scripts, unlabeled UTF-8 is completely unreadable. Fixing this problem without requiring user action and also without making the Web Platform more brittle is a hard problem. There is a separate write-up on that topic alone. This problem might get solved one day in a way that does not involve user action but not today.

Why Remove the Other Submenu Items?

• Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run. Previously, the parser needed to keep track of a specific manually-specified encoding alongside the encoding information for the Web sources.

  Indeed, when implementing support for declaring the encoding via the bogo XML declaration, the above-mentioned complexity got in the way, and I wish I had replaced the menu with a single item before implementing the bogo XML declaration support. Now, I wanted to get rid of the complexity before aligning meta charset handling with WebKit and Blink.

• Elaborate UI surface for a niche feature risks the whole feature getting removed, which is bad if the feature is still relevant to (a minority of) users. (Case in point: The Proton UI refresh removed the Text Encoding menu entry point from the hamburger menu.)

• Telemetry showed users making a selection from the menu when the encoding of the page being overridden had come from a previous selection from the menu. This suggested that users aren’t that good at choosing correctly manually.

Why Not Remove the Whole Thing?

Chrome removed their menu altogether as part of what they called Project Eraser. (Amusingly, this lead to a different department of Google publishing a support article about using other browsers to access this functionality.) Mobile versions of Chrome, Safari, and Firefox don’t have the menu, either. So why not just follow Chrome?

Every time something in this area breaks intentionally or accidentally, feedback from Japan shows up relatively quickly. That’s the main reason why I believe users in Japan still care about having the ability to override the encoding of misconfigured pages. (That’s without articulating any particular numeric telemetry threshold for keeping the feature. However, telemetry confirms that the feature is relevant to the largest number of distinct telemetry submitters, both in absolute numbers and in region-total-relative numbers, in Japan.)

If we removed the feature, we’d remove a reason for these users to stay with Firefox. Safari and Gnome Web still have more elaborate encoding override UI built in (the list of encodings in both is questionably curated but the lists satisfy the Japanese use cases), and there are extensions for Chrome.

Shouldn’t This Be an Extension?

The built-in UI action in Firefox is more discoverable, more usable, and safer against the user getting baited into self-XSS than the Chrome extensions. Retaining the safety properties but moving the UI to an extension would increase implementation complexity while reducing discoverability—i.e. would help fewer users at a higher cost.

Removing the engine feature and leaving to an extension to rewrite headers of the HTTP responses (as in Chrome) would:

• Give Chrome an advantage on day one by the extension(s) for Chrome actually already existing.
• Fail to help the users who don’t discover an extension.
• Regress usability by about a decade due to the extension UI being unaware of what’s going on inside the engine.
• Remove self-XSS protections.