惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

T
Threatpost
博客园 - 叶小钗
T
The Blog of Author Tim Ferriss
Recent Announcements
Recent Announcements
D
DataBreaches.Net
The Cloudflare Blog
阮一峰的网络日志
阮一峰的网络日志
罗磊的独立博客
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
N
Netflix TechBlog - Medium
Microsoft Azure Blog
Microsoft Azure Blog
Microsoft Security Blog
Microsoft Security Blog
B
Blog
U
Unit 42
有赞技术团队
有赞技术团队
博客园 - 聂微东
GbyAI
GbyAI
宝玉的分享
宝玉的分享
F
Full Disclosure
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
MyScale Blog
MyScale Blog
Jina AI
Jina AI
Martin Fowler
Martin Fowler
IT之家
IT之家
酷 壳 – CoolShell
酷 壳 – CoolShell
D
Docker
P
Proofpoint News Feed
A
About on SuperTechFans
I
InfoQ
博客园 - 【当耐特】
C
Check Point Blog
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
P
Privacy & Cybersecurity Law Blog
T
Threat Research - Cisco Blogs
Y
Y Combinator Blog
Project Zero
Project Zero
WordPress大学
WordPress大学
小众软件
小众软件
AWS News Blog
AWS News Blog
博客园 - 司徒正美
T
The Exploit Database - CXSecurity.com
L
LINUX DO - 热门话题
I
Intezer
Engineering at Meta
Engineering at Meta
C
CXSECURITY Database RSS Feed - CXSecurity.com
J
Java Code Geeks
T
Tenable Blog
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
Last Week in AI
Last Week in AI
C
CERT Recently Published Vulnerability Notes

Henri Sivonen’s pages

Parin vuoden tutkimattomuus crates.io: Rust Package Registry Asiakirjatonta toimintaa It’s not wrong that "🤦🏼‍♂️".length == 7 Koulutartuntojen tilastointimenettely Perusteasiakirjoja hallussapitämättä ikärajoitettu Asiantuntijat ja nukkuva vallan vahtikoira Koronapassilausunto Suppealla tietopohjalla ohimeneväksi väitetty Text Encoding Menu in 2021 The Text Encoding Submenu Is Gone An HTML5 Conformance Checker Not Part of the Technology Stack Browser Technology Stack Bogo-XML Declaration Returns to Gecko A Look at Encoding Detection and Encoding Menu Telemetry from Firefox 86 Why Supporting Unlabeled UTF-8 in HTML on the Web Would Be Problematic Rust Target Names Aren’t Passed to LLVM Toimintamalli Activating Browser Modes with Doctype Johtopäätöksiä mallin rakenteesta Tehtävänmäärittelyä kirjoittamatta ja kuolemia laskematta laumasuojamallinnettu Character Encoding Menu in 2014 Erillissuosituksen tarpeettomuudesta yleissuosituksen poikkeukseksi? STM:n maskiaikajana Rust 2021 Oma-aloitteisesti mallinnettu Kokopinovaatimuksin kilpailutettu chardetng: A More Compact Character Encoding Detector for the Legacy Web Varauksia paisutellen tiedotettu Perusteasiakirjoitta tiedotettu IME Smoke Testing The Validator.nu HTML Parser About the Hiragino Fonts with CSS It’s Time to Stop Adding New Features for Non-Unicode Execution Encodings in C++ Rust 2020 The Last of the Parsing Quirks About about:blank Rust 2019 a Web-Compatible Character Encoding Library in Rust How I Wrote a Modern C++ Library in Rust Using cargo-fuzz to Transfer Code Review of Simple Safe Code to Complex Code that Uses unsafe A Rust Crate that Also Quacks Like a Modern C++ Library #Rust2018 No Namespaces in JSON, Please A Lecture about HTML5 Julkisesti luotettu varmenne ikidomainille TLS:ää (SSL:ää) varten -webkit-HTML5 Lists in Attribute Values The Sad Story of PNG Gamma “Correction” If You Want Software Freedom on Phones, You Should Work on Firefox OS, Custom Hardware and Web App Self-Hostablility HTML5 Parser Improvements ARIA in HTML5 Integration: Document Conformance (Draft, Take Two) Schema.org and Pre-Existing Communities Lowering memory requirements by replacing Schematron HTML5 Parsing in Gecko: A Build Introducing SAX Tree NVDL Support in Validator.nu HOWTO Avoid Being Called a Bozo When Producing XML An Unofficial Q&A about the Discontinuation of the XHTML2 WG Thoughts on HTML5 Becoming a W3C Recommendation Four Finnish Banks Training Users to Give Banking Credentials to Another Site Unimpressed by Leopard Sergeant Semantics The Content Sink Inheritance Diagram – 2006-06-30 What is EME? About Points and Pixels as Units The Performance Cost of the HTML Tree Builder Social Media Impression Management The spacer Element Is Gone Openmind 2006 Performance Mistake XHTML and Mobile Devices WebM-Enabled Browser Usage Share Exceeds H.264-Enabled Browser Usage Share on Desktop (in StatCounter Numbers) HTML5 Parser-Based View Source Syntax Highlighting Vendor Prefixes Are Hurting the Web Accept-Charset Is No More Dualroids Writing Structural Stylable Document in Mozilla Editor ISO-8859-15 on haitallinen Hourglass The Scientific Method According to Hixie Maemo Source Code Karpelan lukkovertaus ontuu Digitaalisesta arkistoinnista ARIA in HTML5 Integration: Document Conformance (Draft) XHTML—What’s the Point? (Draft, incomplete) Mac OS X Browser Comparison HOWTO Spot a Wannabe Web Standards Advocate An Idea About Intermediate Language Trees and Web UI Generation Thoughts on Using SSL/TLS Certificates as the Solution to Phishing Bureaucracy Meets the Web Europe Day HOWTO Establish a 100% Literacy Rate What to Do with All These Photos? Charmod Norm Checking Validator Web Service Interface Ideas DTDs Don’t Work on the Web EFFI’s Day in Court Speaking at XTech
Always Use UTF-8 & Always Label Your HTML Saying So
Henri Sivonen · 2020-03-03 · via Henri Sivonen’s pages

To avoid having to deal with escapes (other than for <, >, &, and "), to avoid data loss in form submission, to avoid XSS when serving user-provided content, and to comply with the HTML Standard, always encode your HTML as UTF-8. Furthermore, in order to let browsers know that the document is UTF-8-encoded, always label it as such. To label your document, you need to do at least one of the following:

  • Put <meta charset="utf-8"> as the first thing after the <head> start tag (i.e. as the first child of head).

    The meta tag, including its ending > character needs to be within the first 1024 bytes of the file. Putting it right after the <head> start tag is the easiest way to get this right. Do not put comments before the <head> start tag and <meta charset="utf-8"> to avoid accidentally pushing the latter past the first 1024 bytes.

  • Configure your server to send the header Content-Type: text/html; charset=utf-8 on the HTTP layer.

  • Start the document with the UTF-8 BOM, i.e. the bytes 0xEF, 0xBB, and 0xBF.

Doing more than one of these is OK.

Answers to Questions

The above says the important bit. Here are answers to further questions:

Why Do I Need to Label UTF-8 in HTML?

Because HTML didn’t support UTF-8 in the very beginning and legacy content can’t be expected to opt out, you need to opt into UTF-8 just like you need to opt into the standards mode (via <!DOCTYPE html>) and to mobile-friedly layout (via <meta name="viewport" content="width=device-width, initial-scale=1">). (Longer answer)

Which Method Should I Choose?

<meta charset="utf-8"> has the benefit of keeping the label within your document even if you move it around. The main risk is that someone forgets that it needs to be within the first 1024 bytes and puts comments, Facebook metadata, rel=preloads, stylesheets or scripts before it. Always put that other stuff after it.

The HTTP header has the benefit that if you are setting up a new server that doesn’t have any old non-UTF-8 documents on it, you can configure the header once, and it works for all HTML documents on the server thereafter.

The BOM method has the problem that it’s too easy to edit the file in a text editor that removes the BOM and not notice that this has happened. However, if you are writing a serializer library and you are neither in control of the HTTP header nor can inject a tag without interfering with what your users are doing, you can make the serializer always start with the UTF-8 BOM and know that things will be OK.

Can I Use UTF-16 Instead?

Don’t. If you serve user-provided content as UTF-16, it is possible to smuggle content that becomes executable when interpreted as other encodings. This is a cross-site scripting vulnerability if the user uses a browser that allows the user to manually override UTF-16 with another encoding.

UTF-16 cannot be labeled via <meta charset>.

What about Plain Text?

The <meta charset="utf-8"> method is not available for plain text, but the other two are. In the case of plain text, the HTTP header is obviously Content-Type: text/plain; charset=utf-8 instead.

Why Does Unlabeled UTF-8 Plain Text or HTML Work in XHR/Fetch?

XMLHttpRequest and Fetch post-date UTF-8, so they had a chance to introduce new rules. While the rules being inconsistent with navigating to HTML or plain text is not great, defaulting to UTF-8 is a simple rule that avoids issues related to reloading content in ways that would be consistent with navigation.

What about JavaScript?

If you’ve labeled your HTML as UTF-8, you don’t need to label your UTF-8-encoded JavaScript files, since by default they inherit the encoding from the document that includes them. However, to make your JavaScript robust when referenced form non-UTF-8 HTML you can use the UTF-8 BOM or the HTTP header, which is Content-Type: application/javascript; charset=utf-8 in the JavaScript case.

What about CSS?

If you’ve labeled your HTML as UTF-8, you don’t need to label your UTF-8-encoded CSS files, since by default they inherit the encoding from the document that includes them. However, to make your CSS robust when referenced form non-UTF-8 HTML you can use the UTF-8 BOM or the HTTP header, which is Content-Type: text/css; charset=utf-8 in the CSS case, or you can put @charset "utf-8"; as the very first thing in the CSS file.

What about XML (Including SVG)?

Unlabeled XML defaults to UTF-8, so you don’t need to label it.

What about JSON?

JSON must be UTF-8 and is processed as UTF-8, so there’s no labeling.

What about WebVTT?

WebVTT is always UTF-8, so there’s no labeling.