惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

L
Lohrmann on Cybersecurity
Martin Fowler
Martin Fowler
Engineering at Meta
Engineering at Meta
腾讯CDC
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
Microsoft Azure Blog
Microsoft Azure Blog
G
Google Developers Blog
TaoSecurity Blog
TaoSecurity Blog
博客园_首页
Vercel News
Vercel News
Hugging Face - Blog
Hugging Face - Blog
D
Darknet – Hacking Tools, Hacker News & Cyber Security
Last Week in AI
Last Week in AI
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
T
The Exploit Database - CXSecurity.com
量子位
Project Zero
Project Zero
A
Arctic Wolf
小众软件
小众软件
NISL@THU
NISL@THU
C
CERT Recently Published Vulnerability Notes
有赞技术团队
有赞技术团队
MongoDB | Blog
MongoDB | Blog
博客园 - 聂微东
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
N
News and Events Feed by Topic
宝玉的分享
宝玉的分享
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
T
Troy Hunt's Blog
P
Privacy & Cybersecurity Law Blog
Security Latest
Security Latest
B
Blog
酷 壳 – CoolShell
酷 壳 – CoolShell
D
DataBreaches.Net
Schneier on Security
Schneier on Security
The Hacker News
The Hacker News
K
Kaspersky official blog
C
Check Point Blog
Hacker News: Ask HN
Hacker News: Ask HN
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Webroot Blog
Webroot Blog
www.infosecurity-magazine.com
www.infosecurity-magazine.com
人人都是产品经理
人人都是产品经理
AI
AI
Cisco Talos Blog
Cisco Talos Blog
MyScale Blog
MyScale Blog
Cloudbric
Cloudbric
B
Blog RSS Feed
S
Schneier on Security
P
Palo Alto Networks Blog

Mathias Bynens

A horrifying globalThis polyfill in universal JavaScript JavaScript engine fundamentals: optimizing prototypes JavaScript engine fundamentals: Shapes and Inline Caches Asynchronous stack traces: why await beats Promise#then() ECMAScript regular expressions are getting better! Unicode property escapes in JavaScript regular expressions ES2015 const is not about immutability Valid JavaScript variable names in ES2015 Dear Google, please fix plain text emails in Gmail PBKDF2+HMAC hash collisions explained JavaScript has a Unicode problem Processing Content Security Policy violation reports Hiding JSON-formatted data in the DOM with CSP enabled Loading JSON-formatted data with Ajax and xhr.responseType='json' Reserved keywords in JavaScript How to support full Unicode in MySQL databases How to speedrun Dropbox’s Dropquest 2012 Unquoted font family names in CSS Unquoted property names / object keys in JavaScript Valid JavaScript variable names in ES5 CSS character escape sequences JavaScript’s internal character encoding: UCS-2 or UTF-16? The smallest possible valid (X)HTML documents JavaScript character escape sequences JavaScript foo.prototype.bar notation Ambiguous ampersands HTML element + attribute notation How I detect and use localStorage: a simple JavaScript pattern Unquoted attribute values in HTML and CSS/JS selectors The end-tag open (ETAGO) delimiter Using the oninput event handler with onkeyup/onkeydown as its fallback Everything you always wanted to know about touch icons In defense of CSS hacks — introducing “safe CSS hacks” AirPlay video support in iOS Safari — a bookmarklet Completing Dropbox’s Dropquest 2011 in 60 seconds Using CSS without HTML How to create simple Mac apps from shell scripts Using setTimeout to speed up window.onload Bulletproof JavaScript benchmarks Thoughts on Safari Reader’s generated HTML How to enable Safari Reader on your site? The XML serialization of HTML5, aka ‘XHTML5’ The id attribute got more classy in HTML5 The three levels of HTML5 usage The HTML5 document.head DOM tree accessor Bulletproof HTML5 <details> fallback using jQuery Displaying hidden elements like <head> using CSS Inline <script> and <style> vs. external .js and .css — what’s the size threshold? Using Showdown/PageDown with and without jQuery
Unicode-aware regular expressions in ES2015
Mathias · 2014-08-27 · via Mathias Bynens

ECMAScript 2015 introduces two new flags for regular expressions:

  1. y enables ‘sticky’ matching.
  2. u enables various Unicode-related features.

This article explains the effects of the u flag. It helps if you’ve read JavaScript has a Unicode problem first.

Impact on syntax

Setting the u flag on a regular expression enables the use of ES2015 Unicode code point escapes (\u{…}) in the pattern.

// Note: `a` is U+0061 LATIN SMALL LETTER A, a BMP symbol.
console.log(/\u{61}/u.test('a'));
// → true

// Note: `𝌆` is U+1D306 TETRAGRAM FOR CENTRE, an astral symbol.
console.log(/\u{1D306}/u.test('𝌆'));
// → true

Without the flag, things like \u{1234} can technically still occur in patterns, but they won’t be interpreted as Unicode code point escapes. /\u{1234}/ is equivalent to /u{1234}/, which matches 1234 consecutive u symbols rather than the symbol with code point U+1234.

Engines do this for compatibility reasons. But with the u flag set, this changes too: things like \a (where a is not an escape character) won’t be equivalent to a anymore. So even though /\a/ is treated as /a/, /\a/u throws an error, because \a is not a reserved escape sequence. This makes it possible to extend u regular expressions in a future version of ECMAScript. For example, /\p{Script=Greek}/u throws an exception per ES2015, but could become a regular expression that matches all symbols in the Greek script according to the Unicode database once syntax for Unicode property escapes is added to the spec.

Impact on the . operator

Without the u flag, . matches any BMP symbol except line terminators. When the ES2015 u flag is set, . matches astral symbols too.

// Note: `𝌆` is U+1D306 TETRAGRAM FOR CENTRE, an astral symbol.
const string = 'a𝌆b';

console.log(/a.b/.test(string));
// → false

console.log(/a.b/u.test(string));
// → true

const match = string.match(/a(.)b/u);
console.log(match[1]);
// → '𝌆'

Impact on quantifiers

The available quantifiers in JavaScript regular expressions are *, +, ?, and {2}, {2,}, {2,4}, and variations of those. Without the u flag, if a quantifier follows an atom that consists of an astral symbol, it applies only to the low surrogate of that symbol.

// Note: `a` is a BMP symbol.
console.log(/a{2}/.test('aa'));
// → true

// Note: `𝌆` is an astral symbol.
console.log(/𝌆{2}/.test('𝌆𝌆'));
// → false

// Explanation: the previous example is equivalent to the following.
console.log(/\uD834\uDF06{2}/.test('\uD834\uDF06\uD834\uDF06'));
// → false

With the ES2015 u flag, quantifiers apply to whole symbols, even for astral symbols.

// Note: `a` is a BMP symbol.
console.log(/a{2}/u.test('aa'));
// → true

// Note: `𝌆` is an astral symbol.
console.log(/𝌆{2}/u.test('𝌆𝌆'));
// → true

Impact on character classes

Without the u flag, any given character class can only match BMP symbols. Things like [bcd] work as expected:

const regex = /^[bcd]$/;
console.log(
	regex.test('a'), // false
	regex.test('b'), // true
	regex.test('c'), // true
	regex.test('d'), // true
	regex.test('e')  // false
);

However, when an astral symbol is used in a character class, the JavaScript engine treats it as two separate ‘characters’: one for each of its surrogate halves.

// Note: `𝌆` is an astral symbol.
const regex = /^[bc𝌆]$/;
console.log(
	regex.test('a'), // false
	regex.test('b'), // true
	regex.test('c'), // true
	regex.test('𝌆')  // false
);

// Explanation: the regular expression is equivalent to the following.
// const regex = /^[bc\uD834\uDF06]$/;

The ES2015 u flag enables the use of whole astral symbols in character classes.

// Note: `𝌆` is an astral symbol.
const regex = /^[bc𝌆]$/u; // Or, `/^[bc\u{1D306}]$/u`.
console.log(
	regex.test('a'), // false
	regex.test('b'), // true
	regex.test('c'), // true
	regex.test('𝌆')  // true
);

Consequently, whole astral symbols can also be used in character class ranges, and everything will work as expected as long as the u flag is set.

// Match any symbol from U+1F4A9 PILE OF POO to U+1F4AB DIZZY SYMBOL.
const regex = /[💩-💫]/u; // Or, `/[\u{1F4A9}-\u{1F4AB}]/u`.
console.log(
	regex.test('💨'), // false
	regex.test('💩'), // true
	regex.test('💪'), // true
	regex.test('💫'), // true
	regex.test('💬')  // false
);

The u flag also affects negated character classes. For example, /[^a]/ is equivalent to /[\0-\x60\x62-\uFFFF]/, which would match any BMP symbol except a. But with the u flag, /[^a]/u matches the much bigger set of all Unicode symbols except a.

const regex = /^[^a]$/u;
console.log(
	regex.test('a'), // false
	regex.test('b'), // true
	regex.test('☃'), // true
	regex.test('𝌆'), // true
	regex.test('💩')  // true
);

Impact on character class escapes

The u flag affects the meaning of the character class escapes \D, \S, and \W. Without the u flag, \D, \S, and \W match any BMP symbols that are not matched by \d, \s, and \w, respectively.

const regex = /^\S$/;
console.log(
	regex.test(' '), // false
	regex.test('a'), // true
	// Note: `𝌆` is an astral symbol.
	regex.test('𝌆')  // false
);

With the u flag, \D, \S, and \W match astral symbols too.

const regex = /^\S$/u;
console.log(
	regex.test(' '), // false
	regex.test('a'), // true
	// Note: `𝌆` is an astral symbol.
	regex.test('𝌆')  // true
);

Their inverse counterparts \d, \s, and \w are not affected by the u flag. There was a proposal to make \d and \w (and \b) more Unicode-aware, but it was rejected.

Impact on the i flag

When both the i and u flag are set, all symbols are implicitly case-folded using the simple mapping provided by the Unicode standard immediately before they are compared.

const es5regex = /[a-z]/i;
const es6regex = /[a-z]/iu;
console.log(
	es5regex.test('s'),      es6regex.test('s'),      // true true
	es5regex.test('S'),      es6regex.test('S'),      // true true
	// Note: U+017F canonicalizes to `S`.
	es5regex.test('\u017F'), es6regex.test('\u017F'), // false true
	// Note: U+212A canonicalizes to `K`.
	es5regex.test('\u212A'), es6regex.test('\u212A')  // false true
);

The case folding applies to the symbols in the regular expression pattern as well as the symbols in the string to be matched.

console.log(
	/\u212A/iu.test('K'), // true
	/\u212A/iu.test('k'), // true
	/\u017F/iu.test('S'), // true
	/\u017F/iu.test('s')  // true
);

This case-folding logic applies to the \w and \W character escapes as well, which also affects \b and \B. /\w/iu matches [0-9A-Z_a-z] but also U+017F because U+017F canonicalizes to S which is in the match set. The same goes for U+212A and K.

console.log(
	/\w/iu.test('\u017F'), // true
	/\w/iu.test('\u212A'), // true
	/\W/iu.test('\u017F'), // false
	/\W/iu.test('\u212A'), // false
	/\W/iu.test('s'),      // false
	/\W/iu.test('S'),      // false
	/\W/iu.test('K'),      // false
	/\W/iu.test('k'),      // false
	/\b/iu.test('\u017F'), // true
	/\b/iu.test('\u212A'), // true
	/\b/iu.test('s'),      // true
	/\b/iu.test('S'),      // true
	/\B/iu.test('\u017F'), // false
	/\B/iu.test('\u212A'), // false
	/\B/iu.test('s'),      // false
	/\B/iu.test('S'),      // false
	/\B/iu.test('K'),      // false
	/\B/iu.test('k')       // false
);

Note: An annoying result of this case-folding logic is that, per the original ES2015 spec, /w/iu was no longer the inverse of /\W/iu. Remember how /\w/iu matches [0-9A-Z_a-z] but also U+017F and U+212A? This makes sense. However, in ES2015, /\W/iu also matched U+017F, and strangely, S, because \W includes U+017F which matches either the U+017F symbol itself or its canonicalized version S. The same applied for U+212A and K. In other words, /\W/iu was equivalent to /[^0-9a-jl-rt-zA-JL-RT-Z_]/u. 😕 This was rectified in June 2016. Now, /\W/iu doesn’t match S, K, U+017F, or U+212A anymore, making /\W/iu the inverse of /w/iu again. /\W/iu is now equivalent to /[^0-9a-zA-Z_\u{017F}\u{212A}]/u. Whew.

Impact on HTML documents

Believe it or not, the existence of the u flag has its effect on HTML documents as well.

The pattern attribute for input and textarea elements allows you to specify a regular expression to validate the user’s input against. The browser then provides you with styling and scripting hooks to make stuff happen based on the input’s validity.

<style>
	:invalid { background: red; }
	:valid { background: green; }
</style>
<input pattern="a.b" value="aXXb"><!-- gets a red background -->
<input pattern="a.b" value="a𝌆b"><!-- gets a green background -->

The u flag is always enabled for regular expressions compiled through the HTML pattern attribute. Here’s a demo / test case.

Support

At the moment, the ES2015 u flag for regular expressions is available in stable releases of every major browser. Browsers are slowly starting to enable this functionality for the HTML pattern attribute.

Browser(s) JavaScript engine u flag u flag for pattern attribute
Edge Chakra ✅ issue #1102227 + ✅ issue #517 + ✅ issue #1181 + ✅ issue #4368 ❌ issue #7113940
Firefox Spidermonkey ✅ bug #1135377 + ✅ bug #1281739 ✅ bug #1227906
Chrome/Opera V8 ✅ V8 issue #2952 + ✅ issue #5080 ✅ issue #535441
WebKit JavaScriptCore ✅ bug #154842 + ✅ bug #151597 + ✅ bug #158505 ✅ bug #151598

Recommendations for developers

  • Use the u flag for every regular expression you write from now on.
  • …but don’t blindly add the u flag to existing regular expressions, as it might change their meaning in subtle ways.
  • Avoid combining the u and i flags. It’s better to be explicit and include all letter cases in your regular expression itself than to be surprised by implicit case folding.
  • Use a transpiler to make sure your code runs everywhere, including legacy environments.

Transpiling ES6 Unicode regular expressions to ES5

I created regexpu, a transpiler that rewrites ES6 Unicode regular expressions into equivalent ES5 code that works today. This enables you to play around with these upcoming new features. Try it out now!

The regexpu demo page transpiles ES6 Unicode regular expressions as you type.

Full-blown ES6/ES7 transpilers like Traceur and Babel depend on regexpu for their u transpilation. Let me know if you manage to break it.