terminals really want text to be a 2d grid of characters which can be rendered by just mapping each character onto a box on the screen. now, if your language uses a latin script, this can probably be made to work good enough. unfortunately, it turns out that language is complex, orthography is complex, and if you try to generalize this approach to text rendering to handle unicode properly, you're gonna end up with Problems
with that in mind, let's try to generalize this approach to text rendering to handle unicode properly
wait hang on you said a 2d grid of "characters" what even is a character
great question! this one's actually relatively straightforward: a character is 7 bits encoded as a byte in ascii, with the high bit's meaning depending on the way the terminal is configured
... by which i mean that a character is actually a unicode codepoint, encoded in utf8, which can take up a variable number of bytes in memory but it's fine because we can still think of it as a fixed-width integer
... by which i mean that it's actually an extended grapheme cluster (egc), which is a variable-length sequence of codepoints, but it's fine because we can still reliably define it without any machine-specific context, purely using universally-accepted data and algorithms
... by which i mean that it's actually whatever the font says it is. oops. though do bear in mind that if you're writing an application, you do still want to do most of your unicode handling (such as eg. arrow key movement) based on extended grapheme clusters
terminals have gotten better at this, and at the very least understanding utf8 rather than ascii is table stakes for modern terminal emulators, but support for ligatures is still very much hit-or-miss and egc support is Somewhat Dubious
yep! text is really quite simple like that
... wait oops sorry i got mixed up it's the opposite of the thing i said
so afaik there's not a whole lot of effort that's been put into standardizing this sort of thing. there're some areas we'll get to later where terminals tend to do the Wrong Thing, but at least that Thing has a pretty consistent consensus between everyone on what the standard behavior is. this is not one of those cases
unicode defines a property called "east asian width", which is mostly just for marking cjk characters that're square rather than rectangular, and which doesn't handle emojis correctly. the historically-standard way of doing this has been to calculate text width on a per-codepoint level, which is definitely incorrect. foot does this but limits it to 2, which is more correct (though i'm not sure if it's Actually correct)
as a bonus, any changes in this algorithm which aren't reciprocated on both sides of the terminal protocol can and will lead to cursor desyncs. the only way that i know of to build a unicode-robust program which doesn't rely solely on the terminal's built-in text placement involves the Worst hacks
ok sure, so "character" is a messed-up concept. but surely the "2d grid" part is fine, right?
nope! it's completely impossible for terminals to use non-monospace fonts because of this. there are some benefits to a monospace font in some contexts, but forcing everyone to use them all the time is... not optimal
as a bonus, modern text rendering has given up entirely on the assumption that text can be coerced into a 2d grid of any sort, which means that when we try to fit text back into that box, we stop being able to rely on any widely-used algorithms for text width in this context. joy.
alright, so characters are messed up, and 2d grids are messed up, but surely so long as the characters do happen to be the same width everything will work out?
nope! terminals completely break with rtl text, and while this is a bit less inherent to the fundamental building blocks of how terminals work, in practice fixing it would require rewriting the text-rendering parts of every single tui program in existence, on top of quite a bit of work to fix cli programs without forcing them to start thinking about text rendering in too much detail
ugh, that sucks. how can we fix it?
honestly i don't think we can, and even if we can, i'm not sure if it's worth it. terminals suck for a bunch of other reasons in addition to their poor unicode handling, and i'd much rather see a wholesale replacement for them which tries to keep the good parts without being shackled to backwards-compatibility with hardware from the '50s
and there are good parts of terminals, to be clear. small cli programs are really easy to write without having to think about the interface, even if that default interface isn't ideal, and i'd really like to see something which can keep that programmer-ease-of-use for simple cases
i also rather like plan9's approach to this: windows can have both text and pixels written to them, and text mode has none of the complexity that's required to be able to implement fullscreen programs in it. things which look and feel like tui programs are instead just drawing directly to the same window that until recently had a shell running in it, and because of this they aren't limited to a monospace grid of unicode characters. i would, however, like to see the text mode expanded a little bit from what plan9 has in order to better accommodate keyboard-only usage