I have a weird setup. I type in Dvorak. But, when I hold ctrl or alt, my keyboard reverts to Qwerty.

You see, the classic text-editing hotkeys, ctrl+Z, ctrl+X, ctrl+C, and ctrl+V are all located optimally for a Qwerty layout: next to the control key, easy to reach with your left hand while mousing with your right. In Dvorak, unfortunately, these hotkeys are scattered around mostly on the right half of the keyboard, making them much less convenient. Using Dvorak for typing but Qwerty for hotkeys turns out to be a nice compromise.

But, the only way I could find to make this work on Linux / X was to write a program that uses X "grabs" to intercept key events and rewrite them. That was mostly fine, until recently, when my machine, unannounced, updated to Wayland. Remarkably, I didn't even notice at first! But at some point, I realized my hotkeys weren't working right. You see, Wayland, unlike X, actually has some sensible security rules, and as a result, random programs can't just capture all keyboard events anymore. Which broke my setup.

Yes, that's right, I'm that guy:

Source: xkcd 1172

So what was I to do? I began worrying that I'd need to modify the keyboard handling directly in Wayland or in the Linux kernel. Maintaining my own fork of core system infrastructure that changes frequently was not an attractive thought.

Desperate, I asked the Cloudflare Engineering chat channel if anyone knew a better way. That's when Marek Kroemeke came to the rescue:

Following Marek's link, I found:

#! /usr/bin/env stap

# This is not useful, but it demonstrates that
# Systemtap can modify variables in a running kernel.

# Usage: ./keyhack.stp -g

probe kernel.function("kbd_event") {
  # Changes 'm' to 'b' .
  if ($event_code == 50) $event_code = 48
}

probe end {
  printf("\nDONE\n")
}

Oh my. What is this? What do you mean, "this is not useful"? This is almost exactly what I want!

SystemTap: Not just for debugging?

SystemTap is a tool designed to allow you to probe the Linux kernel for debugging purposes. It lets you hook any kernel function (yes, any C function defined anywhere in the kernel) and log the argument values, or other system state. Scripts are written in a special language designed to prevent you from doing anything that could break your system.

But it turns out you can do more than just read: With the -g flag (for "guru mode", in which you accept responsibility for your actions), you can not just read, but modify. Moreover, you can inject raw C code, escaping the restrictions of SystemTap's normal language.

SystemTap's command-line tool, stap, compiles your script into a Linux kernel module and loads it. The module, on load, will find the function you want to probe and will overwrite it with a jump to your probing code. The probe code does what you specify, then jumps back to the original function body to continue as usual. When you terminate stap (e.g. via ctrl+C on the command line), it unloads the module, restoring the probed function to its original state.

This means it's easy and relatively safe to inject a probe into your running system at any time. If it doesn't do what you want, you can safely remove it, modify it, and try again. There's no need to modify the actual kernel code nor recompile your kernel. You can make your changes without maintaining a fork.

This is, of course, a well-known practice in dynamic programming languages, where it's generally much easier. We call it "Monkey-Patching".

When is it OK to Monkey-Patch?

"Monkey-patch" is often used as a pejorative. Many developers cringe at the thought. It's an awful hack! Never do that!

Indeed, in a lot of contexts, monkey-patching is a terrible idea. At a previous job, I spent weeks debugging problems caused by a bad (but well-meaning) monkey-patch made by one of our dependencies.

But, often, a little monkey-patch can save a lot of work. By monkey-patching my kernel, I can get the keyboard behavior I want without maintaining a fork forever, and without spending weeks developing a feature worthy of pushing upstream. And when patching my own machine, I can't hurt anyone but myself.

I would propose two rules for monkey patching:

1. Only the exclusive owner of the environment may monkey-patch it. The "owner" is an entity who has complete discretion and control over all code that exists within the environment in which the monkey-patch is visible. For a self-contained application which specifies all its dependencies precisely, the application developer may be permitted to monkey-patch libraries within the application's runtime -- but libraries and frameworks must never apply monkey-patches. When we're talking about the kernel, the "owner" is the system administrator.

2. The owner takes full responsibility for any breakages caused. If something doesn't work right, it's up to the owner to deal with it or abandon their patch.

In this case, I'm the owner of my system, and therefore I have the right to monkey-patch it. If my monkey-patch breaks (say, because the kernel functions I was patching changed in a later kernel version), or if it breaks other programs I use, that's my problem and I'll deal with it.

Setting Up

To use SystemTap, you must have the kernel headers and debug symbols installed. I found the documentation was not quite right on my Debian system. I managed to get everything installed by running:

sudo apt install systemtap linux-headers-amd64 linux-image-amd64-dbg

Note that the debug symbols are a HUGE package (~500MB). Such is the price you pay, it seems.

False Starts

Starting from the sample script that remaps 'm' to 'b', it seemed obvious how to proceed. I saved the script to a file and did:

sudo stap -g keyhack.stp

But… nothing happened. My 'm' key still typed 'm'.

To debug, I added some printf() statements (which conveniently print to the terminal where stap runs). But, it appeared the keyboard events were indeed being captured. So why did 'm' still type 'm'?

It turns out, no one was listening. The kbd_event function is part of the text-mode terminal support. Sure enough, if I switched virtual terminals over to a text terminal, the key was being remapped. But Wayland uses a totally different code path to receive key events -- the /dev/input devices. These devices are implemented by the evdev module.

Looking through evdev.c, at first evdev_event() looks tempting as a probe point: it has almost the same signature as kbd_event(). Unfortunately, this function is not usually called by the driver; rather, the multi-event version, evdev_events(), usually is. But that version takes an array, which seems more tedious to deal with.

Looking further, I came across __pass_event(), which evdev_events() calls for each event. It's slightly different from kbd_event() in that the event is encapsulated in a struct, but at least it only takes one event at a time. This seemed like the easiest place to probe, so I tried it:

# DOES NOT WORK
probe module("evdev").function("__pass_event") {
  # Changes 'm' to 'b'.
  if ($event->code == 50) $event->code = 48
}

Alas, this didn't quite work. When running stap, I got:

semantic error: failed to retrieve location attribute for 'event'

This error seems strange. The function definitely has a parameter called event!

The problem is, __pass_event() is a static function that is called from only one place. As a result, the compiler inlines it. When a function is inlined, its parameters often cease to have a well-defined location in memory, so reading and modifying them becomes infeasible. SystemTap relies on debug info tables that specify where to find parameters, but in this case the tables don't have an answer.

The Working Version

Alas, it seemed I'd need to use evdev_events() and deal with the array after all. This function takes an array of events to deliver at once, so its parameters aren't quite as convenient. But, it has multiple call sites, so it isn't inlined. I just needed a little loop:

probe module("evdev").function("evdev_events") {
  for (i = 0; i < $count; i++) {
    # Changes 'm' to 'b'.
    if ($vals[i]->code == 50) $vals[i]->code = 48
  }
}

Success! This script works. I no longer have any way to type 'm'.

From here, implementing the Dvorak-Qwerty key-remapping behavior I wanted was a simple matter of writing some code to track modifier key state and remap keys. You can find my full script on GitHub.