This repository contains the source code for BareMetal-Firecracker. This is a custom version of the BareMetal kernel explicitly for execution within a Firecracker microVM.
The goal of this project was to achieve a <1ms cold start for the BareMetal kernel and its payload. That goal was achieved.
The purpose of this project is to allow for near "instant on" hardware isolated microVMs to be provisioned with as little as 2MiB of RAM assigned to each.
On an AMD Ryzen AI Max+ 395, running Ubuntu Desktop 26.04 with Firecracker v1.15.1, execution times are as follows:
- Init: ~100µs from Firecracker handoff to kernel start.
- BareMetal: ~700µs with network and disk enabled. ~500µs with only network enabled.
This continues the work to support the "Hypervisor as the OS" philosphy as written here.
Components
- BareMetal, an exokernel written in x86-64 Assembly.
- Firecracker, a streamlined virtualization environment.
Contents
src: Source code for BareMetal init and the BareMetal kernelpayload: Payload for the kernel - Currently a minimal version of BareMetal Monitorscripts: Scripts for creating/removing bridge and tap networksimg: Screenshot
Getting Started
Prerequisites
nasm, curl or wget, binutils, screen, firecracker (see FIRECRACKER.md)
A tap0 network device is expected for microVM network connectivity. mkbr0.sh in the scripts directory will configure the bridge and tap.
Building
./build.sh
Running this will build the kernel and the monitor utility. The monitor will run on startup.
./build.sh payload.app
Running this will configure "unikernel" mode. The app specified in the argument will run on startup. A couple examples are included:
webserver.app- lwIP running as a DHCP client and web serverhello.app- "Hello World" in Assembly
Running
./baremetal.sh is the main script. It expects one of several arguments:
start- Start the BareMetal VMstatus- Check if the VM is currently runningsend <text>- Send a line of text to the VM serial consoleoutput [--full]- Print new VM serial console output (--full for entire log)attach- Attach to the interactive screen sessionstop- Gracefully shut down the VM (Ctrl+Alt+Del)
Firecracker
Firecracker provides hardware isolated microVMs.
Overview
What is missing from a "standard" VM:
- No firmware (BIOS/UEFI)
- No PCI/PCIe bus
- No VGA or LFB
- No USB
- No HPET
- Minimal ACPI
What you get:
- VirtIO devices (block, net, and others) addressable via MMIO
- PS/2 keyboard controller (only used for sending Ctrl-Alt-Del)
- Serial console
Note: It is possible to enable a PCIe bus for Firecracker but it is not a default.
Memory usage
Firecracker uses the following memory address on startup:
| Start Address | Description |
|---|---|
| 0x000500 | GDT |
| 0x000520 | IDT |
| 0x006000 | PVH |
| 0x007000 | boot_params |
| 0x008000 | Stack (starts at 0x8FF0) |
| 0x009000 | PML4 (CR3 points here) |
| 0x00A000 | PDPTE |
| 0x00B000 | PDE |
| 0x020000 | cmd_line |
| 0x0E0000 | RSDP |
| 0x100000 | your software |
0xC000-0xFFFF should be free
Startup
Execution starts in 64-bit mode at 0x100000. RFLAGS is set to 0x2, RSP/RBP to 0x8FF0, and RSI to the address of boot_params.
BareMetal Init
Init preps the system for the BareMetal Kernel. It sets the system up in a similar way to Pure64. It is also written in Assembly.
Memory Map
| Start Address | End Address | Size | Description |
|---|---|---|---|
| 0x0000000000000000 | 0x0000000000000FFF | 4 KiB | IDT - 256 descriptors (each descriptor is 16 bytes) |
| 0x0000000000001000 | 0x0000000000001FFF | 4 KiB | GDT - 256 descriptors (each descriptor is 16 bytes) |
| 0x0000000000002000 | 0x0000000000002FFF | 4 KiB | PML4 - 512 entries, entry 0 points to PDP at 0x3000, entry 256 points to PDP at 0x4000 |
| 0x0000000000003000 | 0x0000000000003FFF | 4 KiB | PDP Low - 512 entries |
| 0x0000000000004000 | 0x0000000000004FFF | 4 KiB | PDP High - 512 entries |
| 0x0000000000005000 | 0x0000000000005FFF | 4 KiB | Init data |
| 0x0000000000006000 | 0x0000000000006FFF | 4 KiB | Stack |
| 0x0000000000007000 | 0x0000000000007FFF | 4 KiB | boot_params |
| 0x0000000000008000 | 0x000000000000FFFF | 32 KiB | Stub |
| 0x0000000000010000 | 0x000000000001FFFF | 64 KiB | PD Low - Entries are 8 bytes per 2MiB page |
| 0x0000000000020000 | 0x000000000005FFFF | 256 KiB | PD High - Entries are 8 bytes per 2MiB page |
| 0x0000000000060000 | 0x000000000009FFFF | 256 KiB | Free |
| 0x00000000000A0000 | 0x00000000000FFFFF | 384 KiB | Legacy BIOS ROM Area |
| VGA RAM at 0xA0000 (128 KiB) Color text starts at 0xB8000 | |||
| Video BIOS at 0xC0000 (64 KiB) | |||
| Motherboard BIOS at F0000 (64 KiB) | |||
| 0x0000000000100000 | 0xFFFFFFFFFFFFFFFF | 1+ MiB | The software payload is loaded here |
Init data
| Start Address | End Address | Size | Description |
|---|---|---|---|
| 0x0000000000005800 | 0x00000000000058FF | 256 B | MMIO devices |
| 0x0000000000005900 | 0x00000000000059FF | 256 B | memmap |
| 0x0000000000005A00 | 0x0000000000005AFF | 256 B | cmdline |
BareMetal
The BareMetal kernel in this repo has been adapted from the general version. VirtIO drivers have been reworked to use MMIO.
Virtio-Block and Virtio-Net drivers are present. Virtio-Vsock, and other Firecracker-supported devices, are yet to be added.
SMP is not included in this version of BareMetal and will be added at a later date. BareMetal uses 2MiB of memory - A microVM should be provisioned with at least 4MiB of memory so 2MiB can be mapped at 0xFFFF800000000000. 2MiB is the minimum if the application runs from kernel memory (there is some room).
The kernel binary (actual code + data) is currently ~5500 bytes. About 913408 bytes of padding is added to the end of the kernel so that the monitor utility is already at 0x1E0000 in memory. The monitor utility is also padded to be 131072 bytes in length so that any application will be at 0x200000 in memory (which is mapped to 0xFFFF800000000000).
TODO
- parse ACPI tables for APIC IDs (SMP removed from this version)
- when building in unikernel mode exclude monitor binary
- re-org BareMetal memory usage to make more room for payloads when running with only 2MiB of RAM
- combine build.sh into baremetal.sh
//EOF