How I built a yt-dlp CLI wrapper in Rust

I got tired of googling yt-dlp flags every time I wanted to download something. So I just built trawl, a CLI wrapper that makes the whole thing actually pleasant to use. Here's what's going on under the hood.

The basic idea

trawl is a thin layer on top of yt-dlp. You give it a URL, it figures out what you want, builds the right command, and runs it. Simple in theory. A bit messier in practice.

Running yt-dlp as an async subprocess

The whole thing is built around tokio::process::Command. Instead of blocking while yt-dlp does its thing, everything stays async:

let mut child = Command::new(ytdlp_path)
    .args(args)
    .stdout(Stdio::piped())
    .stderr(Stdio::piped())
    .spawn()?;

The important bit here is piping both stdout and stderr separately. yt-dlp writes progress info to stderr and regular output to stdout. If you just inherit both streams you get an uncontrollable wall of text in your terminal.

Instead I spawn two tokio tasks — one reads stderr and pulls progress percentages out of it, the other reads stdout and optionally shows log lines when the user passes --log.

Building the progress bar

I used indicatif for the progress bar. The tricky part is that yt-dlp spits out lines that look like this:

[download]  47.3% of 128.40MiB at 3.20MiB/s ETA 00:25

So I just do a simple string search for the percentage and feed it into the bar:

if let Some(pos) = line.find('%') {
    if let Some(start) = line[..pos].rfind(|c: char| c == ' ' || c == '[') {
        let percent_str = line[start + 1..pos].trim();
        if let Ok(pct) = percent_str.parse::<u64>() {
            bar.set_position(pct);
        }
    }
}

No regex, no extra deps, just string slicing. The bar itself is a plain line with no spinner:

ProgressStyle::with_template("  [{bar:50.cyan/237}] {pos:>3}%")?
    .progress_chars("━━─")

Auto-downloading yt-dlp

I didn't want users to have to go install yt-dlp manually before they could use trawl. So on startup it checks if yt-dlp is already in PATH. If not, it offers to grab it automatically and saves it to ~/.trawl/yt-dlp.

The download is just the GitHub releases page, with the right binary picked based on the platform:

let url = match (std::env::consts::OS, std::env::consts::ARCH) {
    ("macos", "aarch64") => "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos",
    ("linux", _) => "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp",
    ("windows", _) => "https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe",
    ...
};

For HTTP I went with reqwest using rustls-tls instead of the system OpenSSL. That way cross-compilation just works with no native deps to worry about. After downloading on Unix, chmod 755 is set through std::fs::Permissions.

Spotify without any API keys

This part was the most fun to figure out. For individual Spotify tracks you don't actually need an API key at all. Spotify bakes the full track metadata right into the page as Open Graph tags.

The trick is sending a Googlebot user-agent. Spotify serves complete HTML to crawlers:

let client = Client::builder()
    .user_agent("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
    .build()?;

From there I just pull og:title and og:description out of the raw HTML. The description has the artist name in it, separated by a middle dot. I take that and turn it into a YouTube search query:

ytsearch1:Artist Name - Track Title

Then hand it off to yt-dlp. That's it.

Albums and playlists are a different story since they don't have og: data, so those do need API credentials. But for single tracks this approach works great.

One thing that caught me off guard: some artist names on Spotify use the Arabic comma ، (U+060C) instead of a regular one. Had to explicitly handle that:

fn __clean(s: &str) -> String {
    s.replace('\u{060C}', ",")
        .split_whitespace()
        .collect::<Vec<_>>()
        .join(" ")
}

Real world data is always messier than you expect.

The CLI with clap derive

For argument parsing I used clap with the derive API. The whole CLI is just a struct with attributes, which I find way cleaner than manually matching arguments:

#[derive(Parser)]
struct Args {
    url: String,

    #[arg(short = 'a', long)]
    audio_only: bool,

    #[arg(short = 'F', long, default_value = "opus")]
    audio_format: String,
    ...
}

One annoying thing: clap underlines section headers in the help output by default and it looks rough in most terminals. You have to manually override the styles to turn that off:

fn __styles() -> clap::builder::Styles {
    clap::builder::Styles::styled()
        .header(AnsiColor::White.on_default())
        ...
}

Things that surprised me along the way

mp4 files not opening on macOS. The default yt-dlp format selector picks VP9 video which QuickTime just refuses to play. Had to write a custom selector that explicitly requests H.264: bestvideo[ext=mp4][vcodec^=avc1]+bestaudio[ext=m4a]/best[ext=mp4]/best

yt-dlp output leaking into the terminal. Early on I used Stdio::inherit() for stdout. That sent everything straight to the terminal and completely bypassed the progress bar. Switching to Stdio::piped() on both streams and handling them manually sorted it out.

The Arabic comma thing. Already mentioned this above but it genuinely surprised me. Worth keeping in mind if you're ever scraping text from music platforms.

Give it a shot

cargo install trawl

GitHub: https://github.com/NotKiwy/trawl

If something breaks or you have questions about how any of this works, happy to chat in the comments.

推荐订阅源

DEV Community