I've been building videopython, a Python library for programmatic video editing, processing and AI workflows. Repo: https://github.com/BartWojtowicz/videopython

The core idea is that edit is just a JSON / Pydantic plan. This plan can be validated against metadata of source video(s) via a dry run, then edit plan can be executed via streaming pipeline (many thanks to ffmpeg!) without holding frames in memory.

Thanks to the JSON/Pydantic format of edit schema, you can easily expose it as tool for LLMs and let your agent-of-choice be your personal editor (MCP support included). The usual flow is first analysing videos via a number of ML models to get both visual and audio understanding of all videos. Next, based on the understanding, you build the edit plan, validate it and render the final video. There's a simple refine loop catching errors and fixing the simple mechanical ones, which allows model to converge on executable schema.

Currently, the whole ML/AI stack is deliberately local, mostly via huggingface or ollama, I didn't want to rely on external APIs and want to keep it this way. There are some more features I omitted like automatic dubbing with voice cloning so feel free to check out the repo, play around with it or give some honest thoughts/criticism!

Honest disclaimer - this started as a hobby hand-written project, but nowadays most of the code is written with help of LLMs.