This post is a continuation of a microservice I've been building. You can check out my last post in the series here.

Automate News Source Ingestion with Firecrawl, OpenRouter, and GitHub Actions

TL;DR : I turned a manual, copy‑paste routine for adding news outlets into a fully automated, monthly GitHub Actions workflow. The pipeline scrapes a ranking page with Firecrawl, extracts clean URLs using three free‑tier LLMs on OpenRouter, generates Source YAML files, and opens a PR that lands these sources straight on the live dashboard post merge.

How it all began

When I first ingested sources in source‑score database I seeded it with five manually‑added outlets. It was enough to verify that the endpoints work, but I kept thinking about the “real” world: the top global media brands that people actually read. I wanted the repo to stay fresh without someone constantly creating PRs to add new sources.

The idea was simple on paper: fetch the latest list of popular English‑language news sites, turn each entry into a valid Source document, and let the existing CI validate and ingest them. In practice though, I ran into three big hurdles (which further break down into their own little challenges as you'll see):

1. Finding a reliable source – I discovered a page on PressGazette website that publishes the top 50 news sites each month, but the raw HTML was going to be a mess.
2. Extracting just the URLs – The page’s markup mixed headlines, ads, and footnotes. A regular regex wouldn't have been enough.
3. Keeping the process cheap – I didn’t want to spin up a paid scraper or a paid LLM every month.

The first breakthrough: Firecrawl does the heavy lifting

Firecrawl’s Python SDK made the scraping step painless and their free tier limits are high enough for my requirements. A tiny wrapper takes a URL and returns a clean Markdown:

def main():
    if len(sys.argv) < 2:
        print("Usage: scrape_firecrawl.py URL", file=sys.stderr)
        sys.exit(1)

    url = sys.argv[1]
    api_key = os.getenv("FIRECRAWL_API_KEY")

    fc = Firecrawl(api_key=api_key)
    try:
        doc = fc.scrape(url, formats=["markdown"])
    except Exception as e:
        sys.exit(f"Error scraping {url}: {e}")

    print(doc.markdown)

Running it against the ranking page gives me a tidy Markdown blob that still contains a lot of noise, but it’s a far better starting point than raw HTML. I want to store the scraped content in plaintext and forward it to LLMs as it is. For this requirements, the markdown format seemed the best of all the available options.

Using LLMs to extract news source URLs from the scraped data

1. Loads a Markdown‑processing skill, that I wrote (using the power of googling) to help the model process MD files, from my repo. I have some other skills there as well that I've been experimenting with to help me write these blogs, maybe I'll cover them later 😉
2. Sends the scraped Markdown file content to three free models (gemma‑4‑31b‑it, nemotron‑3‑nano‑omni‑30b, gemma‑4‑26b) until one returns a non‑empty answer. Sometimes requests to OpenRouter API fail because the model doesn't respond. To deal with that I'm iterating over these 3 models and so far at least one of them has always worked.
3. Asks the model to list the top‑10 latest most popular news outlets and output only URLs, one per line.

SOURCE_QUESTION = "What are the top 10 latest most popular news outlets in the world listed in this document? Only output URLs of these news outlets separated by new lines. Do not output anything else."
def process_raw_doc(md_processing_skill: str, md_doc: str, api_key: str) -> str:
    for model in FREE_MODELS_DOC:
        payload = {
            "model": model,
            "messages": [
                {"role": "system", "content": md_processing_skill},
                {"role": "user",
                 "content": f"Here is a web page in Markdown:\n\n{md_doc}\n\n"
                            f"Answer this question:\n{SOURCE_QUESTION}"}
            ],
        }
        raw_source_list = req_openrouter(payload, api_key)
        if raw_source_list:
            break
    if not raw_source_list:
        sys.exit("Error: All models failed.")
    return raw_source_list

The model’s answer still contains stray characters and occasional combined or shortened URLs.

To get a clean list of URLs, I'm running a second OpenRouter call that uses the built‑in web‑search tool to validate each line and verify the URL is correct. Following is the payload for my second OpenRouter request:

content = (f"Here is a raw list of URLs of news outlets with each line containing one or more unformatted URLs:\n\n"
    f"{raw_source_list}\n\n"
    f"Use web search to access these URLs and discard those that are invalid. Do not scrape the web page, only check if the URL is valid\n"
    "Based on successful web searches, return a list of corresponding properly formatted URLs without any extra test. Keep only one URL per line."
)
payload = {
    "model": model,
    "messages": [
        {
            "role": "user",
            "content": content
        },
    ],
    "tools": [
        {"type": "openrouter:web_search"}
    ]
}

From source URLs to proper Source YAML

With a verified list of URLs in hand, the next step is to generate a full Source document for each outlet. To achieve that goal, yeah you guessed it right, another OpenRouter API call.
I'm adding an ingested source yaml doc as a schema example, then asking the model to fill in the blanks by using the web search tool again:

content = (f"Extract schema from the following yaml document and store it as source_schema:\n\n"
    f"{sample}\n\n"
    f"Following is a list of urls of media outlets separated by new lines\n\n"
    f"{sources}\n\n"
    "Use web search to fetch information about these media outlets and create yaml docs for each of them following the source_schema schema. Do not output anything except for the yaml documents for these medial outlets separated by ---."
)

payload = {
    "model": model,
    "messages": [
        {
            "role": "user",
            "content": content
        },
    ],
    "tools": [
        {"type": "openrouter:web_search"}
    ]
}

The response is a string of YAML manifests separated by ---. I split, filter, and deduplicate against existing files to avoid overwriting anything.

Writing the new files (safely)

The deduplication logic avoids creating ingestion docs for sources that already exist in the repository. It loads every YAML under sources/, compares name and uri, and returns only truly new entries. The final loop writes each doc to a sanitized filename:

for doc_str in unique_src_docs:
    parsed = yaml.safe_load(doc_str)
    filename = parsed.get('name')
    if not filename:
        continue
    filename = re.sub(r"[^A-Za-z0-9._-]+", "-", filename.strip()) + ".yaml"
    path = os.path.join(sources_dir, filename)
    if os.path.exists(path):
        continue
    with open(path, 'w', encoding='utf-8') as f:
        f.write(doc_str.strip() + "\n")

All files land in sources/ ready for the existing validation workflow.

The actual automation: GitHub Actions workflow

Now that we have all the scripts ready, time for the final piece of the puzzle, the scheduled CI job.

1. It runs on the first day of each month
2. checks out the repo
3. creates a new branch
4. sets up Python
5. runs a tiny wrapper shell script that strings the three Python helpers together
6. commits any new YAML files
7. and opens a PR:

When the PR lands, the existing validate.yml workflow validates the new YAML files, and the post_on_merge.yml workflow posts them to the API when the new source docs get merged, now ready to be fetched by the live dashboard.

The results so far

• Source fetch workflow: https://github.com/SatyaLens/sources/actions/runs/25617971999/job/75199251257
• Automated PR: https://github.com/SatyaLens/sources/pull/22
• Ingestion run: https://github.com/SatyaLens/sources/actions/runs/25628583734/job/75228007512
• Live dashboard: https://satyalens.github.io/source-score-dashboard/

All the new sources appear with correct URLs and short descriptions, and the CI passes without a hitch. The whole process now takes less than a minute of human time each month, and that too just to merge the PR 😁

What’s next?

Conclusion

By chaining Firecrawl, OpenRouter, and GitHub Actions I turned a tedious, error‑prone task into a reliable, monthly automation. The result is instantly visible on the live dashboard, and the repo stays in sync with the world’s most popular news outlets.
If you try this yourself and have a suggestion or question, feel free to open an issue or drop a comment.

Further Reading

• Firecrawl docs – https://www.firecrawl.dev/docs
• OpenRouter API reference – https://openrouter.ai/docs