A four-stage pipeline that ranks Show HN posts by estimated merit using TrueSkill + LLM-as-judge, then surfaces where that ranking disagrees with actual HN points.

The core claim: HN upvotes correlate with how easy a project is to evaluate from a screenshot. Deep technical work that requires reading the README to appreciate tends to get penalized. This pipeline is a second opinion that weights depth, novelty, and craft instead.

Results for 1,000 recent Show HN posts: report

How it works

Four stages, each reading from the previous stage's JSON output:

scrape.py         →  data/posts.json
fetch_content.py  →  data/content.json
rank.py           →  data/ratings.json + data/matchup_log.json
publish.py        →  data/report.json + data/report.md

Collection — Algolia API (tags=show_hn, sorted by date). For text-only posts, links are parsed out of story_text so there's something to judge.

Content extraction — readability-lxml, truncated to 20k chars per post in content.json, then trimmed to 6k per side at judge time. Fallback chain: post URL → links inside story_text → the story_text body itself. fetch_status records which path won.

Judging — a DeepSeek V4 Flash LLM does pairwise comparison with a rubric that favors depth, novelty, and craft, and discounts marketing polish. Each pair is judged twice with order swapped (A-vs-B and B-vs-A). If the verdict flips, it's recorded as a draw rather than a win — this is the most effective mitigation for LLM positional bias. The judge can also output VERDICT: TIE explicitly; TrueSkill ingests draws natively.

Rating — TrueSkill with draw_probability=0.10. Items start at μ=25, σ=8.33. Matchmaking pairs each item with neighbors in the current μ-sorted order so comparisons are between near-equals. Only items with σ below a threshold enter the published lists; posts under 48 hours old are excluded from "buried gems" since they may simply not have been seen yet.

Every post gets two independent percentile ranks — by merit and by upvotes. The disagreement is the story.

Setup

pip install -r requirements.txt
export DEEPINFRA_API_KEY=...                           # required
export DEEPINFRA_MODEL=deepseek-ai/DeepSeek-V4-Flash  # optional, this is the default

Quick run (50 items, ~250 pairs, ~500 judge calls)

python scrape.py 50
python fetch_content.py
python rank.py
python publish.py

Full run (1,000 items, ~6,000 pairs, ~12,000 judge calls)

python scrape.py 1000
python fetch_content.py --workers 16
python rank.py --matchups-per-item 12 --workers 16
python publish.py --sigma-max 3.5 --top-n 25

At this scale it's worth considering a second pass targeting only high-σ items for extra matchups, and spot-checking the top over/underrated calls with a second model family before publishing.

Limitations

This is not a replacement for HN's ranking. The LLM judge has its own biases — it appears to favor projects with detailed READMEs and clear technical novelty claims, which may overrate well-documented niche work and underrate good tools with terse documentation. Bidirectional judging reduces positional bias but doesn't eliminate model taste. Treat the output as one lens, not a ground truth.