ML research datasets from ArXiv and Semantic Scholar (JSONL, quality-scored)
dangerlego5
·
2026-06-16
·
via HN's home page
 | |
I kept rebuilding the same arXiv scraper at the start of every ML project. After the third time I wrote a dedup pipeline, I automated the whole thing. The interesting part is that the pipeline is shared; if two people subscribe to
the same topic, they share one crawl and one deduplicated record pool. Happy to talk through the pgvector dedup approach if anyone's curious. |
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。
