A Haskell tool that repeats entries from RSS/Atom feeds into new feeds. It fetches entries from source feeds, and selects a random subset using weighted sampling where older entries have higher priority, and inserts them in output feeds. This blog post describes the motivation behind it.
feed-repeat is available as statically-linked binaries
for AArch64 and AMD64 architectures in the releases. It is also
available as a Docker
image in the GitHub Container Repo.
This project is written in Haskell. You don’t need Haskell experience to use this tool, but you’ll need the Haskell compiler and build tools installed to build it.
The easiest way to install Haskell is via GHCup. Run GHCup to install GHC (9.10+) and Cabal. Alternatively, check your system’s package manager for pre-built packages.
Nix is optional. It is required for
nix builds and NixOS module support.
First, clone the repository and navigate into it:
git clone https://github.com/abhin4v/feed-repeat.git
cd feed-repeatEnter the Nix shell:
Run the scripts available in Nix shell:
# Build the project
build
# Build a static binary
build-static x86_64
# or build-static aarch64
# Run the tool with example config
runThis project can be used as a Nix module, a Systemd service, a Docker container, or hosted on GitHub Pages.
The project includes a NixOS module (nix/module.nix) for
easy integration into NixOS systems. Import it in your
configuration:
{
imports = [ ./feed-repeat/nix/module.nix ];
services.feed-repeat = {
enable = true;
# Feed configurations
config = [
{
sourceFeedUrl = "https://example.com/feed.atom";
outputFilename = "example-feed";
saveSourceFeedEntries = true;
repeatedEntryCount = 3;
minimumEntryAgeDays = 7;
maxEntryCountPerDomain = 1;
selectionAlpha = 0.9;
}
];
# Output and cache directories
outputDir = "/var/lib/feed-repeat";
cacheDir = "/var/cache/feed-repeat";
# Run frequency
timerOnCalendar = "daily";
# Optional: serve feeds via Nginx
enableNginx = true;
virtualHost = "feeds.example.com";
virtualHostPath = "/";
enableSSL = true;
};
}The module automatically:
For non-NixOS systems, a systemd service file
(configs/feed-repeat.service) is provided. To set it
up:
Create user and group:
sudo useradd -r -s /bin/false feed-repeatCreate required directories:
sudo mkdir -p /var/lib/feed-repeat /var/cache/feed-repeat /etc/feed-repeat
sudo chown feed-repeat:feed-repeat /var/lib/feed-repeat /var/cache/feed-repeat
sudo chmod 750 /var/lib/feed-repeat /var/cache/feed-repeatAdd web server user to feed-repeat group:
sudo usermod -a -G feed-repeat www-dataThis allows the web server (running as www-data) to read the output
feeds from /var/lib/feed-repeat. Change the user as
appropriate.
Install the service file:
sudo cp configs/feed-repeat.service /etc/systemd/system/Place your configuration:
sudo cp config.yaml /etc/feed-repeat/config.yaml
sudo chown feed-repeat:feed-repeat /etc/feed-repeat/config.yaml
sudo chmod 640 /etc/feed-repeat/config.yamlBuild and install the binary:
cabal install --installdir=/tmp --install-method=copy --overwrite-policy=always
sudo install -D -m 0755 /tmp/feed-repeat /usr/local/bin/feed-repeatOr use the binaries available for download.
Install the timer unit:
sudo cp configs/feed-repeat.timer /etc/systemd/system/Enable and start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now feed-repeat.timerA Docker image can be built with Nix:
# Enter nix-shell, then build the Docker image
build-docker x86_64
# or build-docker aarch64
# Load into Docker daemon
docker load < result
# Alternatively, you can pull the pre-built image from GHCR
docker pull ghcr.io/abhin4v/feed-repeat:latest
# Run the container
docker run --rm \
-v /path/to/config.yaml:/etc/feed-repeat/config.yaml:ro \
-v feed-repeat-output:/var/lib/feed-repeat \
-v feed-repeat-cache:/var/cache/feed-repeat \
feed-repeat:latestThe container runs as a non-root user (UID/GID
1000:1000). If you bind-mount host directories instead of
using named volumes, ensure they are writable by that UID, for
example:
sudo chown -R 1000:1000 /path/to/output /path/to/cacheNamed Docker volumes (as used in the examples above) are handled automatically by the Docker runtime.
Since the container runs once and exits, you need to schedule it externally:
Use the host’s cron or systemd timer to run the container periodically:
# Via cron: add to crontab (runs daily at 2 AM)
0 2 * * * docker run -v /path/to/config.yaml:/etc/feed-repeat/config.yaml:ro -v feed-repeat-output:/var/lib/feed-repeat -v feed-repeat-cache:/var/cache/feed-repeat feed-repeat:latestDocker Compose with Ofelia: Use Docker Compose with the Ofelia scheduler to run the container on a schedule:
services:
feed-repeat:
image: feed-repeat:latest
volumes:
- /path/to/config.yaml:/etc/feed-repeat/config.yaml:ro
- feed-repeat-output:/var/lib/feed-repeat
- feed-repeat-cache:/var/cache/feed-repeat
labels:
ofelia: "enabled"
ofelia.enabled: "true"
ofelia.my-task.schedule: "@daily"
ofelia.my-task.command: "/bin/feed-repeat --config /etc/feed-repeat/config.yaml --output-dir /var/lib/feed-repeat --cache-dir /var/cache/feed-repeat"
ofelia:
image: mcuadros/ofelia:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
command: daemon --docker
volumes:
feed-repeat-output:
feed-repeat-cache:Run with: docker-compose up -d.
Kubernetes: If deployed on Kubernetes, use native
CronJob resources for scheduling.
Docker Swarm: Use native scheduled task features if using Docker Swarm.
To serve the output feeds publicly, you can use any web server. Basic
example configurations are provided for Nginx, Apache, and Caddy in the
configs directory.
You can run and host feed-repeat on GitHub Actions and
Pages: fork this repo, edit config.yaml, and let GitHub
Actions publish your repeated feeds to GitHub Pages. See the full Hosting
on GitHub Pages guide for the step-by-step setup.
feed-repeat --config config.yaml --output-dir ./output --cache-dir ./cache--config FILE: Path to YAML configuration file
containing feed sources (required).--output-dir DIR: Directory where output Atom files
will be written (required).--cache-dir DIR: Directory where cached Atom files will
be stored (default: current directory).--user-agent STRING: User-Agent header to send in HTTP
requests (default: ‘feed-repeat/’).--validate: Only validate the config file and
exit.--verbose: Enable all logging.--quiet: Enable only warning and error logging.--version: Show version information.Create a YAML file with a list of feed tasks:
- sourceFeedUrl: "https://example.com/feed.atom"
outputFilename: "unique-id-1"
saveSourceFeedEntries: true
repeatedEntryCount: 3
minimumEntryAgeDays: 7
maxEntryCountPerDomain: 1
selectionAlpha: 0.9
passthroughNewEntries: true
- sourceFeedUrl: "https://another-site.com/rss.xml"
outputFilename: "unique-id-2"
saveSourceFeedEntries: false
repeatedEntryCount: 1
minimumEntryAgeDays: 14See config.yaml for all available parameters and their
meanings.
MIT
See CHANGELOG.
I consider this is a done software. Maybe some day when JSONFeed gets popular, I’d consider adding support for it. Other than that, I don’t foresee adding any new features. I’ll keep doing bugfixes, security fixes and dependency upgrades.
Please feel free to create an issue if you find a bug. I’m not inclined to accept pull requests unless there is a very compelling reason.
Disclaimer: This is a personal project. The views, code, and opinions expressed here are my own and do not represent those of my current or past employers.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。