Huzaifa Rasheed

Huzaifa Rasheed

Software Engineer

Blogs

Aggregating RSS Feeds Locally with a Simple Bash Script

Posted on July 9, 2024

Quick backstory: Many great RSS readers are out there, some open source as well. However, I wanted a way to get content from my RSS subscriptions as a reading checklist (markdown format) in a lightweight, cross-platform (Unix) fashion. This checklist needed to be in a local document for easier integration with my notes app. Since I couldn’t find anything like that - I decided to create my own.

The use case was straightforward: Get the latest content from my list of RSS subscriptions periodically. I also wanted this to work with a remote client (ex: for home server usage).

I started with a simple Python script but scrapped it since I didn’t want to install special dependencies. The same logic was possible in Bash and would also work in Windows (with git bash or wsl).

The bash script would only depend on some core Unix packages like xmllint, curl, date, and grep, all of which are pre-installed in most Unix environments. The date pkg tho, does have some differences b/w Linux vs other Unix-based systems which I had to handle separately.

The full script is here and it works like this

  1. First of all it requires a rss_subscriptions.csv file with a list of RSS subscriptions in this format.

    RSS Subscriptions CSV format

    Two things to consider:
    - The date_subscribed is in ISO 8601 format with UTC Time
    - The CSV file could have any number of subscription entries

  2. Now, the script needs to be made executable ex: $ chmod +x ./rss_watchdog.sh
  3. And run $ ./rss_watchdog.sh (with a cron job optionally)

It creates a ./rss_watchdog.log file with script logs and a ./reading-list.md file having subscribed content as a markdown checklist. The output directories are modifiable from the script.

You can find the full code on rss-watchdog

Question: Why not categorize feed?

If you did notice, the current setup doesn’t categorize feeds which is possible in most RSS readers. I simply didn’t require that feature.

If you want the script to categorize content, just modify the CSV file to make separate reading lists for different feed categories and modify the script to output to those reading lists (or any other way you prefer).


This is a cool little approach that helped me and maybe it helps you as well 😊!