← All Posts

Better news


The goal of this post is to explain how to set up a personal news aggregation server.

Why bother?

Primarily because it gives you back control over your news consumption. To break that down, a self-hosted news aggregation setup is:

What you’ll get

Twitter Feeds, YouTube channel updates, new blog posts, breaking news from your favorite news outlets, newsletters, new subreddit posts, and more — all in one place, on all your devices. Sorted chronologically, not by an algorithm.

A screenshot of the Reeder RSS client on macOS.

How to set it up

Okay, now on to the fun part. Let’s start with a high-level overview of what we want to achieve and the different components needed.

Schematic overview of the components I use in my self-hosted news aggregation setup.

Our ambition is to leverage RSS, an open standard for getting website updates (think: new blog posts) in a machine-readable format. The simplest possible way to use RSS would be to use e.g. an iPhone app like Reeder and subscribe to websites that support RSS directly.

This e.g. is the RSS feed for Quartz. That’s already something, but there’s one main drawback: if you want to read news on more than one device, you’d have to keep track of subscriptions and read articles manually. Which of course you’d want to avoid.

Syncing subscriptions across devices via Miniflux

So, the first thing that we’d want to set up is an aggregation service that does this for us. I’ve grown to like Miniflux for its speed and simplicity, but there are other open-source ones out there, e.g. FreshRSS. For setting up Miniflux though, here’s a rough outline of what you’d want to do:

  1. Get a server up and running — e.g. a Raspberry Pi that you’d run in your own home or a VPS instance run by a cloud provider. Setting the server up from scratch is not in the scope of this post, but for setting up a Raspberry Pi, I’d recommend the official tutorial and for a cloud VPS, I’d recommend going for the preconfigured Docker droplet on Digital Ocean as it’s easy to use and cheap.
  2. Install Docker — because it will make setting up the array of services we’ll need to complete our toolchain a lot easier and more portable. Here’s the official documentation on how to do that. If in doubt whether Docker is already installed, you can enter docker -v in the shell of your server.
  3. Since all of the services we’ll use live in Docker containers, it would be great if we could manage it all in one simple place — that’s where Docker Compose comes in. Install it on your server too.
  4. Let’s now set up Miniflux, using their installation instructions for Docker and specifically their Docker Compose code snippet.
  5. After starting the containers via docker-compose up -d, you should be able to open the Minuxflux web interface by navigating to your server’s IP address in your web browser. (Miniflux’s default port is 80 so you can omit it, but if you specified a different one you’ll need to specify it too, making it ip-address-of-server:chosen-port). Be mindful to follow their specific instructions for Docker Compose.
  6. If you see a login prompt now, you’ve made it 🎉 Use the password you’ve specified for the first user.

Keep in mind that currently, Miniflux is running without SSL. To achieve better security, I recommend setting up a reverse proxy via Traefik, leveraging the fact that we use docker-compose. Here’s an excellent article that describes how to pull that off. I have my news setup running on a Raspberry at home and use a Wireguard tunnel to connect to it.

Connecting clients via Fever

You could perfectly well stick to just using the Miniflux web interface on all your devices. However, I prefer using the excellent Reeder client on both my iPhone and my Mac. I’m just a sucker for beautiful, focused interfaces. You’ll notice that Reeder has a built-in feed sync service via iCloud, so you might ask yourself why setting up a server with Miniflux was necessary. While I’m sure syncing via iCloud works well too (and it’s most likely less of a hassle to set up), I like to keep my toolchains as modular as possible. In this case, I don’t want to depend on iCloud, which has a hard dependency on iOS/macOS. Plus, we’ll need the server anyways for a couple of services we’ve yet to discuss, so we might as well go all in.

So, how do you access your feeds in Miniflux via Reeder? Reeder supports the Fever API — and Miniflux does too 🤝.

  1. Follow the official instructions to enable Fever on your Miniflux instance.
  2. After downloading Reeder on your device, go to Settings > Add Account. The labels might be different depending on the version, but essentially adding an account is what we want to do.
  3. Choose Fever.
  4. Under Server, enter the address of the Miniflux web interface you’ve used previously, but append /fever.
  5. Enter the login data you’ve chosen in Miniflux for the Fever integration.

Done. Any feed that you’ve subscribed to, should now show up in Reeder. Repeat steps 2-5 for every device you want to use Reeder on.

RSS All the things

Currently, we’re only able to add news sources that support RSS feeds. Sadly, not all of them do. If we truly want to have all our news consolidated in one single place, we need to step up our game.

Email newsletters (Substack etc.) via Notifier.in

With an increasing number of high-quality content now being exclusively on Substack, for me, a news setup wouldn’t be complete without supporting email newsletters.

What we need is a tool that receives email newsletters on our behalf and converts it to an RSS feed. I personally use Notifier.in. If you feel uneasy with a third party hosting this for you, you can go for self-hosting Kill-The-Newsletter, which looks like an excellent alternative.

Now, when subscribing to a newsletter you can either directly submit the email address that Notifier/KTN created for that purpose or — in the name of modularity — use your main email address and set up forwarding rules. This way, should you change the setup, you don’t need to re-subscribe to your newsletters.

Make sure to subscribe to the RSS feed given by Notifier/KTN. Ideally, test whether emails received show up in Miniflux by sending a test email to the newsletter email address (or, a message to your main address matching the forwarding rules you’ve set up).

Telegram channels, Twitter feeds, and more via RSS-Bridge

Now onto the other walled gardens that we’d have to graze manually for new content if we don’t find a way of adding them to our news setup.

Luckily, there’s a great self-hostable tool called RSS-Bridge. It does what its name suggests: you can set up multiple bridges (read: sources). It will then go on and scrape the corresponding website and convert its content into an RSS feed. RSS-Bridge does this in regular intervals and also hosts the RSS feeds, so you can be sure your content is up to date.

Since RSS-Bridge works by scraping HTML content, things can break when the website changes. Especially for more popular bridges, updates are released quickly though to patch things up in case they stop working.

Let’s set up RSS-Bridge via the official Docker Container — add the following lines to your docker-compose.yml:

rss-bridge:
  container_name: rss-bridge
  image: rssbridge/rss-bridge:latest
  volumes:
    - /folder-containing-a-whitelist-file/whitelist.txt:/app/whitelist.txt
  ports:
    - 9091:80
  restart: unless-stopped

Keep in mind that with YAML, indentation matters.

Two things to point out here:

  1. The whitelist file is necessary to enable bridges. I’ve whitelisted all bridges by having a * in the text file. If you want to be more prudent than that, you can specify the names of the bridges you want to use. Put the file into e.g. your home folder and change /folder-containing-a-whitelist-file in your docker-compose.yml to the actual path so RSS-Bridge can access it inside its container.
  2. Upon starting the container, we can access the web UI via ip-address-of-server:9091. Keep in mind that we also want Miniflux to be able to subscribe to the feeds RSS-Bridge outputs. The option I’ve used to have both services running inside Docker and be able to communicate with each other is a user-defined bridge in Docker. I then replace it with the IP address assigned within the network. For this to work, both Miniflux and RSS-Bridge need to be in the same user-defined bridge. This is how my docker-compose looks like with RSS-Bridge and Miniflux running side-by-side, including the user-defined bridge:
version: '3'
services:
  miniflux:
    container_name: miniflux
    image: miniflux/miniflux:latest
    ports:
      - '7878:8080'
    networks:
      your-custom-network-name: { ipv4_address: 172.19.0.2 }
    depends_on:
      miniflux-db:
        condition: service_healthy
    environment:
      - DATABASE_URL=postgres://xxx:yyy@172.19.0.3/miniflux?sslmode=disable
      - RUN_MIGRATIONS=1
      - CREATE_ADMIN=1
      - ADMIN_USERNAME=xxx
      - ADMIN_PASSWORD=yyy
    restart: unless-stopped
  miniflux-db:
    container_name: miniflux-db
    image: postgres:13
    networks:
      your-custom-network-name: { ipv4_address: 172.19.0.3 }
    environment:
      - POSTGRES_USER=xxx
      - POSTGRES_PASSWORD=yyy
    volumes:
      - miniflux-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'miniflux']
      interval: 10s
      start_period: 30s
    restart: unless-stopped
  rss-bridge:
    container_name: rss-bridge
    image: rssbridge/rss-bridge:latest
    volumes:
      - /folder-containing-a-whitelist-file/whitelist.txt:/app/whitelist.txt
    ports:
      - 9091:80
    networks:
      raspbi: { ipv4_address: 172.19.0.4 }
    restart: unless-stopped
volumes:
  miniflux-db-volume:
networks:
  your-custom-network-name:
    driver: bridge
    ipam:
      config: [{ subnet: 172.19.0.0/16, gateway: 172.19.0.1 }]

Let’s walk through an example of adding a Twitter feed. Let’s say you want to subscribe to Nassim Taleb’s Twitter feed via Miniflux. First, open up the RSS-Bridge interface. Search for the Twitter bridge and hit Show more. Take the “By username”-Option and enter the respective Twitter handle, nntaleb in our case (note that the @-segment of the handle is not needed for RSS-Bridge).

After hitting Generate feed, should now be redirected to a page showing you his latest tweets in chronological order. On top of them, hit the Atom-button to get a format that we can subscribe to in Miniflux.

Head over to Miniflux and subscribe to the feed, note the IP-adress from our docker-compose.yml shown above: 172.19.0.4/action=display&bridge=Twitter&context=By+username&format=Atom&norep=on&noretweet=on&u=nntaleb

Done 🎉 Once subscribed, new Tweets will show up in the Miniflux Web Interface and Reeder (or any other RSS client you’ve set up).

The steps for other bridges are similar, in case you want to check out a list of supported ones before installing RSS-Bridge, you can do so here. I also highly recommend reading their rant.

Always having to visit the RSS-Bridge web interface when you want to subscribe to a new Twitter account might seem cumbersome to you — it indeed is, and that’s why we are going to make things a lot simpler for us in the next section.

Add new feeds via iOS shortcut

Miniflux supports subscribing to new feeds via URL parameters. Knowing this, we need to find a way to open up the Miniflux URL including a dynamic variable (the feed URL itself). In practice, the URL would look something like this: http(s)://your-server-ip:port/bookmarketlet?uri=https://cms.qz.com/feed, in this example using the RSS feed for Quartz.

The Shortcuts app on iOS (and macOS) allows you to take all kinds of input parameters and execute user-defined actions on them. This is exactly what we’ll use to create a shortcut which you can trigger from the Share Sheet inside Safari on iOS.

It’ll take the current URL as an input, append it to our base URL for the Miniflux bookmarklet and open the result in Safari. There, the Miniflux bookmarklet page will receive the URL and upon confirmation look for RSS feeds to subscribe to.

A screenshot showing how macOS shortcuts can be used to subscribe to RSS feeds.

Hint: you can assign a fitting icon + name to the shortcut before adding it to your share sheet. You can also create & edit the shortcut on macOS, it will be synced to your iPhone via iCloud.

Now, with “native” RSS feeds taken care of, let’s take a look at the trouble makers that we’ve dealt with via RSS-Bridge before. Wouldn’t it be great if upon coming across an interesting Twitter feed, you could directly subscribe to it via Miniflux?

With Shortcuts, we can manipulate the string that is being processed and passed on to Miniflux, which is exactly what we need to subscribe to Twitter feeds (and all kinds of other sources) quickly.

You might have noticed how the feed URL which RSS-Bridge outputs is structured straightforwardly. So, if in addition to subscribing to Nassim Taleb, you’d like to subscribe to Morgan Housel, it’s easy to see which part of the feed URL you’d have to replace to make it work:

172.19.0.4/action=display&bridge=Twitter&context=By+username&format=Atom&norep=on&noretweet=on&u=nntaleb

So, given this input https://twitter.com/morganhousel, we want to open 172.19.0.4/action=display&bridge=Twitter&context=By+username&format=Atom&norep=on&noretweet=on&u=morganhousel.

This is how the action looks like in the Shortcuts editor on macOS, with ([^\/]+$) being the RegEx to match the Twitter handle.

A screenshot showing how to use macOS shortcuts to subscribe to a Twitter feed.

For YouTube channels, the approach is the same, though I don’t use RSS-Bridge to subscribe to YouTube channels, since they provide RSS feeds for channels via https://www.youtube.com/feeds/videos.xml?channel_id=xxx. So, if you repeat the same trick as above, just using this URL schema, you’ll have our little trick working for YouTube channels too.

As you will have noticed, Shortcuts is quite a powerful tool — if you have any other sources you’d like to get into Miniflux, I’m pretty sure that you can set up a way to subscribe to them conveniently via Shortcuts. Heck, Shortcuts even allows you to send HTTP requests and process their response, so in theory, you could create a serverless function to offload more complex logic if necessary.

Watching website changes via changedetection.io

With RSS + RSS-Bridge, we’ve got almost all potential news sources. But what if you want to watch website changes of a local retailer to see if prices changed?

This is where the last piece in our toolchain comes in, changedetection.io. It monitors a website in regular intervals and keeps track of its (text) changes. You can restrict it to watching only certain selectors and filter for desired text as well to avoid false positives. Among other output formats, changedetection.io offers RSS feeds, making it easy for us to consolidate its output together with the other news sources in Miniflux.

We’ll add changedetection.io to our setup via — you guessed it — via their official Docker container. Let’s add the following lines to the services in our docker-compose.yml:

changedetectionio:
  container_name: changedetectionio
  image: ghcr.io/dgtlmoon/changedetection.io:latest
  volumes:
    - /folder-containing-configuration-files-for-the-service:/datastore
  ports:
    - 9092:5000
  networks:
    raspbi: { ipv4_address: 172.19.0.5 }
  restart: unless-stopped

Once you’ve started the container via docker-compose up -d, you can set up new change detections with its web interface and subscribe to the RSS feed via the RSS icon in the bottom right corner of the list.

Make sure to also use its different options to restrict the change detection to the parts of the website you care about, otherwise, you’ll end up with a lot of noise.

News sources

So far, we’ve been stitching different tools and technologies together to build a modular and flexible news setup — let’s spend some minutes to talk about the heart & soul of your news setup — the actual news sources.

You’ve now got a pretty powerful news setup on your hands — to give you some ideas about what you can do with it, here are some examples of the news sources I’ve set up:

SourceVia
GitHub Trending RepositoriesRSS-Bridge
OECD Report ReleasesNative RSS feed
New paper abstracts via e.g. the Oxford Quarterly Journal Of EconomicsNative RSS feed
Glassdoor Interview Reports for companies I’m interested inRSS-Bridge
Glassdoor Reviews for companies I’m interested inRSS-Bridge
Google Search Agent for my nameNative RSS feed
Google Search Agents for companies I’m interested inNative RSS feed
Twitter feeds, new posts in Subreddits, Telegram channelsRSS-Bridge
Substack newsletters like The DiffNotifier
Non-Substack newsletters like Exponential ViewNotifier
Personal blogs like David Perell’sNative RSS feed
YouTube channelsNative RSS feed
New articles from magazines and newspapers like The EconomistNative RSS feed / RSS-Bridge/Notifier
New refurbished MacBooks (in the market for one) via Refurbed-TrackerNative RSS feed
Whether my favorite sneakers are on sale on the Geox sales websiteChangedetection.io

Housekeeping

With everything self-hosted, you’ll inevitably need to concern yourself with applying updates and system security. With great power comes great responsibility, I guess 🤷

In case you use a Raspberry for hosting, I recommend this guide to get you started. Most of the advice applies to non-Raspbian systems too, so I think it’s a good resource generally to get started with system security on Linux.

The examples and code snippets above are not meant for the setup to be accessible to the wider internet (as mentioned before, I use Wireguard to connect to the server). If you want to do so, I highly encourage you to set up a reverse proxy and SSL.

For automating container updates, people seem to like Watchtower.

Where to go from here

The steps above outline the setup that I’m running currently. As mentioned in the introductory paragraph, however, having an own self-hosted news setup is an effort that compounds in its usefulness over time.

Not only because you’ll actively curate your news sources and with it, your world view — but also, because you can extend the tech stack according to your needs.

Here are some ideas to extend your setup:

Have fun 👋