The Hidden Blog

As it turns out, I do have a hostname.

WIP: Building a sitemap to internet archive tool

Published on Jul 23, 2024

I started working on a small—just for fun—side project recently and I thought I'll just use my newly rejuvenated blog to write some status updates from time to time.

Sometimes I come across smaller, niche website projects run by one or multiple enthusiasts where I'm wondering how well they are backing up the data. It's very easy to lose great resources just by someone forgetting to renew a domain, people going offline for a while or just someone's interest in a project fading.

I was looking for some way of submitting a sitemap to the Internet Archive itself, but it seems this use case isn't supported so I'm working on building exactly that. If there would be a way to make this paid and funnel all the proceeds directly to the Internet Archive as a donation that would be great, but this is something I haven't figured out yet.

It should probably have a way to auto-discover a sitemap too, but for now this will work.

This time I'm planning on using password less authentication (aka "magic links") so I don't have to set up Devise. I'm building it with Rails and SQLite. I might use Sidekiq again for the job scheduling. While Rails now has a nice Active Job implementation out of the box (Solid Queue) it lacks the web interface which I always liked about Sidekiq to keep an eye on things.