joseph schmitt

Keeping an Up-to-date Twitter Archive

A faster, simpler way to browse your (embarrassing) tweet history

Mar 20 2013

Fork me on GitHub

Twitter, while a great source of real-time or quasi-recent information, is pretty terrible at showing you anything much older than a few weeks. Most of the time this isn't an issue since the likely use case is simply an answer to the question they ask when composing a tweet: what's happening [right now]?. However, if you're fascinated by history or are just curious as to how awful and embarrassing you sounded 5 years ago, Twitter's search isn't of much help.

Luckily, I'm not the only one with the morbid fascination of finding old embarrassing/fascinating tweets. Hosted services like Manton Reece's Tweet Library or self-hosted solutions like Tweet Nest did a stellar job of scratching this itch. I had a spare server lying around and am a fan of owning my data whenever possible, so I opted for a nice Tweet Nest install that I've been running happily for the last few years.

The Twitpocalypse Cometh... Again

Unfortunately, for various reasons that I won't get into nor pretend to fully understand, Twitter is retiring the unauthenticated version of their API that my beloved Tweet Nest relies on. This means that, come late March, I'll no longer have an up-to-date backup of my tweets on a server that I own. So, what's a Twitter archivist like myself to do?

I found a few options. First, there's the aforementioned Tweet Library. Again, while Manton has made a great app and service, I want my publicly viewable tweet archive to be on a server that I own and manage so I don't have to worry about the service running out of money or shutting down1. The second solution I found was Dr. Drang's: a pretty good solution that's completely in his control wherein he saves all his tweets to a large text file on Dropbox. This is neat in that it's completely format agnostic and makes searching pretty quick and painless, but it's not easily shareable nor viewable from the public web.

However, the closest solution I found to my original Tweet Nest setup was from an unexpected source: Twitter itself. A few months ago, Twitter announced a downloadable Twitter Archive for your tweets. At first I thought this was merely a convenient way for me to get a local copy of tweets older than 3,200 tweets ago. However, after opening my archive up, I found not only the aforementioned local copy of tweets as wonderfully formatted JSON, but also an HTML page with a nice design to browse said tweets in. A quick FTP upload up to my server, and I was in business!

So, so, so close

But all was not sunshine and rainbows. While I finally had a nice, publicly accessible webpage of all of my tweets since joining the service2, it wouldn't auto-update, meaning I would have to periodically request a new version of the archive from Twitter if I wanted to see all of my latest tweets.

Sigh.

All is not lost! The magnanimous Dannel Jurado took it upon himself to write a ruby gem called Grailbird Updater that will fetch your latest tweets and fill them in with the same JSON format that the original Twitter archive came in. And best of all, after a little prodding, the gem even works with API v1.1. So now I had a publicly viewable, always up-to-date, complete archive of my tweets on a server I owned: victory is mine!

Or so I thought...

Slightly out of touch

The tweet browsing experience on desktop browsers is fantastic. It's clear the designers and developers who worked on it put some amazing time and effort into a pet project that I bet very few people at the company cared anything about. However, the experience on mobile devices was... sub-par. Lots of uses of position: fixed and JavaScript scrolling events made things slow, buggy, and just plain unusable. I tried writing some CSS on top of the ones they shipped with to see if it could be saved, but that effort ended in futility.

So I said: “to hell with it, I'm a web developer: how hard can this be?”. I set out to re-write the front-end to the tweet archive from scratch with a focus on speed, simplicity, and most of all, touch-compatibility. A week and a half and a 6-hour Internet-enabled flight later I had Simplebird.

The Simplest Bird

Simplebird is a complete replacement to Twitter's Twitter Archive front-end, while keeping 100% compatibility with their JSON archive format. It uses a relatively small amount of jQuery and a templating framework called Mustache to loop through all the JSON and write out your tweets on the page grouped by month. It's far from complete3 and I'm sure has some bugs, but it's pretty easy to try out and use with your own tweets: just drop in your data/ folder from your tweet archive in the tweets/ directory, upload it to a server, and you're good to go. This will even work with Jurado's updater: pass the tweets/ directory in as the archive location instead of the one you got from Twitter and it should auto-magically keep your tweets up-to-date!

Finally: I have my ideal Twitter archive solution, which you can check out here.


  1. There seems to be enough of that going around lately. 

  2. A feat even Tweet Nest couldn't manage since I set it up after I already had more than 3,200 tweets. 

  3. I already have a GitHub Issues list going with features I want to work on. If you have any requests feel free to add them, or if you're a developer with a kind soul and want to help out, feel free to get in touch.