Anecdotes about web APIs

Services that provide web APIs are amazing. Tom Scott made a video about this recently, where he tells the story of how all the newfangled web 2.0 services provided programmatic access to their data via simple web requests, and how great it was to be able to play around with all of that data freely.

I thought this would be a good time to reflect on my own experiences with some web APIs, and talk a bit about why they’re an important part of a great service for me.

Part 1: Oh, Twitter…

The video mentioned above covers Twitter specifically, noting that they eventually released new features in their apps that were never added to the public API, therefore third party clients had no chance of implementing them for their users.

I have very, very painful personal experience with this specific problem. A few years ago, I wrote a very basic Twitter app as a learning project for Android development. The feature that caused me the most headache by far (other than the infinite scrolling timeline) was implementing replies.

Replies are one of the most basic features of Twitter: people share their thoughts, others reply to them. Users build up threads of tweets using replies, either by themselves when they have a lot to say and don’t know how to write a blog post, or by having a conversations with others with replies. When you’re viewing a given tweet, ideally, you’d like to be able to see the replies to it. To my utter disbelief, Twitter simply did not have an API for querying the replies to a tweet.

What was the recommended way, according to the wisdom of the internet at the time? Well, you could…

• Search all of Twitter for mentions of the user who posted the tweet. Hopefully, this sounds as painful as it is in practice. To do this in a slightly saner way, you could at least add a filter to only get tweets that were posted after the tweet in question.
• Then, you’d comb through all of those results, and check each tweet to see whether it is a reply to the original, as there was an ID referencing the original tweet in replies.

Doing this costly operation would get you direct replies to a tweet. If the tweet was relatively recent, this was… Fine, thanks to the date filter. If it was a bit older, you were in trouble. But the real trouble began when you wanted to follow threads by fetching second or third level replies, or perhaps more. This would go very exponential*, very quickly.

*That is the precise technical term for this complexity.

API limitations

APIs like this, especially when public, can not be unregulated. There are so many people who could access them that the requests coming in must be controlled and limited somehow. It simply doesn’t make sense from a financial standpoint to provide the whole world unlimited access to your server resources (again, something covered well in the video mentioned at the top).

Twitter has rate limits on specific requests like anyone else, limiting the number of requests they’ll serve for a given endpoint in a given timeframe. However, they also have a rather strict limit of another kind on third party apps: they only hand out 100,000 tokens per app (barring special exceptions from them).

This means that if you put in the time and effort to create an amazing Twitter app - fighting your way through all the hardships that their APIs present - you have to do so with the thought of never being able to acquire more than 100k users hanging over your head.

Developers of third party Twitter apps would actually unlist their apps from the store when they got close to this limit. This was done to prevent new people from buying the apps, only to find that they can’t log into their account after downloading it, because there are no tokens left for that app.

Twitter has a clear financial incentive for doing this, of course. If you’re not using the official apps, they can’t serve you their own ads. It’s a small wonder they haven’t shut down their APIs entirely yet.

Part 2: Wallet’s API story

One of my favourite apps is Wallet by BudgetBakers. This is an expense tracker that lets you create records of transactions, attach all sorts of information to them, and then organize and view them in various ways. A highlight of this service for me was the API they provided for their users (if memory serves, this was tied to the payed premium tier).

Discovering the API

When I moved to Wallet from my previous expense tracking system, I imported my data, but there were a few things that weren’t imported quite right, and needed some editing. A few hundred records or so.

I initially started doing this by hand. When I realized how long it would take and how repetitive it was, I wrote a script that issued ADB commands to my phone, performing the tapping sequence by x and y coordinates over and over again. This worked… Surprisingly well. However, I then stumbled upon the API documentation for the service. Score.

It took me about half an hour to spin up a new Kotlin/JVM project, and write the simple, script-like code that would fetch all my records, filter them for the ones that needed fixing, and then hurl POST requests at the server with the updated contents. It was marvellous.

Not only could I do this kind of mass updating to my data when I wanted, but I also knew that I’d have arbitrary access to it. At any time, I could fetch my data from code, store it, and analyze it easily. I could build applications, or periodic processing jobs on top of this.

| || || |_

All of that is written in past tense because Wallet shut down their API last year, with essentially no communication about it. It was just gone, as their users leaving feedback about this can attest to. I have also contacted their support about this, but unfortunately got no sensible response, despite still being a paying user of the service.

I know that there are costs of maintaining an API for clients users outside the app developers. But when your app is so data driven, and especially when you had an API in the past that was so powerful, just cutting it off one day is terrible practice. And if the costs need to be covered, it can always be a part of a higher tier subscription plan. I, for one, would happily pay for it.

Alternatively, a reliable export-all, import-all feature (emphasis on this being all the data, with metadata and properties) could also substitute this, if API access is just too costly. This would allow processing the data offline at least.

Part 3: Dynalist

The last little chapter in my adventures with public APIs will be about Dynalist. This is an infinite-zoomable-tree style outliner app, a genre perhaps established by WorkFlowy.

The app’s creators would encourage you to use this tool for a huge variety of things: taking lecture notes, writing, todo lists, journaling, schedules, checklists… Anything that’s text based, really. I found that the app is indeed quite capable of these things, especially task management, thanks its powerful filters.

Most importantly, however, the service has a simple, yet excellent API, which allows you to query and update all of your data stored in it. This includes not just the structure of the tree and the basic text content of the nodes, but everything that the app keeps track of for each node. Colors, checked states, modification dates, and many other properties are available in the API.

I’ve already used this to create an alternative view of some of my tasks outside of the app - my own dashboard backed by my data stored in the app, if you will. It simply fetches all my data from the API, builds the tree locally in memory, and renders filtered and transformed views of it as HTML (for now). Having this kind of direct access to my important data is reassuring, and increases both my trust in the service, as well as its usefulness. If it’s missing a certain view of the data, I can basically retrofit it myself, without getting too hacky.

Closing thoughts

Well, there’s three stories about services and their web APIs. One that’s always been terrible, one that fell from grace, and one that’s still holding going strong.

What’s the takeaway, then? Here are a few of my key thoughts, as a web service power user of sorts:

• APIs simply cannot exist without usage limitations.
• Even with limitations and payment requirements, many seem to find them difficult to maintain.
• A simple, limited API (in terms of access or functionality) is probably still better than having no programmatic access at all.
• A robust export/import feature might substitute an API in some cases.
• API access can be positioned as a separate, payed feature, perhaps even outside regular payment tiers.

I’ve realized that I used the words “app” and “service” quite interchangeably in this article, because I didn’t pay too much attention to the wording initially. If I think about it though, it’s the presence of an API that differentiates the two terms for me.

If I only have access to my data through the provided client, I’m just using an app. But if I get clients on multiple platforms, and I can also have API access… Well, then I’m using a real service.