How I'd Fix Twitter (and why that won't happen)

I started playing with twitter in January 2007, well before that year's SXSW breakout. I didn't start playing around with the twitter API until recently, joining the Twitter Development Talk Google Group, writing first my own command line update client (a very cunning combination of bash and curl which until this week totally failed to URL encode parameters. Doh.) and more recently a followers history tool (which isn't public, and won't be until I either get over accepting people's twitter credentials or some sort of third party authentication scheme is rolled out). My other posts about twitter are collected here: Topics/Web Services/Twitter.

I'll just preface the rest of this with: I have no inside insight, and have no idea what is wrong with twitter other than the general it's not scaling particularly well, is it?.

So, this is all just supposition.

I just wanted to make that clear.

The biggest problem with twitter? It's free.

• There's no cost to posting updates to twitter.
• There's no cost to pulling updates from twitter.
• There's no cost∗ to using the twitter API.
• There's no cost∗ to writing incredibly poor client code to use the twitter APIs.
• There's no cost∗ to distribute client applications which amplify twitter's usage dramatically.

I know, you're thinking WTF is that ∗ character there for? And there is a cost you bozo, there's a rate limit!.

I concede your point, there is a rate limit, which applies on a per–user basis.

But that's not a cost. That's not a price. The per–user limit doesn’t cause a user to sit back and think: Ooh, do I really want to tweet "Taking the dogs for a walk to the Telectrascope on Fulton Ferry."

I'm old school. I think it's the right, the duty of anyone running a web site to protect that site from abusive behavior, whatever that may be. I regularly rant on our nextNY list that people can and should take proactive measures to protect their sites (blocking users, bots, what–not). I don't think that we, as site managers (webmasters, whatever you want to call us these days) have to suck up all of the traffic thrown at a site just because as a general principle we're open to user generated content, APIs into our services, means of extending whatever it is we're providing (and, in theory, profiting from in some way).

So, I think twitter's problems come down to three separate areas which are intersecting:

• There's no cost to using twitter.
• Twitter wasn't designed for the way it's being used.
• Unintended side–effects of network effects.

 

How I'd fix twitter

Stabilize the current system but limit its capabilities.

What? Yeah, I'd lock down the system, perhaps even in its current requests limited to 30, non-Jabber state. Stabilize it in that configuration, declare victory, and move on.

Create an operations team with no development responsibilities.

When you're firefighting a technology fire two things usually occur: the same people deep in the bowels of the problem are usually expected to also keep the system running, while trying to figure out how to solve the problem. It doesn't work. Over time at ibm.com, we evolved to have a "duty webmaster" who had the run of the site, while everyone who was "off-duty" worked on application development or bug fixing. Asking the programmer to solve the problem while running the site, is like asking a firefighter to battle the fire while working with the architect and fire marshal to investigate the fire and prevent it from happening again.

Segregate API traffic to api.twitter.com

This has been mentioned on the mailing list several times, but I suspect that the twitter team is so busy keeping their heads above water that they just can't focus on this. After years of trying to consolidate IBM web sites into www.ibm.com, I've come to accept that there's certain traffic and application patterns which best serve the overall audience by being segregated off into their own play area.

Roll out some sort of third party authentication scheme

Whether it's oAuth, something comparable, or completely RYO, separating out third party API actions from individual users' accounts would go a long way to establishing an accurate accounting of just what's going on within the twitter system.

Create an "open" API sandbox

Currently anyone can create a twitter client and that has been one of the great factors in its growth. I think that should continue with a couple of minor modifications. This sandbox is one of them: basically move the current API setup to a sandbox with a reasonable rate limit for developers. Make it easy for developers to register. But make it a pain in the ass to use as a production API.

Lock down the public API

In parallel with the sandbox, make api.twitter.com a locked down site: you can't go into production without some sort of approval by twitter. Now, this review could be a technology review (how will this new client impact our service?) or a business review (how much will this client cost our service?). But right here, by simply saying Stop! You have to be reviewed. imposes a cost to using the API, and that is a good thing. If you're just screwing around, use the sandbox. If you want to distribute code built on the twitter API, then there's a cost. Whether there's a financial cost or not is up to twitter.

Clean up the API

I'm not going to go into an API-by-API breakdown here, perhaps later, but many of the API functions spit back a lot of redundant data, meaning many more database calls, JOINs, UNIONs, etc. than necessary. Many of the API functions make it much easier to deploy twitter clients, for example the routine to get a list of my followers returns not only my followers, but their profiles and most recent status. All of that information is available through other API calls, but given the current imposition of API limits, I could quickly burn through the current request count simply trying to get the latest status for each follower (I have ~150 followers at present). So this API cleanup would have to be concurrent with rollout of oAuth or something comparable.

Tell Your Critics to Chill

I find some of the venom spewing throughout the blogosphere to be just shocking and disgusting. Twitter's a free service. Various pundits have written about writing a distributed twitter (Hey, let's not just slander them, let's take out what chance they have at a successful business model too!), or claimed that it's some sort of public resource. Just chill. Twitter's problems are solvable. Not necessarily easily solvable, but solvable.

Why this won't happen…

.net culture is fickle. The amount of time it would take to rebuild twitter on–the–fly as I’ve outline isn't very long, maybe weeks if one starts with a clear hand and blank sheet of paper, months under the current firestorm setup. But the various .net glitterati would just as soon shove twitter under the water, than see it succeed.

Imposing a cost to using twitter would have its own cost of course: what's made twitter grow is the API and the complete flexibility to develop applications and mashups against that API. Many of the mashups are only possible when the data and API use are financially free. Many of the clients are only possible when API use is transactionally free (or close enough).

Someone has to pay to use these services. If so many people find them to be useful, why hasn't a successful business model appeared for recouping the cost?

Watching twitter's struggles have been an education for me, not just for the technical issues they face, but also the network effects of the API usage, and the reaction and criticism they've received as they work through the technical issues.

Followup: Kee Hinckley has written up an excellent post to the twitter development group: An odd request for Twitter - Please stop fixing bugs in the API.

Posted in Twitter