The End of the World pt 2: Algorithms

The End of the World pt 2: Algorithms

Let’s start with a list of predictions. No, not the kind involving a crystal ball and a woman called Mme Arcady who claims to be able to see a tall dark stranger in your future. This is a list of practical predictions, the kind we might expect professionals to make in the course of their work.
• predicting the success of electroshock therapy
• predicting criminal recidivism
• diagnosing patients as either neurotic or psychotic
• predicting academic performance in college entrants to medical and law school
• assessing loan and credit risk
• assessing newborns at risk for Sudden Infant Death Syndrome
• predicting the quality of a vintage for red Bordeaux wine (I’m pretty good at this one)
They all might seem very different, but in fact they all have something in common. What?

Here’s the answer. Machines do all these better than we do.

Or to put it another, less eyecatching (but possibly more accurate) way: all of these tasks have been subjected to studies which have shown that an algorithm can outperform a human.

On the face of it, that seems incredible. An algorithm might sound fancy, but it’s really just another name for a set of instructions which lead you step by step from one point in a process to another â€" usually but not necessarily from its beginning to its end. A recipe for chocolate chip cookies is an algorithm. So are the instructions for building an atomic bomb. In the broadest sense of the term, algorithms are a way of storing a process so that it can be repeated reliably.

But the true power of algorithms comes when they are used as a way not to record information, but to process it. We’re all familiar with flow charts and decision trees â€" usually with a binary yes no response at each node. These charts are a basic form of information processing â€" a way to categorise an input, or to change it according to a set of rules and, complicated to a billionth degree, they form the basis of programming languages and ultimately, all computing systems.

And what makes algorithms so useful, is that although we like to think that nothing can beat years of experience and practice when it comes to making complex decision, as the examples above demonstrate, years of experience and practice can actually cloud our judgment. We’re swayed by irrational beliefs, by our temperament, by the influence of others, all factors to which algorithms are immune. In the world of decision making, algorithms are coolheaded Mr Spocks to our impulsive Captain Kirks. Even though it would mean shorter and more boring episodes of Star Trek, if an algorithm had a tricky call to make, instead of transporting the crew into the jaws of a waiting Klingon ship as Kirk is wont to do, it would hit warp drive and hightail the Enterprise to a safe galaxy several light years away.

Which is why algorithms now form the cornerstone of many complex decision making systems. Credit card ratings, automatic triage systems, candidate selection â€" all of these use algorithms. And if their uptake in medical and academic settings is still limited, in the world of finance, especially complex, risky finance like the stock exchange, algorithms rule. The world of financial trading is now so brain bogglingly complicated, that it’s probably safe to say that no human on the planet is now capable of understanding it, let alone predicting what course it will take. Nowadays, deciding when to buy and when to sell, a task which in the past was surrounded with the mystery and superstition more usually associated with betting on horse races, is controlled by algorithms of such power and majesty that they probably have a permanent suite at the Ritz, where they retire to have their nodes rubbed once the day’s trading is over.

And if you are now wondering where the End of the World resides in all of this, here it comes. Because on 6 May 2010, the Dow Jones began to fall. And not just fall, but freefall. In five minutes, the index fell by 1000 points. That’s equivalent to a trillion dollars. During that time, the value of shares in many major listed companies fell to one cent. Yes â€" one cent.

The cause of all this hoo-hah? You guessed it. Algorithms.

The sequence of events is complex, but the post mortem pointed to a single large trade, which triggered a spate of buying. All controlled by algorithms. As was the spate of selling which almost immediately followed. With no Captain Kirk to tell Scotty to kill the engines, a buy-sell loop started, each trade pushing down the share price. Four minutes later and the world’s biggest finance system was three percent poorer.

The share price did bounce back â€" 600 points of the 1000 lost were regained once human hands regained the tiller. But disaster had come perilously close. A full market crash might seem a technical issue â€" we don’t eat stocks and shares after all â€" but looking back on what happened after the crash of 1929, and the Depression which followed and how that led, more or less directly to the rise of National Socialism in Germany, a world war and the deaths of millions, makes the technical suddenly turn very real and threatening. Financial instability causes political instability. Political instability causes wars. The end of the world? Maybe not, but the end of world peace, quite possibly.

So after the flash crash, as it is now known, safeguards were put in place to make sure that it would never happen again.

We are now safe, say the experts. Nothing to see, people. Keep calm. Carry on.

Except since the flash crash, similar if smaller dips in the market have taken place not once, but at least three times.

One problem is speed. These fluctuations occur over tiny periods of time: the Flash Crash at six seconds was stately in comparison to the one second it took for the sugar market to drop 6% in February 2011. The very fastest a human can react is 140 milliseconds. An algorithm can react in 1 millisecond. By the time Kirk is reaching for the stop button, his machine equivalent Spock can make 140 bad decisions.

Another problem is greed. Algorithms are not only being used to make the trading calls, as Donald Mackenzie explains in this article for the London Review of Books they are now being developed for much blacker purposes. The new generation of algorithms can sniff out the signatures of other algorithms and predict what they are going to buy and when, making their owners money on the back of that information. Some can even fake behavior which will trick other algorithms into making sales or purchases, again allowing their human owners to profit. Such activity is illegal, but difficult to prosecute. With over 50% of transactions now carried out by automatic systems, aren’t we heading for a scenario where packs of roving algorithms hunt each other down, in a virtual dog eat dog world, eventually destroying the financial system they were designed to exploit?

Not necessarily, says Mackenzie. Volatility, the key measure of how stable the market is, has not increased in pace with the use of algorithms. If anything the converse is true â€" the data set is small but analysis seems to indicate that as computer systems begin to take over the decision making process, the system begins to become more stable than those run by human hands. Immune to the emotional factors which cloud human judgment, algorithms seem to be better equipped to avoid making the same mistakes twice.

So is the problem solved? Can we say with confidence that the end of the world will not be brought about by algorithms careening out of control, destroying our economy and sparking a final conflict?

As much as we can say anything with confidence, the answer is probably yes. Algorithms may occasionally behave in ways we don’t expect, but as their designers, we’re in a perfect position to ring them with safeguards which will make sure those unforeseen effects can be stopped before they get out of control. Remove the human element and the story might be different.

But so far no one has attempted that. The Starship Enterprise succeeds in its missions through a mixture of human emotion and Vulcan rationality. It’s a partnership which appears to work in real life too.