An account of why risk management failed

From Saul Hansell in the New York Times:

“There was a willful designing of the systems to measure the risks in a certain way that would not necessarily pick up all the right risks,” said Gregg Berman, the co-head of the risk-management group at RiskMetrics, a software company spun out of JPMorgan. “They wanted to keep their capital base as stable as possible so that the limits they imposed on their trading desks and portfolio managers would be stable.”

One way they did this, Mr. Berman said, was to make sure the computer models looked at several years of trading history instead of just the last few months. The most important models calculate a measure known as Value at Risk — the amount of money you might lose in the worst plausible situation. They try to figure out what that worst case is by looking at how volatile markets have been in the past.

But since the markets were placid for several years (as mortgage bankers busily lent money to anyone with a pulse), the computers were slow to say that risk had increased as defaults started to rise.

It was like a weather forecaster in Houston last weekend talking about the onset of Hurricane Ike by giving the average wind speed for the previous month. [Emphasis added.]

My late father used to say, "To err is human, but to really screw up, you need a computer."

Comments

[NC Reader]

Your late father was definitely right. I love computers and love to analyze data but there is an old saying that is still correct -- GIGO -- garbage in, garbage out. And even if the data is not garbage, if it's incomplete (and it almost always is in some way) you can have an equally disastrous outcome. You sure can get in a heapload of trouble if you don't mix some good old common sense in with what the computer puts out.

[Acad Ronin]

We will always have Type I and Type II errors, false positives and false negatives. You have to have a loss function over both types. I suspect the models often signaled that random fluctuations were changes in trend. This time they didn't catch the change.

[david foster]

It appears that most of the markets didn't consider feedback loops--for instance, if housing prices stall, home buying slows down, which causes further reductions in housing prices, which reduces value of mortgage portfolios, which reduces bank lending capacity...

It is simply invalid to use ordinary statistical methods to analyze risk in systems like this, just as it would be invalid to use probability-of-getting-a-disease numbers in the early stages of an epidemic. The internal causality relationships in the system must be understood and modeled if there's to be any hope of getting it right.

Most people have a very difficult time understanding & controlling even very simple systems if there are time lags or feedback loops involved. See the temperature control simulation, here:
http://photonplaza.blogspot.com/2003_12_28_photonplaza_archive.html#107275628844222087

[david foster]

...meant to say "most of the *models* didn't consider feedback loops," but it's true the other way, too.