Footballmetrics

In less than a week, another season of professional (American) football will begin. Football fans (me included) -- especially devotees of 'fantasy football' -- are no doubt pouring over the statistics of their favorite players. Ironically, from a statistical standpoint, this devotion to numbers may be misleading.

For example, writing in the 'Keeping Score' column in the New York Times Sports section(Numbers Often Lie When It Comes to Football, Aug. 27, 2006), economist Martin B. Schmidt argues that individual stats are so interlinked with factors beyond the inherent athletic prowess of the individual player (e.g., the individual stats are also a function of team effort, good or bad coaching, the quality of the opponents, etc.) that they are highly misleading. A good example of this is the high variance in a quaterback's passer ratings from season to season:

How much of a quarterback’s rating is predictable? Not much. If we look at six seasons of quarterbacks who threw at least 224 passes in successive seasons — the minimum to be ranked in the N.F.L.’s quarterback rating — a past rating has a poor predictive value.

Take Peyton Manning. He produced a quarterback rating of 104.1 last season. This season, based on the six seasons of ratings, there is a 95 percent chance that his rating will be between 73 and 111. In other words, he is going to be either pretty bad or pretty good.

Some innovative thinkers have tried to come up with alternatives to the currently popular statistics used by football fans, sports broadcasters, and the teams themselves. Football Outsiders is a website devoted to applying Moneyball style statistical analysis and thinking to football (Aaron Schatz, the founder of Football Outsiders, also writes a book of football statistics and analysis called Pro Football Prospectus 2006: Statistics, Analysis, and Insight for the Information Age.)

My opinion on this is that, at the very least, improved football (and sports) statistics must incorporate team effects as opposed to crediting (or blaming) the individual for too much of the success (or lack of success).

BTW, although I avoid making predictions about the stock market or the economy on this blog, I will make a prediction about a far more important issue ... football. My guess is that the Miami Dolphins will do relatively well this season (at least a playoff appearance). I should note that I am NOT a Miami Dolphin fan nor have I any ties to that region. Why am I making this bold prediction? Nick Saban, the coach of the Dolphins, is a disciple of Bill Belichick, the highly successful coach of the New England Patriots (Belichick, an economics major in college, is a supporter of applying quantitative thinking in football), and Saban will be into his second year at Miami. The Dolphins also have a new quarterback, Daunter Culpepper, who brings a lot to the table in terms of his skills as a passer and on-the-field leader of the offense. Add all of that to a defense that has been consistently good even during the lean years, and it makes for a compelling case to 'go long' (so to speak) on the Dolphins.

So if the Dolphins do well this year, you heard it here first!

PS: October 24, 2006 -- I don't know if dolphins eat crow, but I am over picking them to have success this season!!! ;)

Grigori Perelman - The Tolstoy of Math?

I thought some more about the Grigori "Grisha" Perelman situation since writing about it in a previous blog post: Waiting for Grisha: What REALLY Happened to The Man Who Proved the Poincare Conjecture (Aug. 24, 2006). As a reminder, Dr. Perelman seems to have succeeded in proving the Poincare Conjecture as well as the more general Thurston Geometrization Conjecture (both conjectures are from topology and relate to the idea that geometric structures can be deformed into some basic 'shapes' identified by mathematicians).

His refusal to accept the Fields Medal has been attributed -- unfortunately in my opinion -- to some sort of eccentricity on the part of Dr. Perelman. As I wrote in the blog post, I feel that (and this is backed up by some evidence from Sylvia Nasar and David Gruber's New Yorker article, Manifold Destiny) the more plausible reason behind Dr. Perelman's declining the honor had to do more with his desire to uphold his high moral, ethical, and intellectual standards rather than to the less than flattering reasons offered up by most of the press and by (again, unfortunately) some mathematicians. As Dr. Perelman himself so eloquently put it (in the New Yorker article): “'It is not people who break ethical standards who are regarded as aliens,' he said. 'It is people like me who are isolated.'”

All of this reminded me of another Russian who possessed both a great intellect and a great sense of moral integrity: Leo Tolstoy. Like Dr. Perelman, Tolstoy -- after giving the world some of the best novels ever written (War and Peace, Anna Karenina) -- retired to an ascetic/quasi-monastic life motivated by a deep sense of personal morality and ethical integrity.

So it is my sincere hope that Grisha Perelman will not be seen as some sort of 'nutty professor' -- has he has been unfairly portrayed by many in the press and by some mathematicians -- but will, instead, be seen as the 'Tolstoy of math.'

Macroeconomic Derivatives

Another interesting article from the NBER Digest: Macroeconomic Derivatives. Justin Wolfers, of Wharton, and another economist studied the effectiveness of markets to purchase binary/digital options (so-called because you either get the payoff if the event happens or you don't if the event doesn't occur) in predicting macroeconomic events. Their findings were that forecasts based on these markets were at least as good as (and often superior to) surveys of forecasters/economists. A particularly interesting aspect of the research was that it might be possible to construct a probability distribution -- what they call a "density forecasts" -- using the market-based data. Creating actual probability distribution from economic/financial data is very hard to do for a number of reasons, so it is remarkable that one can use these binary options to create such "density forecasts."

A copy of the paper can be found here.

Investors Without Borders

Mark Hulbert's New York Times column in Sunday's New York Times (Aug. 27, 2006) dealt with the increasing tendency for investors to invest internationally: Investors Without Borders.

Applying Behavioral Economics to Consumer Finance

This month's issue of the NBER Digest had an article on how a group of economists carried out a field experiment in behavioral economics (which uses tools from psychology to expand upon traditional notions of rationality in economics): A Field Experiment in the Consumer Credit Market. The academics -- from the University of Chicago, Harvard, Yale, Princeton, and Dartmouth -- used psychological factors in marketing consumer loans in South Africa. The researchers found that "a firm can exploit consumers' psychological biases, thereby increasing demand without lowering prices." What was most impressive about their field experiment was that their methods allowed the lender to create demand for loans equivalent to dropping the interest rate by 1 to 4 % without having to actually drop the interest rate!

A preliminary draft of the research paper can be found here.

Malcolm Gladwell on Risk Pools

Malcolm Gladwell, who I consider to be the finest non-fiction writer out there, has a new article in the New Yorker dealing with the economics of pensions: The Risk Pool: What’s behind Ireland’s economic miracle—and G.M.’s financial crisis? In the article, he focuses in on dependency ratios, demographics, and the pooling of risk (financial, health, etc.).

Waiting for Grisha: What REALLY Happened to The Man Who Proved the Poincaré Conjecture

There was a minor scandal in the mathematics community this week when Grigori "Grisha" Perelman -- a Russian mathematician that is widely credited with having proved the Poincaré Conjecture (one of the $1 million Millennium Prize Problems offered by the Clay Mathematics Institute) -- refused to accept the prestigious Fields Medal (in the math world, the Fields Medal is considered to be on par with the Nobel Prize).

The Poincaré Conjecture has been an important unsolved problem in topology. Topology is the branch of mathematics that deals with what can be loosely termed 'fundamental shapes' -- spatial properties that are perserved even after deformation. For example, to a topologist, a coffee mug is topologically equivalent to a doughnut -- they both have one hole. Thus, the 'fundamental shape' of both a coffee mug and a doughnut is a torus.

Dr. Perelman used a technique called a Ricci flow (which smooths out the 'bumps' of geometric space to reveal it's 'fundamental shape') in order to prove the conjecture. Other mathematicians had tried to use Ricci flow in order to tackle the conjecture, but they stumbled over singularities whose properties were difficult to predict and manage. Grisha Perelman overcame this obstacle by proving that these singularities could be 'managed' (so to speak) and the Ricci flow allowed to continue to its logical conclusion, thus solving Poincaré's age old conundrum. (For those interested in the gory technical details, you can find Dr. Perelman's papers -- as well as related material -- at Notes and commentary on Perelman's Ricci flow papers.)

Most media reports of Dr. Perelman's refusal to accept the Fields Medal has chalked it up to his alleged 'eccentricities' (some have even gone as far as using the word "crazy"). Some examples in this vein include stories in the New York Times, The Guardian (UK), and The New Scientist.

From a journalistic/editorial point of view, this is all suppose to be reminiscent of the brilliant but troubled mathematician John Nash (of A Beautiful Mind fame (BTW, the real John Nash doesn't look like Russell Crowe) ) -- the inventor of Nash equilibrium in game theory. I suppose that type of analogy has some superficial appeal for media types, the public at large, and, apparently, even professional mathematicians.

Despite the general tenor of how the majority of the press (and some mathematicians) are portraying Dr. Perelman, I'm a little skeptical of this 'kooky math wizzard hiding out in the Russian woods' tag people have tried to place on Dr. Perelman. My suspicions are confirmed by an excellent article in the New Yorker magazine, Manifold Destiny: A legendary problem and the battle over who solved it (Aug. 28, 2006) by Sylvia Nasar (the author of A Beautiful Mind) and David Gruber.

According to the New Yorker article, several prominent mathematicians have unfairly tried to minimize Grisha Perelman's contributions in proving the Poincaré Conjecture (to be fair, others have made important contributions leading to the proof, but, as I believe Sir Issac Newton said, scientists and mathematicians always "stand on the shoulders of giants" and Dr. Perelman -- although standing on others' shoulders (which he generously acknowledges) -- saw what others did or could not). I strongly believe that this type of situation (not specifically this incident since Dr. Perelman seems to be a genuinely humble and shy man who would rather defer to others) -- the perception of unfairness and, even, injustice in what is suppose to be a pure and noble intellectual pursuit -- is the real reason behind Dr. Perelman's refusal to accept the Fields Medal.

The following passage from the New Yorker article seems to back up my conjecture:

Perelman repeatedly said that he had retired from the mathematics community and no longer considered himself a professional mathematician. He mentioned a dispute that he had had years earlier with a collaborator over how to credit the author of a particular proof, and said that he was dismayed by the discipline’s lax ethics. “It is not people who break ethical standards who are regarded as aliens,” he said. “It is people like me who are isolated.” We asked him whether he had read Cao and Zhu’s paper. “It is not clear to me what new contribution did they make,” he said. “Apparently, Zhu did not quite understand the argument and reworked it.” As for Yau, Perelman said, “I can’t say I’m outraged. Other people do worse. Of course, there are many mathematicians who are more or less honest. But almost all of them are conformists. They are more or less honest, but they tolerate those who are not honest.”

The prospect of being awarded a Fields Medal had forced him to make a complete break with his profession. “As long as I was not conspicuous, I had a choice,” Perelman explained. “Either to make some ugly thing”—a fuss about the math community’s lack of integrity—“or, if I didn’t do this kind of thing, to be treated as a pet. Now, when I become a very conspicuous person, I cannot stay a pet and say nothing. That is why I had to quit.” We asked Perelman whether, by refusing the Fields and withdrawing from his profession, he was eliminating any possibility of influencing the discipline. “I am not a politician!” he replied, angrily. Perelman would not say whether his objection to awards extended to the Clay Institute’s million-dollar prize. “I’m not going to decide whether to accept the prize until it is offered,” he said.

Mikhail Gromov, the Russian geometer, said that he understood Perelman’s logic: “To do great work, you have to have a pure mind. You can think only about the mathematics. Everything else is human weakness. Accepting prizes is showing weakness.” Others might view Perelman’s refusal to accept a Fields as arrogant, Gromov said, but his principles are admirable. “The ideal scientist does science and cares about nothing else,” he said. “He wants to live this ideal. Now, I don’t think he really lives on this ideal plane. But he wants to.”

(Emphasis added.)

That's my two cents on the matter. Frankly -- although I wish Dr. Perelman will eventually accept the accolades (including the Millennium Prize and the Fields Medal) that he so richly deserves -- I have to say that I admire his integrity and sincerity in trying to stand up for and protect the purity of mathematics. The Mandarin Chinese word for 'sincere' is 'zhenxin' -- it literally means 'pure heart' or 'genuine heart.' Dr. Perelman, from the bottom of my heart, I want to let you know that you have both a pure mind and zhenxin, a pure heart.

What Options Trading Can Tell Us About Stock Prices

Professors Jun Pan, of MIT, and Allen Poteshman, of the University of Illinois at Urbana-Champaign, conducted research into how much information about the expectation of future stock prices are encoded in the trading behavior of options traders. In their paper, The Information in Option Volume for Future Stock Prices (which will be published in an upcoming issue of the The Review of Financial Studies), the professors conclude that there is strong evidence for option trading volumes containing substantial information about future stock prices.

The gist of this paper is that stocks with a low put to call ratio -- based on trading volume (adjusted to focus in on new trading positions as opposed to re-adjustments of existing positions and/or market making transactions) -- tended to outperform those with high ratios. Why? A low put to call trading volume ratio (suitably adjusted) for a stock means that option traders expect that the price will rise in the future since a low ratio means a greater interest in calls (which is a long position). A high ratio for a stock would mean that option traders expect the stock price to drop since a high ratio means a greater interest in puts (reflecting a short position).

A hypothetical portfolio using these relationships was found to consistently produce double digit returns even during bear markets. This phenomenal performance does require frequent trading (the researchers hypothetically re-adjusted their portfolio weekly). Frequent trading would entail high transaction costs which would dampen performance. However, the returns were still fairly high (at least for institutional investors who get favorable transaction fees). Without taking into account transaction costs, the annualized return was 62% over the 12 years studied (1990 to 2001); with transaction costs (for institutional investors with low costs) the return would have been around 50% annually.

In order to arrive at these results, Professors Pan and Poteshman had to use a proprietary database from the Chicago Board Options Exchange. Without this unique dataset, it would have been extremely difficult to focus in on trading volume that reflected trading information that would have been relevant to their thesis. This is an example of the importance of having proper and useful data in econometrics and statistical analysis.

By the way, this private data on trading volume is now available to investors. Thus, what was once semi-private information will now become public information. A lot of the informational advantages found in the research will probably be aribitraged away. The degree to which the markets are efficient are debatable, but the markets should be efficient enough to trim the informational advantages so that the fantastic returns from the hypothetical portfolio in the paper will be reduced considerably.

Soros Has Lunch with the FT

This weekend's Financial Times has an interview with hedge fund legend George Soros (aka "the man who broke the Bank of England") in its 'Arts & Weekend' section (August 5, 2006): The billion-dollar memory lapse. Soros talks about his philantrophic work, his life story, and Karl Popper -- the famous philsopher who has served as a major intellectual influence for Soros ever since his student days at the London School of Economics (where Popper taught for many years). Soros avoided talking too much about his successes in investing ... even playfully feigning amnesia about "Black Wednesday" ... although he suggests his theory of "reflexivity" -- the interaction between misconception and reality -- somehow led to his great fortune.

The FT article had the following tidbits about Soros' days as a trader (as opposed to his status as a "stateless statesman"):

Soros, however, is in a different league - so engagingly Olympian that he affects absent-mindedness about his greatest coup - Black Wednesday, September 16 1992. That day, when Soros successfully bet the British pound would be devalued, broke John Major's government, led to the election of the Labour party and entered political folklore as an unforgettable date. But not, it seems, to the man who made the most money out of it.

"Was it Wednesday?" he asks. "It was Thursday, I think."

"Wednesday," I confirm. "Definitely Wednesday."

"Was it?" he asks again, seeming to distance himself from his former self.

....

But it is for making money that he is best known. Stories abound about Soros the financier - about how he lost millions in Russia and Japan, about how he always wanted to raise the stakes. His $1bn profit on Black Wednesday, for example, came because he had bet $10bn. Investing also brought him pain - because of his fear of losing what he had risked. Despite all of Soros's talk of reflexivity, he admits he often sensed trouble with his investments because of an ache in his back.

Evolutionary Economics & Adaptive Market Hypothesis

In the July 20th issue of The Economist, its 'Economics Focus' column had an article dealing with evolutionary economics: The Cambrian Age of Economics. Evolutionary economists seek to look at problems in economics using methodology adapted from evolutionary biology (especially the more mathematically inclined branches of biology). Although some economists and biologists might snicker at the marriage of economics and biology, it should be noted that both fields have drawn inspiration from each other. For example, John Maynard Smith, FRS (not to be confused with John Maynard Keynes) -- an important figure in mathematical biology -- used game theory to develop the kind of evolutionary biology popularized by Richard Dawkins' The Selfish Gene.

Reading the article reminded me of Andrew Lo, a finance professor at MIT, and his 'adaptive market hypothesis' (his Journal of Portfolio Management article on the subject can be downloaded here; a less technical overview of the adaptive market hypothesis can be read from this article in Business Week). Basically, the adaptive market hypothesis is an alternative to the efficient market hypothesis, which underlies much of academic and (even) quantitative finance. Unlike the efficient market hypothesis, the adaptive market hypothesis incorporates evolutionary models of human behavior in order to better explain and account for what we actually see empirically from financial market data.