The
power law -- sometimes referred to as the
Pareto distribution,
Zipf's law, or the
80-20 rule -- has drawn a great deal of attention lately as an alternative to the 'normal' (Gaussian) distribution (i.e, the bell curve). The power law has gained in popularity among more numerate intellectuals, policy makers, and business people because it seems to fit better with common sense than what we were told in Statistics 101: Extreme and rare events have a greater than expected impact; a few products, people, and websites seem to have the bulk of market share, wealth, and mindshare; etc.
As a descriptive model of the way many things work, I have and will extol the virtues of the power law. As a purely descriptive framework, I will even go so far as to say that the power law is so ubiquitous that the so-called 'normal' distribution may not deserve that moniker.
However, as a normative or
prescriptive approach to how things
should work, I believe that the gaining popularity of the power law holds some dangers as well as benefits. After all, should we find it acceptable -- even if it is an accurate description of how things really are -- that the top 20% of people control 80% of the wealth in a society? Is it really a good idea that a few 'hits' (often ginned up to be that way rather than by popular demand) swamp out more meritorious 'misses'? Is it really ultimately beneficial, efficient, or rational for a society to have a few people at the top of the power law -- at least those who got there by inherited privilege and/or genetic accident -- hold sway over the lives of the many who live down along the 'long tail' of the power law?
In this blog essay -- which was originally intended to be a follow up to
Netflix and the Sandpile -- I will argue that the power law, if misapplied so that it becomes a model of the way things 'ought to be,' is fraught with dangers. This sort of misapplication of the power law could lead to what I call the "tyranny of the power law." On the other hand, properly applying the power law, by using it as a mostly descriptive model rather than as a platonic ideal, can help in guarding against catastrophic risks as well as assisting those who are striving for a more democratic and meritocratic society.
An Intuitive Introduction to the Power LawIn the bell curve view of the world, the most likely -- i.e., the 'expectation' in mathematical statistics speak -- place you will find something is close to the average. With the power law, however, the concept of an 'average' (or 'expected value') is essentially irrelevant because substantially more (relative to the Gaussian) of the distribution of observations or events is located at the extremes.
Think of it as Shaquille O'Neal versus Bill Gates. Shaq, in our example, represents the distribution of human heights -- which follows the normal distribution; Bill Gates represents the distribution of human wealth -- which follows the power law.
Everyone can agree that Shaq is at the extreme (both in terms of magnitude and probability) of the distribution of human heights. Yet, Shaq is only about a foot (and a bit more) taller than the average American male. It is almost prohibitively unlikely for an eight foot tall human being to exist and impossible to have a ten foot tall human. Since even the 'extremes' of human heights are within a relatively tight range of the average (or mathematical expectation), the concept of an average is important in describing the distribution of human heights and most heights will tend to be bunched around more-or-less symmetrically around the average (hence, the bell curve shape).
The distribution of human wealth, however, behaves differently. The 'average' American makes somewhere around $45,000 to $50,000 a year. Bill Gates makes several thousands times that a year. Bill Gates' wealth, in terms of his Microsoft shareholdings alone, is in the
mid $50 billion range. The 'extremeness' (in terms of magnitude) of Bill Gates' extreme wealth essentially makes the notion of an 'average' wealth or income much less useful in describing the distributions. Having a Bill Gates in the wealth distribution is like having a ten foot tall human being; it just can't happen if the distribution of wealth was Gaussian.
Wealth is among the many phenomena -- social and natural -- that seem to obey the power law. In nature, the number of earthquakes, the behavior of granular piles, population dynamics in some ecological models, the number and magnitude of solar flares, the x-ray signature of black holes, the observed distribution of digits from 1 to 9 (i.e.,
Benford's Law), and the impact of forest fires, are a few examples of the many natural phenomena that conform to the power law distribution.
Even more examples of power laws are being found in the social sciences. As I have already mentioned, the 80-20 rule (first formulated by the Italian economist, Vilfredo Pareto) applies to the distribution of wealth -- the top 20% of the population holds 80% of the wealth in a society. According to economist,
Arthur De Vany, the 80-20 rule also applies to major motion pictures -- about 20% of the films will grab about 80% of the market share (the research is contained in his book
Hollywood Economics). The same principle can be applied to many other products and services.
When it comes to admissions at Ivy League and quasi-Ivy League universities, the principle can be called the 75-25 rule: nearly three quarters of the undergraduate students at Harvard and other so-called "most selective" universities and colleges come from the top income quartile (see next section for references).
The power law, where the extreme and the rare have a disproportionate impact, also applies to financial markets. As repeatedly pointed out by Benoit Mandelbrot and Nassim Nicholas Taleb, "just
ten trading days represent 63 percent of the returns of the past 50 years" (emphasis added) (from
A Focus on the Exceptions that Prove the Rule, Financial Times supplement on
Mastering Uncertainty, 2006). The power law also applies to concentration of market capitalization to the shares of a small number of publicly traded companies (e.g., the 80-20 rule approximately applies to the market value of the FTSE 100 compared to all shares traded in London), the size of hedge funds, and a number of other areas in finance.
Can Power Laws Corrupt?: The Power Law versus MeritocracyLord Acton has been credited with coining the phrase "power corrupts, and absolute power corrupts absolutely." In light of the fact that the existence of the power law across so many social phenomena often means that social, commercial, and political power (as opposed to mathematical 'power,' which is where the power law gets its name) is concentrated among a small segment of people, institutions, etc., can the misapplication of the power law -- as a platonic ideal rather than as a tool of empirical science -- lead to a corrupt and less meritocratic society? Unfortunately, the answer seems to be yes.
Take culture for example. Because of the 80-20 power law principle, movie studios, tv networks, record companies, and even book publishers tend to promote what they hope will be hits at the expense of other products and services. Although hits can be created because of popular demand, these media companies often try to artificially create hits whether the public truly wants them or not.
This type of hit-based culture would not be so bad if it did not hurt higher quality fare. Unfortunately, however, the hit-driven marketing and business models used by media companies often crowd out or even prevent more meritocratic movies, tv shows, music, and books from reaching potential audiences. For example, in the music industry, instances of 'Payola' have surfaced from time to time. Even without overt bribery, the hit-driven mentality of major music labels and the various outlets for music (major radio networks, etc.) often prevent higher quality 'indie' offerings from getting heard and noticed.
Another egregious example is the movie industry. It is quite routine for many Oscar nominated films to have never been shown in many markets. Even in markets, like Los Angeles and New York, that have outlets for art house films, many higher quality movies are relegated to a few screens located at relatively obscure locations. Documentaries, until very recently, have rarely been available to the majority of movie audiences due in part to theater owners needing to cater to major studios and their hit fictional films. Needless to say, most of the top 20% of films are often of poor quality.
A much more troubling aspect of the potential misuse of the power law is in the unhealthy concentration of political and social power. Not only is it the case that the wealthiest quartile of Americans tend to make up the vast majority of the student population at so-called 'elite' universities, at these universities "only three percent of students come from the bottom income quartile and only 10 percent come from the bottom half of the income scale" (the quote comes from
the Harvard Gazette, which is the official newsletter of the university's administration, and is based on various research studies studying the makeup of various Ivy League and quasi-Ivy League universities). Anyone who is seriously and sincerely interested in a fair and meritocratic society should be appalled by these statistics.
This type of concentration of wealth at institutions that people hope would be meritocratic -- along with the 80-20 rule of wealth in society -- bodes ill for socio-economic mobility (aka, 'the American Dream'). Nobel laureate and University of Chicago econometrician,
James Heckman, has warned that "the big finding in recent years is that the notion of America being a highly mobile society isn't as true as it used to be" (from Aaron Bernstein,
Waking Up From the American Dream, Business Week, December 1, 2003). This combined with programs to help the already privileged -- see, e.g.,
Lexington: The curse of nepotism -- A helping hand for those who least need it (The Economist, January 10, 2004) -- add weight to Princeton economist, Paul Krugman's, comment that "the underlying economic, social, and political trends will give the children of today's wealthy a huge advantage over those who
choose the wrong parents" (emphasis added) (New York Times,
The Sons Also Rise, November 22, 2002).
Education has traditionally been seen as a vehicle for socio-economic mobility. If the most highly esteemed universities -- schools that are often seen (rightly or wrongly) as the breeding ground for the 'power elite' -- have in place anti-meritocratic barriers (see, e.g.,
Malcom Gladwell's excellent article in The New Yorker on Ivy League admissions,
Getting In: The social logic of Ivy League admissions) that unfairly reinforce the current power law dynamics, then social, political, and economic power that is already concentrated in the hands of a privileged few will be further concentrated based -- not on the democratic ideas of the Founding Fathers (the so-called "natural aristocracy") -- but on the plutocratic idea that those who inherited privilege by paternal accident should be 'protected' from having to fairly compete with those from more humble backgrounds.
If the power law is misapplied in this way -- so as to reinforce unmeritocratic privilege -- then society could suffer dire consequences. We will have leaders -- in what are suppose to be democratic societies -- that will be out of touch with their people. Rather than being public servants, politicians and policy makers will become callous and self-serving. Entrepreneurs that are from the 'long tail' rather than at the top of the power law may find themselves unable to find suitable business opportunities regardless of the merit and objective worth of their ideas. Otherwise intelligent and deserving people may be shut out of educational opportunities largely because they 'chose' the wrong parents.
More insidious than even those worrisome scenarios is the possibility that those who have the most power and influence in society will have too much in common with each other and too little in common with the masses down the long tail. This sort of cookie-cutter conformity will lead to a tacit lack of diversity and may even create an atmosphere of thinly disguised discrimination and bigotry.
The elites will suffer from 'groupthink' -- locked into a feedback loop where they will mostly look out for their own interests and reject counsel from 'outsiders' (e.g., good people of modest means, those with unorthodox but interesting perspectives, etc.). Our political, economic, and societal 'leaders' will be devoid of creativity and a sense of fairness and justice. Rather than solving problems -- whether they be in matters of war & peace, domestic policy, the economy, business management, or academia -- the 'power elite' will simply force us to accept mediocre to terrible ideas and throw up barriers to those who could offer up more creative and fair solutions. Just as the hit-driven popular culture leads to bland or even awful entertainment, a system of giving or denying socio-economic and educational opportunities based on plutocratic criteria will lead to unimaginative and foolish approaches to making policies, running businesses, and advancing knowledge.
Returning to our original question, can misapplying the power law corrupt? Absolutely.
The Right Way to Apply the Power Law -- Becoming EclecticSo how can an awareness of the power law further rather than hinder progress in society? To find the answer, I have to return to the
original inspiration for this series of posts: Netflix.
According to
David Leonhardt's New York Times article on Netflix -- instead of having a power law for on-line DVD rentals that is similar to the power law at work with theatrical releases -- the distribution of DVD rentals on Netflix is actually closer to the relatively tight bell curve of the normal distribution:
Out of the 60,000 titles in Netflix's inventory, I ask, how many do you think are rented at least once on a typical day?
The most common answers have been around 1,000, which sounds reasonable enough. Americans tend to flock to the same small group of movies, just as they flock to the same candy bars and cars, right?
Well, the actual answer is 35,000 to 40,000. That's right: every day, almost two of every three movies ever put onto DVD are rented by a Netflix customer.
(Having about 66%, "two of every three," of the movies being rented, rather than 10, 20, or 25%, would be consistent with the familiar bell curve of the Gaussian distribution.)
This is a paradox. Why would only 20% of movies find a commercially meaningful audience when they are released in theatres, but 66% of movies find an audience when distributed via Netflix? David Leonhardt's article offers up an important clue to resolving this paradox:
"Americans' tastes are really broad," says Reed Hastings, Netflix's chief executive. So, while the studios spend their energy promoting bland blockbusters aimed at everyone, Netflix has been catering to what people really want and helping to keep Hollywood profitable in the process.
The solution to this paradox is simple: becoming eclectic. As it turns out, people have broad tastes and interests that extend far beyond the manipulative strictures imposed by media giants. People value diversity; they're eclectic.
On a profound level, Netflix -- in contrast to pre-existing distribution channels (including other DVD rental outlets like Blockbuster) --
became as eclectic as the audience. Either by serendipity or by epiphany, Netflix has found a way to offer people what they
really, sincerely want. Netflix is one business model where an eclectic supply is meeting eclectic demand.
Reflecting on what we've discussed thus far, I would argue that the correct lessons to learn from the power law are the following:
(1) The power law is often an empirical fact (the way things are) but not necessarily the platonic idea (the way things ought to be); and
(2) We can tame or mitigate at least some of the negative aspects of the power law by encouraging eclectic and diverse strategies.
Chris Anderson -- in his new book
The Long Tail, his
Long Tail blog, and his original
Wired magazine article -- seems to favor this correct application of the power law. Chris Anderson's conception of "the
long tail" turns the logic of the power law (mis-applied) on its head: Rather than catering solely to the concentrated extremes of the power law (the biggest hits, the wealthiest people, etc.), the logic of the long tail calls on companies and institutions to address needs and desires all along the long tail of the power law. To draw on a baseball analogy (reminiscent of Michael Lewis'
Moneyball), rather than
just focusing in on home runs, a wise manager will also try to get his players on base via doubles, singles, and, even, drawing walks.
Another example of the long tail that comes to mind is in the world of pop music. Most music radio stations are owned by major media companies. They tend to preprogram their music and give little or no control to DJs (that is if they even employ them in some markets) in the selection of music. Not surprisingly, most (if not all) of the airplay is devoted to hits. It's as if these radio networks, and the record companies that influence decision making, are trying to only go for home runs.
Fortunately, there are some wonderful exceptions to this general trend. For example,
KCRW -- a National Public Radio station in the Los Angeles area -- has championed reaching out to the varied tastes of music-loving audiences by playing music by independent and lesser known artists. The flagship music program on KCRW,
Morning Becomes Eclectic (as well as its weekly syndicated program,
Sounds Eclectic), has even helped to launch the careers of once fledgling musicians -- but now bona fide stars -- like Coldplay.
Can this eclecticism be applied to other arenas as well? I believe it can.
For instance, we can achieve a fairer and more meritocratic society by opening up legitimate opportunities for socio-economic advancement all along the long tail. Rather than "only 10 percent" of the students at so-called 'elite' universities coming "from the bottom half of the income scale" (i.e., where only 10% of the student body at prestigious universities reflect the working class and the poor in America), a larger proportion of the places at these schools should be opened to the talented but disadvantaged. Not only will this lead to a truly democratic society, it will also give us a more eclectic and imaginative set of leaders in policy making, business, and in academia. Having people in positions of power and influence from a more diverse set of circumstances than what we currently have will benefit all of us by opening up the possibility of more optimal decision making compared to what we can get from plutocratic groupthink or plutocratic autocracy.
Diversity and eclecticism -- the right lessons to learn from the power law -- can also be applied to finance and investing. It is widely recognized that diversification in an investment portfolio is important to managing and controlling risk. This type of risk management strategy is usually based on Gaussian thinking where only linear and static correlations between the valuation of assets are taken into account with the hope that there will be offsettinging set of simultaneous gains and losses. In other words, risk management is often done with the tools of "mild randomness" (in the terminology of
Benoit Mandelbrot and Nassim Nicholas Taleb).
As it turns out, this kind of diversification is not nearly diverse enough. Because markets reflect the power law -- a world of "wild randomness" -- non-linear and dynamic co-movements between the prices of assets have to be taken into account. In this wilder power law world, what we think will offset losses using the mindset of the normal distribution may actually exacerbate losses. Therefore, a much more eclectic risk management strategy -- e.g., investing in as great a number of offbeat asset classes as possible -- will be a better hedge against catastrophic risk than can be achieved by ignoring the correct lessons of the power law.
For example, a recent Buttonwood column,
Sting in the Tail (July 11, 2006), on the website of
The Economist magazine, has argued that contrarian investment strategies focusing in on small and mid cap stocks is a manifestation of long tail logic being applied to equity investing. Applying this long tail investment strategy allowed some investors to weather, and even profit from, recent stormy periods in the market like the 2000-2003 bear market.
Another way in which the correct lessons from the power law -- such as a long tail strategy -- might be fruitfully applied is in the timing of investments and trades. As I discussed extensively in my blog post,
'Mind Time' & Market Time, the different ways in which traders and investors perceive and react to time can have a large impact on market returns and risk as well as contribute to the power law characteristics of market returns. Perhaps, by acting more calmly in times of market turbulence and being more active in times of relative tranquility (thus evening out, somewhat, the disjointed behavior of investors and traders across time), some of the wild randomness brought on by the 'mind time' phenomenon can be reduced.
The lesson to take away from the long tail and the power law for investors is that they -- like the movie and music businesses -- would be wise to become eclectic.
Overcoming the Tyranny of the Power LawTo reiterate, I strongly believe that the power law is a useful paradigm and tool for studying the many natural and social phenomenon that are consistent with that model. As a descriptive ('positive' in philosophy of science speak) framework, the power law is often superior to the 'normal' distribution and should be adopted and utilized more widely.
From a normative, prescriptive perspective, however, the logic of the power law could easily be abused (just as the 'bell curve' has been). To any person of both intelligence and conscience, it is not a laudable nor a desirable thing to have 20%, 25%, or some other percentage of small concentrated elites or 'hits' crowd out higher quality competition and, even, endanger meritocracy and democracy. The catastrophic financial, economic, and other risks implied in a power law world of "wild randomness" -- as discussed elsewhere by this blog and by others -- is also less than desirable. (Reflecting on the Netflix example, one is tempted to go so far as to say that -- from a purely normative standpoint -- a world of the normal, Gaussian distribution -- a framework that underlies the utopic world of efficient markets -- may be more desirable than the reality of a world with power laws.)
We can reduce some of the harmful effects of the power law by using a long tail strategy of encouraging eclecticism and accommodating niches. The power law can then become a useful and desirable normative framework to tackle problems in business, finance, policy making, and culture. We can overcome the tyranny of the power law by what 'indie' music radio station,
KCRW, has encouraged its listeners to do ... become eclectic.
