August 17, 2010

Don't Trust Pretty Graphs

Numbers may lie, but so do pretty looking graphs. A good friend shared this image and its accompanying post from Fast Company's Co.Design blog. It suggested that the US is already very far behind the world in terms of innovation efficiency. (Just look how minuscule our teeny little pink box is.) The article is called: "Does the US Waste Money on Innovation?"

Now this struck me, because one of the staples of the American culture and economy is its excellence in innovation. And true, while some headlines do talk about how a greater percentage of innovative companies are starting abroad, most of the top 10 or top 25 most innovative companies are American. Bloomberg, Fortune, etc. all have lists that show this.

The article measures innovation efficiency by a simple ratio: number of patents granted to amount of money invested in R&D. The higher the number, they say, the more effective. This is the first problem. Consider an industry where a lot of R&D expenses are used on a single product that generates a ton of revenue, but files only a few patents. This undermines the ratio they use, because the ratio doesn't take into account the revenue generated by the patents, only the total number of patents.

If a country filed millions of patents on different kinds of unprofitable paperclips and spent only 2 dollars on R&D, it'd be higher than a company that filed one patent on Google's profitable algorithm but spent millions on R&D.

For example, they compare biotech vs. consumer electronics. A similar amount invested into biotech yields only a few patents versus CE, which yields hundreds of patents. So even if biotech and CE returned the same profits for the exact same amount of research, the ratio they use would make a country that favored biotech seem less efficient.

Efficiency should be about how much revenue each dollar of research yields - measuring quality of patents not the quantity of patents. So the biotech/CE example they bring up undermines the very metric the article is based on.

So, while I recognize that it's a tall task to trace revenue data back to patents, that's the only way to find any sort of meaningful metric. Without it, all we are left with is a pretty graph.