From October 2016 to March 2017 the team is joined by Guest Kats Rosie Burbidge and Eibhlin Vardy, and by InternKats Verónica Rodríguez Arguijo, Tian Lu and Hayleigh Bosher.

Monday, 21 November 2011

Katonomics 3: Evidence-Based Policy – The challenge of data

A third slice of Katonomics. The IPKat again welcomes new readers who may be unfamiliar with the term 'Katonomics'. As regular readers now know, it's the title of a mini-series on IP economics for IPKat readers -- the vast majority of whom are not (yet) economists. This series is brought to you by the IPKat's own Katonomist, Nicola Searle, who turns this week to a theme which seems to excite policymakers today -- the increasing demand for IP policy to be based on evidence, rather than on the cherished tenets of our own received wisdom:

Evidenced Based Policy – The challenge of Data 
Recommendation number one of the Hargreaves Review begins with “Government should ensure that development of the IP System is driven as far as possible by objective evidence.” The thrust for evidence-based policy continues and, along with health care and other government policies, Intellectual Property policy looks set for a bit of a nudge

Not everyone is irrational 
Economics is grounded in theory and models, and economists are mildly obsessed by equations, graphs and demand curves. However, the economy has many variables, including, to economists’ great annoyance, irrational beings. In order to ensure that models and theories are robust, they must be tested. A preferred test method is empirical analysis, which examines whether economic policies supported by theory are also supported by evidence.

What constitutes evidence in economics? The current preference is for quantitative as opposed to qualitative methods and data. Data collection is not an easy task. For a lot of economic analysis, the government provides this service (e.g. the Office for National Statistics, the ONS). However, some interesting data may not be collected or made public. Researchers must instead collect their own data or generate it through surveys or experiments. Direct measurement of concepts (e.g. innovation) is not possible, so economists use proxies. For example, wealth serves as a proxy for the standard of living. Even money may be a proxy for the abstract concept of value. 
The economic theories and resulting policies discussed in earlier posts (here and here) call for testing. If IP is a means of incentivising innovation, then the innovation and the incentives created by IP should be analysed. Proposed changes to IP policy also merit analysis and, in some cases, require investigation via regulatory impact assessments. 
However, how do you measure IP? How do you test whether copyright incentivises creation? What is a trade secret worth? As IP is, by definition, intangible, data analysis is often achieved through proxies. These proxies vary by the type of IP. 
Patent data has long been a favourite of economists. It is a data rich source as patents are formally registered, publicly available, and have useful information such as citations. Further, patent data gets tantalisingly close to a proxy for innovation. The number of times a patent is cited, or the number of claims, suggests the innovative value of the patent. It is not without its problems, for example, newer patents will be less cited ("citation lag"). Dietmar Harhoff has done a lot a work in this area, including this presentation at an EPIP workshop (with Karin Hoisl and Colin Webb) which examines European patent citations and the problem of citation lags. Harhoff also investigates European patent examination in this paper with Stefan Wagner.
While patent data has its problems, copyright data is even more challenging. Unlike patents, no central source for copyright data exists. In particular, piracy data is problematic. A 2010 U.S. government report addresses this:
“Three widely cited U.S. government estimates of economic losses resulting from counterfeiting cannot be substantiated due to the absence of underlying studies. Generally, the illicit nature of counterfeiting and piracy makes estimating the economic impact of IP infringements extremely difficult, so assumptions must be used to offset the lack of data.”
To compound this, much of the relevant data (e.g. royalties stemming from licensing copyright material) is held privately. Policy makers may only have access to conclusions based on privately held data and no ability to verify conclusions. This is problematic given the challenges to economic analysis of copyright data and the contentious nature of copyright debates. 
When data is made available, it can offer surprising insights. For example, a computer games developer in the Netherlands, Joost "Oogst" Van Dongen, shared detailed stats on his recent Proun game. Unusually, he offered the game as pay-what-you-want and found that only 1.8% of players actually paid. He also discusses his revenue (about $0.09 per game) and estimates a level of piracy at around 41% based on detailed stats. 
However, where privately held data is unavailable, researchers may use a variety of public data sources to investigate copyright policy. Ruth Towse uses artists' earnings to analyse copyright and the royalty system of payment. In a UK IPO report, Martin Kretschmer uses sales prices of copying devices to investigate the levy system. Richard Watt details other examples of empirical work in a WIPO paper, which include North American copyright registrations and comparative analysis of book pricing. 
Copyright is not alone in its data challenges. Unregistered rights are particularly tricky; for example, the mere existence of a trade secret may not be known. Even registered rights are complicated; economic analysis of trademarks is limited by the lack of economic data in trade mark registrations. 
Given these challenges to data, how should researchers proceed? Is evaluating policy based on evidence the way forward? Or, for my question of the week, should governments compel rights holders to share data?


Anonymous said...

Interesting article. However, "The number of times a patent is cited, or the number of claims, suggests the innovative value of the patent" - nonsense.

Ron said...

My experience from working in both government and industry is that policy makers like to see data that can be expressed in numerical quantities that can be played with using a spread sheet and displayed in graphs and charts. This leads to data being collected that is of the type that is both easy to collect and which can be expressed numerically.

Such data is not necessarily the best way of measuring what is to be investigated. Thus a recent IPO report on comparison with other patent offices seemed to measure everything except quality, something you cannot readily measure numerically.

It would be sad if policy were made on the basis of data that was sound enough in the limited field of its terms of reference, but which was not appropriate to other fields.

Mention is made of Kretschmer's report. As the report itself states, its scope was restricted to three specific types of device, namely MP3 players; computer printers; and PCs. Policy decisons have been made in the past on the results of limited research such as this, because no other data was available.

I can see it happening: after all, the usual approach of policy makers is to decide policy first and then identify and present only those facts and evidence that support the policy. To that extent, all policy is based on evidence, but usually the cart comes before the horse.

As the Kat notes, data that might be meaningful is singularly lacking in most fields of IP, and obtaining data in the form that policy makers like to see it, and which is actually meaningful, will not be easy.

Nicola said...

To see what can be done with data-driven decision making, check out Super Crunchers ( It's fascinating but likely to make those with a preference for qualitative research shudder.

Nathan Wajsman said...

This is very timely indeed. I have enjoyed this series. At OHIM, we are embarking on several empirical projects of interest, including an IP Value study for the Observatory on Counterfeiting and Piracy, and other interesting projects. As it happens, yesterday, we held a seminar for OHIM employees on the link between trademarks and innovation, presented by Professors Ard-Pieter de Man and Meindert Flikkema from the VU University of Amsterdam, who have done interesting empirical work in this area using Benelux data and will now expand their study to the whole of the EU using OHIM data.

Evidence based policy said...

Thankyou to give such important information for us

Subscribe to the IPKat's posts by email here

Just pop your email address into the box and click 'Subscribe':