« New Rules | Main | Grasp by Analogy »

March 2, 2008

Practice Makes Perfect?

Statistics is a craft; running the numbers is the easy part. The tough bit comes down to data quality: how reliable is the base of data employed? Heterogeneity is the bane of statistical reliability: comparing apples to oranges tells you nothing about either apples or oranges.

By definition, every patent is supposedly novel. So, statistically comparing patent litigations inherently prances into a minefield. Only prudence, that numbers can lie, but that the smell test of sensibility grants good guidance, makes numbers truly meaningful.

Straight faced, Dave Schwartz concluded: "data do not reveal any evidence that district court judges learn from prior appeals of their rulings. There is no suggestion of a significant relationship between experience and performance." Schwartz, by example, empirically reminds that statistically inclined and logically challenged are stochastically independent.

Schwartz drafted an essay: "Practice makes perfect? An empirical study of claim construction reversal rates in patent cases."

Schwartz surveyed district and appellate court patent cases for claim construction reversal. Reversal rate averaged around 30%, with the top ten most active districts having a low of 22% in Delaware and East Virginia, to 44% in Central California.

Many have criticized district court judges for the high claim construction reversal rates. This empirical study of the Federal Circuit's review of district court judges indicates that the reversal rate may be essentially constant, regardless of the prior claim construction experience of the district court judge. All judges have access to the universe of reported decisions. If district court judges are supposed to learn from appellate court review of their cases above and over the background learning from the universe of reported cases, one would expect some improvement of the reversal rate as experience increases. Contrary to theory, district court judges do not appear to improve based upon various measures of experience. As to the specific problem in patent law, the article points to three possible explanations for the lack of improvement: (1) an indeterminate nature of claim construction; (2) a failure of the Federal Circuit to teach properly how to construe claims; and (3) a failure of district court judges to learn claim construction. The data suggests that quasi-specialized patent trial judges, as proposed in pending legislation, will not automatically reduce the reversal rate.

What one would reasonably surmise is what everyone knows: patent litigation is intrinsically complex. Contrary to Schwartz nonsensical conclusion, if claim construction is difficult, it cries out for specialization. Economics is founded upon the bedrock of the wisdom of division of labor based upon the quality gained from specialization. While drawing the opposite conclusion, Schwartz himself makes the argument.

Patent law is difficult. Not only is the law intricate and ever-changing, but the patents themselves describe complex and often cutting-edge technology. District court judges are typically legal generalists, with no training in patent law. They also commonly lack scientific training or a technical background.

In a sense, each patent case is as unique as the patent asserted, as unlimited in variation as the technologies brought forth. The rulebook for claim construction continues to evolve. Hence, for all judges, multi-faceted, learning curves are constantly mounted.

Schwartz did dust his data with doubt, but heavily laced that with sordid speculation, unfounded by fact or logic.

Notwithstanding the high validity and reliability of both the appellate decision and district court lawsuit databases, there were several internal shortcomings for which the study was unable to account. Both databases are affected by the nature of litigation because the merits of the cases, the parties and their resources may not be spread equally across cases. There is reason to believe that forum shopping is another significant problem in patent litigation. For example, it may be that a particular judicial district may hear more cases brought by entities that own a patent, do not produce any products and only desire to settle the lawsuit for a licensing arrangement. Or, more patent disputes between Fortune 100 companies may be filed in a particular judicial district. Thus, it is highly likely that the types of parties involved are not randomly distributed throughout the judicial districts. The skewed distribution across judicial districts means that district court judges are not assigned a random sample of the total pool of patent lawsuits. This is because district court judges are assigned cases within the judicial district in which they sit.

The lack of random distribution of patent cases may be particularly profound if the types of parties to the lawsuits have some relationship to the difficulty or closeness of the claim construction issues. If, for example, lawsuits brought by non-practicing entities whose business goal is to license an entire industry are weaker suits, we would expect those cases to be easier to resolve correctly by district court judges. Thus, district court judges that hear a higher proportion of those cases would have inflated affirmance rates. In addition, different types of parties to patent lawsuits may be correlated to appeal rates. For example, generic drug manufactures or non-practicing entities may be more likely to appeal adverse decisions than lawsuits involving competitors. A higher appeal rate of certain types of parties would magnify the effect of any selection bias.

However Schwartz may have faulted his data base, he was unwilling to treat it with the caution that sense would dictate. Even at the appellate level, as any regular reader of the Patent Prospector knows, claim construction case law fluxes. One might naively think, after all these years, claim construction would be cleanly codified. But dissent in CAFC claim construction rulings is common. Would one then reasonably surmise that appellate court judges are unable to learn from their own experiences and precedents? Emphatically no.

Schwartz' bias against forum shopping is nothing short of a non-sequitur serving of tripe. Admittedly non-random distribution of patent cases indicates that plaintiffs select district based upon criteria. Unless everyone is an imbecile, as Schwartz' statistics may show, forum shopping involves rational selection.

Plaintiffs with patents of merit would want courts where the merit could be discerned, particularly for complex technologies where the case could be highly contentious and close-fought. If your patent is weak, the last thing you'll want to do is rush to a patent-savvy district, where the judge will blow through the smoke. Just as patents vary, so too the quality of the district.

History gives the best example. A centralized patent appeals court became law in 1982 because of scandalous disarray: certain appeal circuits were anti-patent and others pro-patent, thus churning radically different case law, so forum shopping was rife. Now, at least, the dissent and flip-flopping is centralized.

Schwartz capsulated the irony of his article by his preamble quotation.

"Experience is the name everyone gives to their mistakes." - Oscar Wilde

The problem is seeing the mistake in the first place. Obviously, Schwartz didn't see his, however apparent it was: assuming statistical homogeneity in an heterogeneous reality is a fool's game.

Schwartz was so hasty to publish that he slopped a draft out with the pathetic plea: "Please do not distribute or quote without permission of the author," then touted his results in Patently-O in multiple installments.

Posted by Patent Hawk at March 2, 2008 2:23 PM | Litigation


He must have gone to the Kimberly Moore School of useless statistics and self promotion.

Posted by: Scott at March 7, 2008 2:25 PM

Because your post is directed to my article, I feel the need to respond briefly.

First, you state that because "every patent is supposedly novel," empirical analysis of multiple patent litigations is unreliable. I respectfully disagree. While the "facts" are different for each patent, the law is not. Even if the law changes over time, the district court judges all have access to the same body of case law. For that reason alone, I believe it is fair to analyze reversal rates based upon judicial experience with patent litigation. At a minimum, my study provides more information than was previously available. For example, we now know that judges with the largest dockets of patent cases are reversed at roughly the same rate as judges with tiny dockets of patent cases.

Second, I disagree that I did not present my data and analysis with caution. At numerous places in the article (including several paragraphs you quoted in your blog post), I was quite explicit about various possible limitations in the data. In fact, I even attempted to quantify any potential selection bias. More specifically, in Appendix B of the article I analyzed whether judges substantially above and below the median reversal rate were appealed at different rates than other judges.

Finally, you believe that a specialized trial court would be beneficial. While the results of my study do not appear to support the idea that patent experience lowers the reversal rate, my article suggests that “a limited pilot program would operate as a real world experiment to confirm or refute the findings of the present study.”

Posted by: Dave Schwartz at June 29, 2008 5:16 PM

Hi Dave:

Thanks for writing.

Your first objection in your comment, based upon my observation that “statistically comparing patent litigations inherently prances into a minefield,” was a series of non sequiturs without conclusion, other than to state that your numbers don’t really tell you anything of significance. In other words, with a seeming ipso facto faith in numbers per se, you don’t seem to realize the minefield you pranced into.

Your second stated disagreement practically disowns any conclusions your drew, as well as you practically damning your own data. Yet you still felt compelled to sew a silk purse from a sow’s ear: “The data suggests that quasi-specialized patent trial judges, as proposed in pending legislation, will not automatically reduce the reversal rate.”

If anything is to be done to lower reversal rates, there is only one possible thrust: education/training and specialization. If the situation appears hopeless, as your numbers seem to suggest, that’s not an argument for doing nothing in this instance.

The elephant in the room on this one is that U.S. case law is badly bunged up. Claim construction is much more a black art than it should be. But if claim construction is rocket science, then judges in patent cases need to be rocket scientists. Do you hear the echo from the previous paragraph?!

My last paragraph stands. You don’t appear to have a grasp on the patent litigation situation. Another academic with a paper. The numbers you came up with tell you next to nothing, other than to delve deeper.

Posted by: Patent Hawk at June 29, 2008 7:10 PM