There’s been a fair bit of chat in the wine world about ‘The Judgment of Princeton’ (here). This was a wine tasting organized by the American Association of Wine Economists (AAWE) along the lines of the famous Judgment of Paris from 1976 in which Californian wines gave some classic French wines a bit of a kicking in a blind tasting judged by French professionals. And this was at a time when new world wines were seen as distinctly inferior to the old world cousins.
The Princeton tasting pitted French classics against wines from New Jersey in a blind setting, with nine judges from the USA, France and Belgium. The full set of scores plus analysis is here.
I was fascinated to see that Tyler Colman (the only one of the judges I know personally) gave his highest red wine score to a New Jersey wine, and gave 2004 Mouton and 2004 Haut Brion 11/20 each, his lowest scores. I’m not saying he’s wrong, because I wasn’t there tasting alongside him, and don’t know how the wines showed. It’s just interesting.
As well as publishing the preferred wines in rank order, the AAWE also did a statistical analysis which showed that, with the exception of a couple of wines (the winning white and the losing red):
‘the rank order of the wines was mostly insignificant. That is, if the wine judges repeated the tasting, the results would most likely be different. From a statistically viewpoint, most wines were undistinguishable.’
This, rather than the fact that the New Jersey wines did quite well, was the take home message for many of the commentators on the tasting.
Jonah Lehrer, writing in The New Yorker, delves into the questions this tasting raises here.
While I agree with some of his conclusions, others I have a problem with.
“What can we learn from these tests? First, that tasting wine is really hard, even for experts. Because the sensory differences between different bottles of rotten grape juice are so slight—and the differences get even more muddled after a few sips—there is often wide disagreement about which wines are best.”
“The results are even more distressing for non-experts. In recent decades, the wine world has become an increasingly quantitative place, as dependent on scores and statistics as Billy Beane. But these ratings suggest a false sense of precision, as if it were possible to reliably identify the difference between an eighty-nine-point Merlot from Jersey and a ninety-one-point blend from Bordeaux—or even a greater spread. And so we linger amid the wine racks, paralyzed by the alcoholic arithmetic. How much are we willing to pay for a few extra points? These calculations are almost certainly a waste of time.”
I would make the following points.
- Sorry: wines don’t all taste the same. For some reason, newspapers love to run these stories suggesting that all wine taste pretty much the same, and therefore you are just wasting your money buying expensive wine. That is a false conclusion. Of course, there is not a strong correlation between wine price and quality, for all sorts of complex reasons. But this doesn’t mean that it isn’t possible to get much more interesting wines by paying more money.
- Blind tasting is difficult, and few can do it well. Is it possible that the line-up of judges in the Princeton tasting wasn’t a strong one? I don’t recognize many of the names. Are they experienced wine judges with a broad experience of international wines and good palates? I would bet a pricey bottle of wine that I could put together a list of experienced, able tasters who would produce much more robust results in this sort of tasting. Such tasters are rare, even among wine professionals, but they do exist, and the fact that they can perform well in blind settings like this shows that there are indeed real differences among wines.
- About the same time as this tasting was taking place, a sizeable number of students were sitting their Master of Wine examinations. These exams include a challenging blind tasting paper. The fact that many perform well in this examination shows that well trained tasters are able to differentiate quality in blind tasting with a degree of reliability. And in the Advanced Wine Assessment Course run by the Australian Wine Research Institute, participants’ palates are are tested in a statistically robust manner to see how good they are. Some are better than others, but those who reach show judge standard score consistently within 0.5 points on a 20 point scale.
- Yes, there is a degree of preference and subjectivity involved in wine assessment. But good professionals can set aside their preferences, and work more objectively in judging quality. There will certainly never be perfect agreement among a group of professionals, and in some cases there will be strong disagreement. But to conclude that wine tasting is entirely subjective and that all opinions are valid, and that expensive wine is a waste of money, is just silly.
- One further point about blind versus sighted tastings. Seeing the label influences our perception of the wine; it brings our knowledge about the wine into play. But it also helps us understand the liquid in the glass better. We can put into context the flavours we are experiencing. This is important: it is more than just bias; it is making sense of the sensation, which allows us to reach a more robust conclusion.
- Did this tasting place New Jersey wines on the map? No, because I think it was flawed. I don’t think the tasters did a good enough job, performing in an almost random fashion. I haven’t tasted the New Jersey wines, but I would be interested to—and I’m always open minded when it comes to non-classic wine regions. Yet I would be really surprised if they genuinely could compete in terms of quality and complexity with benchmark wines from Bordeaux in this sort of blind tasting setting with a highly competent, experienced tasting panel.