Is everything we know about wine wrong? asks the headline of an article that appeared earlier this week in UK national newspaper The Telegraph. ‘No’ is the answer, and I’ll explain why this article gets it wrong.
Over the last couple of years, there have been quite a few articles in the national press about wine competitions, and how professionals aren’t very good at tasting wine blind. This, the latest,
is a poor article that casts wine professionals in a bad light.
It’s yet another piece relying on the statistical work by Robert Hodgson from The California State Wine Fair. You can’t argue with statistics (when they are applied correctly, of course), and Hodgson’s study shows that the results from this wine show aren’t very reliable. I can’t argue with that: he has the data, and they appear sound. Hodgson also looked at the performance of different wines across a range of US-based competitions, and found that the same wine got different results at different competitions. His suggestion was that wines were performing randomly, according to chance.
Will Storr, the author of The Telegraph article, throws a lot of other rather unconnected material into his piece, including an extensive quote from Jilly Goolden where she claims to have invented the modern language of wine, among other things. But I’m bothered by his final conclusion that ‘despite the work of the scientists, it still appears as if there’s very little rational behaviour to be found among wine lovers and makers.’
It’s as if he’s bundled together as many interesting snippets about wine tasting that he could find, and then run out of space, so he’s reached a conclusion that certainly isn’t justified by what went before. Outsiders frequently want to believe that wine tasting is nonsense, and because it seems rather opaque and difficult to them (many find all wines taste the same), they are reassured when any evidence emerges that those of us in the wine trade are bluffing or making it up.
The problem is, we aren’t. Yes, tasting wine blind is difficult, but it is not impossible. If it were impossible – and experts really couldn’t tell the difference between cheap and expensive wine, for example – then no one would pass the blind tasting section of the Master Sommelier or Master of Wine exams.
The Australian Wine Research Institute regularly tests the blind tasting ability of winemakers and show judges. No one is perfect, of course, but some people are very good at it, and are consistent tasters. There are data on this, but journalists like Storr don’t bother to ask around properly – or simply ignore these relevant data.
And sensory scientists work with panels of tasters and come to statistically significant results. People are not perfect tasters, but some people are very good. Again, the data show it.
So how do I explain Hodgson’s results? Tasting blind in a competition setting is difficult. Many wine shows and competitions simply do not have the required calibre of judges. To get good, reproducible results in a competition, you need excellent tasters (and to get these, you’ll likely need to pay them), and you need a robust system that helps these tasters get the right results. It is likely that the shows Hodgson has looked at simply don’t have these required attributes.
Storr is right in as far as some wine professionals aren’t very good tasters. But many are, a fact which he’s ignoring on the basis of Hodgson’s sampling of individuals who clearly weren’t very good tasters, and that’s unfair.
9 Comments on Is everything we know about wine wrong?
Another possibility is that the California State Wine Fair is a particularly poor example of wine judging and wine competition. Based upon W. Blake Gray’s experience, that might be the appropriate conclusion. http://palatepress.com/2013/06/wine/wine-competition-for-whom-the-medals-toll/
I agree with you that not everything we know about wine is wrong. But this article does highlight the fallibility of competitions. It also shows two contrasting opinions. That wine experts have the same sensory ability and they are special. I suppose both kinds of experts exist. But there is no wine expert who does not have any experience.
When an expert takes part in a blind tasting panel for a competition, they are being asked to perform in a unnatural way. Tasting a hundred wines in a day without knowing what they are is not a normal way of appreciating wine.
I think Jamie is right about the professionalism of the main international wine competitions, although the one in california mentioned in the original article wouldn’t be on my list of major competitions, and not even particularly important in the USA.
However even with professional judging, I can tell you for an absolute fact that results do vary greatly for the same identical wine in various competitions. As a wine producer for over 2O years now, I have put wines into competitions year on year and seen the results vary greatly for the same wine in different competitions in the same year, from trophey to comended level for the same wine.
I don’t think that means the judges were wrong, it is just that any competition or tasting only judges the wines that are in that particular tasting and even the biggest competitions do not contain but a fraction of the wines available on the market so they cannot claim to be an absoluteguarantee that the winners are the best wines available.
Expensive wines especially don’t tend to be submitted to the larger competitions at all, – you don’t see chateau petrus very often in a wine competition, so I don’t think Jamie is correct to say that the wine competitions confirm the current value heirachy of the market.
I think Mr Hodgson is correct to say that there are many very expensive wines that are not really worth the price, and that there are equally many low/medium priced wines that are very good, but don’t have the history and marketing behind them to become well known.
A very thoughtful and thought-provoking piece, thank you Jamie. Having recently gone through the process of setting up a new competition, the Champagne & Sparkling Wine World Championships (we start tasting 12 May!), I have of course given considerable thought to these very issues. Firstly, we should distinguish between two types of blind tasting: assessment and guessing. Guessing is great nerdy fun, but it has nothing to do with competitive blind tasting. When wine judges make blind assessments, they are not, of course, totally blind: the judges know very broad parameters such as origin, year and variety or style. If they are not given this criteria, they could not possibly assess the typicity of a wine. However, although judges are given certain criteria, they are not told the identity. Nor are they expected to guess the identity. In fact, when a judge tries to guess the identity, he loses objectivity because his evaluation of that wine’s intrinsic quality has been corrupted. Anonymity is, after all, the entire raison d’être of competitive blind tasting. Probably the biggest problem for any wine competition is that the larger it becomes, the more judges that are drafted in, and the more judges drafted in, the more disparity in judging, making wineries wonder what the result might have been had their wine had been judged at a different table. Different competitions cope with this problem in different ways. Decanter’s World Wine Awards, for example, is the largest wine competition on earth, with more than 14,000 entries every year, but by awarding all the medals on a geographical basis and encouraging panel chairs to build their own team of judges experts, they can focus on terroir and typicity despite the huge number of judges involved. The International Wine Challenge, possibly the second-largest competition in the world, does not taste by teroir, but has installed an increasing number of checks and balances behind the scene that has resulted in each wine being tasted more times than at any other competition before it is given an award. Thankfully, the Champagne & Sparkling Wine World Championships has no desire to conquer the world. It is a niche competition by scope, and it is made all the more niche by declaring bronze medal to be failure for sparkling wines and silver medal failure for Champagne. Given the obvious impact this will have upon entries, it becomes logistically possible for the same panel of judges to taste every wine, and I have chosen the two best Champagne and sparkling wine tasters I know to join me: Essi Avellan and Tony Jordan. Tasting this style of wine competitively blind is not something that many people have much experience of in the world, but I have literally tasted thousands of Champagnes at competitions with both Essi and Tony, so I know I have the best for this niche competition. The number of times I have sat on a panel with an even number of judges that includes a chair who has a casting vote, and I always think why? Common sense dictates that any panel should have an odd amount judges and three is the perfect number to reach a consensus.
Well said, Jamie. My unsolicited two cents: a large part of the problem is lack of a tasting language, or system, and “experts” who do not know how to taste. A surprising number of wine industry professionals do not know how to critically access a wine
I think the wine industry needs a standard tasting system to access the quality and attributes of wine. The WSET SAT is a great start. The industry ask needs to be on the same page
Like you Jamie, I’m beginning to be fed up with these so called demystification of our humble trade.
How come our esteemed colleagues like M. Storr never ask us directly what we, wine writers and wine judges do, how we’ve learnt what we try to pass on, how we try our best to assess the wines as objectively as we can; how some of us also try to avoid jargon as far as they can. These so-called journalists always seem to rely on indirect information. They pretend to be on the customers’ side but rather than trying to educate them, it seems they they would prefer us to write like we are wine-morons.
We know we are not infallible, let us reassure them. But I think it is better that we exist; the alternative would be useless articles full of platitude; or flowery but empty verbiage, not to forget trite family wine experiences (yes, a lot of blogs are already like that); or plain advertising, communication by the producers fort the producers, no criticism anymore.
I wrote something pretty similar a year ago. And I would (obviously) defend good wine competitions. http://www.timatkin.com/articles?996
Spot the scientist that knows the word “data” is a plural!
A great piece Jamie, thank you. And some great comments too.
I am a director of Wine Competition Ltd and we own and run two competitions here in New Zealand. I have also had extensive experience across a number of other wine competitions.
Picking up on Herve’s point – I have an ‘open door’ policy for the naysayers and I have personally invited them – but never yet have they turned up. This includes so called influential wine writers. I agree that many writers comment who are not qualified to do so and they would benefit from seeing how a big wine show works so that they can talk from a base of fact rather than fiction.
Wine shows must be run with integrity – from who judges them, to how they do it and why, through to the ‘disposal’ of unopened bottles that were excess to requirement. No personal agendas, no bias.
The more we can showcase this to both the industry and the public, the more we can demonstrate value in the show system – well, those that are run professionally!