More thoughts on panel tastings
More thoughts on panels, prompted by yesterday’s judging experiences.
There are some levels of discrimination of quality where, irrespective of biological differences, personal preferences and cultural likings or dislikes, a panel of experienced tasters can reasonably hope to agree on broad-brush ratings of wines. [I’m thinking here of whether a wine is worthy of a bronze, silver or gold medal in a competition.]
These sorts of panels are good at filtering out poor or badly made wines, but can lack discrimination at the higher end. We were averaging points (on the 20 point scale) yesterday, and that makes it quite hard to get silver medals, and very hard to get golds – especially if you are using 18.5/20 as your benchmark for gold as in the Australian show system.
For this reason, the benchmark for gold was set lower at 17/20, with silver 15.5 and bronze 14. These may sound low, but they are realistic when you consider the way that averaging marks tends to bring down the overall score considerably.
To come up with sensible results, some level of conferring is necessary after each flight, to make sure that every wine is given a fair chance to get the medal it deserves. We found that we agreed on the majority of the wines (perhaps with one outlier out of the six), but some wines seemed to split opinion somewhat. We went back to these to reassess them.
The quality of the tasters is really important. One or two ‘random’ tasters in a panel can really mess up the results.
Panels like these can lose their effectiveness when dealing with the highest quality wines. While they serve a useful purpose in rating commercial wines, panels (and averaging scores) don’t work well for fine wine. They end up creating too many anomalies.
posted by Jamie @ 8:43 AM
11 comments
links to this post
11 Comments:
Oh, get over yourself! You assume your tastes are gospel for the rest of us! Random tasters can mess it up, can they?
Obviously there is quit a bit about taste which is personal (cultural and biological). Being a trained taster might or might not improve results even thought it might help achieve consensus. When you say conferring is needed it raises red flags for me. Opinions can be swayed by talking introducing random bias in the results. I would also be interested to how you judge quality versus random in a wine tasters.
some level of conferring is necessary after each flight
And there we have it. So, the tasters with the most dominant personalities ensure the wines they like get the highest scores.
A deeply flawed system.
Anon, I think you are being unfair.
A random taster is someone whose palate is not accurate. That is, someone who makes judgements that are unpredictably at odds with the majority of tasters.
Such a random taster will, when presented with duplicate wines in a tasting, give them strikingly different scores.
I've tasted with people like this before.
The conferring I refer to is not an attempt to convince others they are wrong, but rather asking people to have a second look. We all have a second look together. Sometimes we converge after this; other times, no change is made to the scores.
The conferring I refer to is not an attempt to convince others they are wrong, but rather asking people to have a second look
Yes, because you believe that that person got it wrong.
Personality and strident views should not sway others on a tasting panel. Like I said, a flawed system.
Nonetheless, I have had the experience (tasting in 'proffessional' competitions) that some judges score across the mid-range, being too scared of being emabarassed by their 'real' score when asked to reveal it by the presiding panel chairman (usually a MW, and who wants to be embarassed then!).So, flawed as it may seem, perhaps it is best to have no conferrals.Rather select experienced judges.
Derek KOCH
Seems to me that panel tastings ARE inherently flawed and that publishers who really care about the usefulness of the results should publish individual scores prior to conferring. That only happens in a minority of cases (I believe WOFW does it).
There's no way in which those involved would be prepared to admit that these competitions are really little more than marketing gimmicks. However, I think the industry does need them and the wary wine buff should just treat the results with a large dose of salt.
I value individual TN's far, far higher - whether my palate is aligned to the taster or not.
It is of course, very true that these competitions help with marketing - and most commercial entrants want to do well so they can promote their wines, but it is also a useful tool for the consumer.
To know that a wine has been awarded a medal by a very high calibre panel (in this cases 4 MW's and 2 experienced wine journalists) will give a consumer confidence over a wine that has no award. On the whole a Silver medal winning wine will be better than a Bronze and Gold will have needed very close consensus of opinion across the panel.
It is at bronze level where the biggest divergence of opinion is more likely to have occurred. Last years competition resulted in only one gold medal, yet over 30 wines had at least one judge rate them as gold medal material.
The biggest benefit of this style of blind tasting, is that it is a level playing field, without prejudice. Small scale unknown producers stand as good a chance of gaining awards as the larger scale producers, and consumers may get an introduction to a winery they never knew existed.
If I am perusing a shelf of wines that I do not know well, then a silver or gold award from a recognised competition is going to be a pretty safe bet, and could be a real stunner. Whereas a bronze or HC will have slightly greater risk of disappointment, but could still surprise!
Full results will be on www.swva.info very soon!
Jamie - out if interest what are your qualifications as a wine judge??
Have you been formally trained in wine quality assessment and/or judging??
Simon, seriously....you take some score, gold, siler or bronze, from the millions of medals out there to choose a wine for you? For shame...how about simple experimentation? Why does everyone have to be guaranteed of a good wine purchase? It's somebody elses tastebuds for crying out loud...put on your big boy pants and commit to a wine choice of your own. Leave Jamie alone...he is published, he must know something...except he doesn't know MY palate!!!!
Wow, did Anon get out of the wrong side of the bed?
First off, any system involving humans is inherently flawed - no individual or group of individuals can ever be perfect all the time. So, trying to argue that wine competitions are flawed because people discuss a wine which has a wide score range is, to be honest, ridiculous.
Second, very few competitions require judges to be trained or assessed; I have done a one-day AWAC course with the AWRI and I have a good idea of how varied a taster I am, but whether or not I've done this the people who run the competitions can ask me to judge if they wish. I do know that all the competitions I've judged in which are run under OIV rules assess judges and do not ask back those whose scores are too variable.
Jancis Robinson once stated that the best panel is a panel of one - you give your straight opinion and everyone knows where you stand. Once another person is involved you're ALWAYS into compromise. FWIW, Anon should realise that these discussions, in my experience, do NOT come down to the loudest getting their way; frequently, it's the opposite.
Finally, wine competitions exist essentially for the marketing departments of producers, wholesalers and retailers - not as a guide for consumers, even if they're advertised as such. If you don't believe in them then fine - ignore them. But I really think it's a bit rich to castigate JG for allowing "the dominant personalities" to take over in judging while, at the same time, berating people for taking the results seriously. It seems to me that Anon is behaving in much the same way as these alleged dominant personalities - worse, he's anonymous.
I judge all over the world and believe that many systems have faults but that, overall, these are not that serious and that, interestingly, a number of wines get medals which would not be expected to do so - some stars are discovered by these competitions. If there is one enduring fault it is that scoring systems cannot adequately reflect the different criteria for assessing high-end wines and high-volume wines; but, as I said, no human system can ever be perfect.
Dermot Nolan MW and judge at Micheleangelo, Mundus Vini, International Wine and Spirit Competition, Food Hotel Asia and Les Citadelles du Vin.
Post a Comment
Links to this post:
Create a Link
<< Home