Does organic wine taste better? A study whose findings have been widely misinterpreted

does ecocertified wine taste better

Are organic wines better than conventional ones? An interesting study from researchers at the University of California Los Angeles suggests that they might well be. However, the difference in point scores between eco-certified wines and conventional wines is actually much smaller than press reports on this study have stated, because they haven’t read the paper carefully.

This study examined critic ratings of almost 75 000 Californian wines, with vintages ranging from 1998 to 2009, from just over 3800 wineries. The scores were taken from The Wine Advocate, Wine Enthusiast, and Wine Spectator. The researchers then checked to see whether any of these wines were certified as organic or biodynamic from those wineries (in those vintages), to create a subset of what they call eco-certified wines which they then compared with conventional wines.

[As an aside, the analysis of scores is really interesting. The authors published the distribution of scores from each of the wine publications, and also together, and found that they roughly followed a normal distribution with the median score in this distribution being 89. But they found a rounding up effect, in that many of the 89 point scores appeared to have been rounded up to 90. There are fewer wines scored at 89 points (5153 wines) than there are at 88 (7584 wines) and at 90 (6989 wines).]

ecocertified wines scores

Only 1.1% of the wines in the sample were eco-certified, but being eco-certified increases the score of the wine by 4.1 points. But the important detail here is that we are not talking 4.1 points on the 100 point scale, and this is something that no one who has written about this has emphasized or even picked up, as far as I know. This is 4.1 points on a standardized scale devised by the authors, which has a much broader discriminatory power, because it is more spread out than the very bunched 100 point scale. This detail really does change everything. The standard deviation of the 100 point system is around 4 for these three publications, whereas it is 28 for the scaled score, a sevenfold difference. So the advantage of being organic on the 100 point scale is much smaller than 4.1 points!

It also needs to be clarified that this result doesn’t mean that converting to organics or biodynamics is going to raise the score of your wine 4.1 points on this standardized point scale. There is no evidence for causation here.

In the first instance, there is no stratification by price in this study. It could be that the average price of the eco-certified wines could be a lot higher than that of the average of the conventional wines, and as there would likely be a strong correlation between price and wine score this would be a confounder. Also, it could be a selected subset of higher-achieving wineries that makes the switch to organics. It takes an ambitious, conscientious producer to switch from conventional farming to organics. In California, organics is a very small subset of all wines.

The other problems with this study is that quality is measured though critic ratings. If you look at the particular critics who gave those ratings, then the question is one of whether you think their palates actually differentiate quality in a meaningful way (to your own particular palate).

Despite any evidence for a causal link, there remains the possibility that farming organically could improve wine quality. Potential mechanisms exist, and I think that there is a lot of potential benefit to farming vines with healthy, living soils (although certified organic/biodynamic is not the only way to achieve this). But this study shouldn’t be taken to show that shifting to organics results in better quality wines, because it doesn’t.

4 comments to Does organic wine taste better? A study whose findings have been widely misinterpreted

  • Good one!

    Mildly off topic, but I’d also be interested to see if the median score per year has increased from 1998 to 2009.

  • Bob Rossi

    Very interesting study and posting. I’ve visited many many wineries in France, almost all of them small producers. I’m not doctrinaire about organics, etc., not by a longshot. But it’s always been my opinion that there are certain things that indicate to me that the producer is conscientious, and consequently is likely to make better wine. That includes things like converting to organics or biodynamics, or adhering to a charter of a group like the Vignerons Independants. On the other hand, I’m not impressed when a producer says he or she makes “natural wine,” since that has no legal meaning. But I actually don’t recall ever visiting a winery in France that touted itself as “natural.” Maybe that’s more an American marketing thing.

  • Bob Henry

    One would implicitly “think” that an organic wine grape grower would have a greater economic (e.g., higher selling price) and emotional (e.g, pride in one’s family name affixed to the brand) investment in harvesting a higher quality crop.

    And implicitly “think” that a contract fruit buyer of those same organic wine grapes would demand a higher quality crop because s/he has a similar economic and emotional investment in the (family?) branded wine.

    Organic wine grapes may be the beneficiary of a “virtuous cycle” of ever higher expectations of quality exerted by contract fruit buyers upon contract fruit growers.

  • Bob Henry

    With Jamie’s indulgence . . .

    There is this separate consideration: do the writers at all three cited publications review wines the same way?

    Evidence indicates otherwise.

    Robert Parker stated in an interview with Wine Times magazine (circa 1989) that his 50 point scale [*] assesses a wine based on its components:

    “The wine gets up to 5 points on color, up to 15 on bouquet and aroma, and up to 20 points on flavor, harmony and length. And that gets you 40 points right there. And then the [balance of] 10 points are . . . simply awarded to wines that have the ability to improve [i.e., age] in the bottle.”

    Whereas Wine Spectator stated in a reply to a reader’s letter-to-the-editor (circa 1994):

    “In brief, our editors do not assign specific values to certain properties [i.e., components] of a wine when we score it. We grade it for overall quality as a professor grades an essay test. We look, smell and taste for many different attributes and flaws, then we assign a score based on how much we like the wine overall.”

    These differing methodologies explain why Wine Advocate and Wine Spectator reviews don’t agree — either by their invoked word descriptions or point scores.

    Those who have studied math know that when dealing with disparate “nominal,” “ordinal,” “interval,” and “ratio” scales, you cannot forge a “conversion” between them.

    Seems to be this UCLA study is intrinsically flawed because it “assumes” all 100 point scales are interchangeable and convertible.

    [*Robert Parker: “If you start at 50 [points] and go to 100 [points], it is clear it’s a 50-point system, and it has always been clear. Mine is basically two 20-point systems with a 10-point cushion on top for wines that have the ability to age.”]

    With Jamie’s further indulgence . . .

    With Jaime’s indulgence . . .

    Here’s an example of wine writers at the SAME magazine ostensibly using the SAME 100 point scale couldn’t agree when sampling the same bottle of varietal wine.

    In a Wine Spectator article titled “The Cabernet Challenge” (September 15, 1996, pp. 32 – 48), their Bordeaux wine critic James Suckling and their California wine critic James Laube compared and contrasted various 1985 and 1990 vintage red Bordeaux and California Cabernet Sauvignons/Cabernet-blends.

    After controlling for pours out of the same bottle, using the same brand/model stemware, experienced in “real time” in the same shared room, what were the results?

    Two divergent word descriptions and numerical scores/place order rankings — with the same bottle score differing by upwards of 9 points on their 100 point scale.

    First example: 1990 Chateau Margaux red Bordeaux . . .

    JAMES LAUBE: “A tight, hard-edged and unyielding young wine. Some cedar and currant flavors attempt a coup on the finish, but they’re tightly wrapped in tannin. 86 points.” [20th place personal ranking in the comparative 1990 California versus Bordeaux tasting.]

    JAMES SUCKLING: “Slightly dumb now. Ripe, almost raisiny aromas and flavors that develop a minty, menthol accent. Full-bodied and rich with loads of tannins. Needs time. Better after 2005. 90 points.” [10th place personal ranking in the comparative 1990 California versus Bordeaux tasting.]

    Second example: 1990 Beringer “Reserve” [Napa Valley] Cabernet Sauvignon . . .

    JAMES LAUBE: “Dense and massive, but for all its weight and intensity it delivers a rich, ripe mouthful of currant, cherry, plum, anise and cedary, toasty oak flavors. With its impressive length, depth and concentration, this wine should age with ease for another decade. 98 points.” [1st place personal ranking in the comparative 1990 California versus Bordeaux tasting.]

    JAMES SUCKLING: “Smashes you over the head with masses of fruit and full tannins. Full-bodied, with a long finish. A little tiring to taste, even more so to drink! Better after 2005. 89 points.” [15th place personal ranking in the comparative 1990 California versus Bordeaux tasting.]

    Third example: 1985 Lynch-Bages red Bordeaux . . .

    JAMES LAUBE: “Classic Bordeaux from the first cheesy, cedary whiff – an aroma rarely duplicated by California Cabernets. Drinks better, with currant and anise notes, and earthy, funky flavors, but struggles to maintain focus. Tannins still a bit raw. Tasted several times, with consistent notes. 87 points. [14th place personal ranking in the comparative 1985 California versus Bordeaux tasting.]

    JAMES SUCKLING: “Our wine of the year in 1988 and still well worth it. The first bottle was slightly cheesy but the second one was superb, showing outstanding ripe berry, cherry and currant flavors and layers of silky fine tannins. Sexy and exciting. Drink now or hold. 95 points.” [2nd place personal ranking in the comparative 1985 California versus Bordeaux tasting.]

    Self-evidently, Laube and Suckling “agreed to disagree” on the relative scores and ranking comparisons of the wines.

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

*