Does Age Matter? (Malt Maniacs Awards 2013 Edition)

So, the Malt Maniacs Awards for 2013 were announced today. These are, as most whisky geeks would agree, the best by far in what is a rather dubious genre. The whiskies are tasted and evaluated (more or less*) blind by a large number of the Malt Maniacs (with none of the professionals in the group participating) and the scores averaged. Those above 90 points on average are given “Gold Medal” status, those between 85 and 90 points are given “Silver Medal” status and those between 80 and 85 points are given “Bronze Medal” status.

In the past I’ve criticized the number of entrants that receive medals but I’ve come to see–courtesy discussion on the Malt Manics Facebook group–that this was based on a misunderstanding on my part. I read the medals in the sense of the Olympics, say, and from that perspective it seemed odd that such a large fraction of entrants should get medals. But as Serge Valentin and others pointed out to me, the proper frame of reference is an older one for evaluating consumer goods in which these levels merely signify tiers of quality. So, very few complaints from me about these awards per se (at least in this post).

What I was struck by this year while skimming the awards is both the low number of Gold Medals (a trend I wholeheartedly support) and the fact that just as many NAS whiskies got Gold Medals as did very old whiskies. And scanning the entire list quickly it seemed to me that there were some interesting and perhaps unexpected correlations between age and awards. So, here’s a quick rundown (it was compiled during breaks at a workshop I was attending so there may be some errors–please forgive or excoriate me for them as it pleases you):

Total Medals: 159

Gold Medals: 4 (2.5%)
Silver Medals: 64 (40.3%)
Bronze Medals: 91 (57.2%)

Now, let’s place the whiskies into three groups: Young (NAS-10 yo); teenaged; older than 20yo.

This is the breakdown of medals per age group:

Young: 1 Gold, 1.5%, 23 Silver, 35%, 42 Bronze, 63.5% (66 total awards)
Teenaged: 1 Gold, 2%, 17 Silver, 33.3%, 33 Bronze, 64.7% (51 total awards)
Old: 2 Gold, 4.7%, 24 Silver, 57%, 16 Bronze, 38% (42 total awards)

What jumps out is that very young to young whiskies are on par with whiskies from twelve up to 20 years old. This is already quite striking. But it gets more striking still if we drill down a little further and make five groups: NAS, age stated less than 12 yo, 12-20 yo, 20-30 yo, and older than 30 yo.

Now, this is how they break down by medal category:

NAS: 2.2% Gold, 24.4% Silver, 73.3% Bronze (total awards: 45)
<12yo: 0% Gold, 57.1% Silver, 42.9% Bronze (total awards: 21)
12-20yo: 2% Gold, 33.3% Silver, 64.7% Bronze (total awards: 51)
20-30 yo: 3.7% Gold, 57.6% Silver, 40.7% Bronze (total awards: 27)
>30 yo: 6.7% Gold, 60% Silver, 33.3% Bronze (total awards: 15)

I have no idea, of course, what the total of all entrants was or how the remainder broke down by age group. With that caveat in mind, this seems really striking to me as the performance of whiskies <12 yo (not including NAS) is on par with that of whiskies from 20-30 yo and better than that of whiskies from 12-20 yo.

Quick takeaways:

1. Blind tasting is great!

2. At least when tasting blind, the Malt Maniacs cannot be accused of rewarding age over all else. Not sure how this maps onto their non-blind scores though. That is to say, do younger whiskies do worse relative to older ones on the Monitor when they’re not being tasted blind?

3. There’s a lot of good young whisky out there.

Thoughts? Have I made any huge howlers? I will freely acknowledge that I have no training in statistics. And again, I have no idea what the age breakdown among whiskies that did not receive any medals was.

*More or less because it appears that some fraction of the judging panel may be involved in packaging and labeling the samples for distribution to the jury each year. This seems like an unavoidable problem but still worth noting.

[Note: Post edited an hour or so after it was first made and put out there.]

4 thoughts on “Does Age Matter? (Malt Maniacs Awards 2013 Edition)

  1. As a card-carrying statistician I feel obligated to respond to your invitation to comment on your interesting analysis.

    On the whole it is very good and your observations are correct: among the whiskies that receive medals, it’s obvious that there is no difference in the medal distribution between the NAS & <12yo combined and 12-20yo groups, or between the <12yo and the 21-30yo groups. This is visible with the naked eye; also, Statistics doesn't have much to say here, since it is usually concerned with detecting differences, not similarities.

    It is also true that overall age correlates with better scores/medals; Goodman's Gamma is a measure of correlation of ordinal variables (here, age and medal color), that counts the proportion of all pairs of observations (here, bottles) that have the presumed correct ordering, i.e. the older bottle of the two has the shinier medal. Here, Goodman's Gamma is about 0.30 using either of the age scales you discuss; p-value = 0.019 for the 3 age groups, and 0.005 for the five age groups. A p-value < 0.05 marks a statistically significant association (i.e., non-zero correlation) between older age and better medal.

    As for whether <12yo are doing better than the 12-20yo, while the numbers seem to incline in that direction, there are not enough data points (I never thought I'd call a whisky bottle *that*!) to draw this conclusion statistically; Goodman's Gamma = -0.39 (i.e., *younger* whiskies get better medals), but the p-value is 0.12, so not statistically significant.

    But the more important issues are the non-technical aspects of the analysis:

    1. As you noted, this analysis only relies on a part of the outcome, i.e., the bottles that received medals. If, for example, 50% of the NAS bottles do not receive medals, but all the 12-20yo bottles receive some medal, this would certainly affect the statistical analysis and its interpretation: the medal distributions of the two age groups are suddenly different when the non-awarded whiskies are taken into account . You recognized this limitation and decried the fact that MM did not disclose the number of non-medaled bottles in each category. This information would certainly be very helpful! I understand that there are at most 200 bottles entered, of which 159 received medals. Still, that's up to 41 bottles not accounted for, and we don't know how they are distributed by age groups.

    2. More importantly, there could be potentially important *confounders* in this analysis, i.e. other factors that may contribute to the medals being awarded, that are also related to the age group. So what we see as an association of age with better medals may in fact be an artifact, covering an underlying association between, say, higher strength and shinier medals. The most obvious factors to my mind are a) whisky strength, b) type of cask, c) peat. It is likely that the MM rate as better a cask strength whisky than the same whisky at 40% or 43% – after all, so do most of us. Also, one could make the case that MM are biased in favor of ex-sherry casks as opposed to ex-bourbon casks. Or at least, that they have some preference to one type of cask (or e.g. against cask finishing). Now, if proportionally more NAS whiskies turn out to be at cask strength, this could explain their success, despite the younger age(*). Same goes for the type of cask. A proper analysis would need to account for the individual bottle strength and type of cask – or other such potentially relevant factors. This analysis would not only tell us whether indeed strength matters, and how much, but it would also separate the effects of age from the effects of strength (or cask type) when it comes to the rating.

    In fact, I'd be quite interested in doing such an analysis. I could see this published in a statistical journal, and possibly in a proper whisky-relevant venue as well. What do you think, are you game?

    (*) Now this is getting really technical: The issue of strength is a little tricky since older whiskies tend to have lower strength simply by virtue of being older. One can argue then that strength is in fact a mediator, not a confounder, in the analysis, and therefore should not be adjusted for. There are ways around that, such as instead of including the strength per se, to adjust for whether the whisky is at cask strength (no water added) or not (water added).

    Like

  2. I agree with Florin’s point, above, on the analysis takeaways being partly based on award outcome (leaving the distribution of non-award bottles unknown) and that cask strength, or relative strength, could be having an effect on the results.

    On a somewhat separate note, NAS-labeled bottles can contribute nothing conclusive as to the merits of young whisky because the age of their contents are unknown (which is how the producers wanted it). Whether in cases where an NAS-labeled bottle is single vintage (where the age is unknown) or multi-vintage (where the vintage ages AND content proportions are unknown), one can only speculate, even with extensive experience, as to the ages involved.

    As for there being “a lot of good young whisky out there”, I would agree, but the Devil’s in the adjectives. Good, or even just worth purchasing, is one thing, but it’s a long way from making the argument that many in the industry would WANT to make, which is that age doesn’t matter. On the question of does age matter, it might be worth looking at Malt Maniacs E-pistle #2010-04, by Serge Valentin: “SO, DOES AGE MATTER? A BACKED UP ANSWER, FOR ONCE!” (http://www.maltmaniacs.net/E-pistles/Malt-Maniacs-2010-04-Does-the-age-of-Scotch-whisky-matter.pdf) – the sample base is much larger and the analysis is of scores, rather than medals.

    Like

    • Serge’s analysis is very nice especially because of the large sample size (865 bottles) and the fact that he limited it to blind reviews. However, it is not really a statistical analysis, it’s only computing and displaying the mean values for the age groups – there is no accounting for statistical variation, or for other factors as I mentioned above.

      A further complication in any analysis is accounting for the reviewer him/herself. If all reviewers score all whiskies that is not an issue. But that is likely not the case. If some reviewers tend to give higher scores than others, then the bottles reviewed will have biased scores, and this needs to, and can be corrected for in the analysis

      Like

  3. Just a few comments:

    1. This was not meant as a very serious analysis, just a kind of quick gloss of appearances. Those missing 41 whiskies could change this completely.

    2. Yes, the more meaningful question with the Malt Maniacs would probably be “Does Sherry Maturation Matter?” But I don’t have the patience to go through 159 listings and sort them by cask type (it’s not listed for many).

    3. All that said, the takeaway for me is that younger whiskies still performed better than I would have expected among a group of evaluators that generally rewards age.

    4. But the answer, in any case, is that age does matter, as the group that performed above expectations is the group most in danger of going extinct in favour of young, NAS whisky: age-stated whisky 12 yo or below.

    5. Florin, I would be happy to take you up on the statistical analysis as it’s obvious you would do all the work and I would just take all the credit.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s