This post is timed to coincide with a meeting tomorrow, the Royal Meteorological Society’s “Communicating Climate Science”. If you are going, do come and say hello. If you aren’t, look out for me tweeting about it from 2-5.30pm BST.
On not blogging
I haven’t forgotten about you. I’ve still been churning over ideas and wanting to share them with you. I’ve thought of all of you that comment here, and those that silently lurk, whether friends, family, scientists, sceptics, passers-by, or a combination of these. But two big things this year have had to take priority over blogging (and the even more time-consuming process of moderating and replying to comments).
The first was a deadline. As some of you know well, the Intergovernmental Panel on Climate Change (IPCC) produces a report summarising the state-of-the-art in climate science research, and related topics, about every six years. They do this so policymakers have a handy (in practice, enormous and not very handy) reference to the evidence base and latest predictions. The IPCC set cut-off dates for including new research: one date for submission to journals, and another for acceptance after the peer-review process. The first of these dates was the 31st July this year. Translation: “try to finish and write up every piece of work you’ve ever started by this date”. Not every climate scientist chose to do this. But the project I work for, ice2sea, actually had it written into a contract with its funders, the European Union. We had no choice but to submit whatever was our current state-of-the-art in sea level predictions. I was a co-author of six papers* finished and submitted during June and July, and had several other studies on the go that didn’t make the deadline. So it was a rather intense time, and science had to take priority over talking about science.
The second was personal. I hesitated about whether to say this here. But part of my motivation for being a climate scientist in the public eye was to show the human side. And I also wanted to let you know that this blog is so important to me, has been so transformative, that it took something very big to keep me away. My husband and I separated two months ago.
I’m back, and I’m preparing for a big move. The US-based publisher and organisation PLoS (Public Library of Science) has invited me to be their climate blogger. It’s a fantastic opportunity to gain a big audience (more than 200,000 visitors per month, and a feed to Google News). I’m very happy to support PLoS because they publish open access journals, and because one of these (PLoS ONE) goes even further in its commitment to transparency in science. It will publish anything scientifically valid, whether or not it is novel. This might not sound important, or even a good idea, but it is an essential counter to the modern problem that plagues journals: that of only publishing new results, and not repeat studies. For the scientific method to work, we need studies that repeat and reproduce (or contradict) previous research. Otherwise we risk errors, chance findings, and very occasionally fraud, remaining unnoticed for years, or forever. I’m hosted at PLoS from the second week in December and will be posting twice a month.
The first post at PLoS will be a (long overdue) introduction to predicting climate change. It will probably be based around a talk I gave at the St Paul’s Way summer science school, at which I was the final speaker, which made Prof Brian Cox my warm-up act.
In other news, I talked about the jet stream and climate change live on BBC Wiltshire (9 mins), which was well received at the climate sceptic site Bishop Hill, and did a live Bristol radio show, Love and Science (1 hour). I also returned to my particle physics roots, with a Radio 4 interview about the discovery of the Higgs Boson (3 mins).
Our new(-ish) paper
Now the science bit. This is an advertisement for a paper we published in August:
Stephens E.M., Edwards T.L. and Demeritt D. (2012). Communicating probabilistic information from climate model ensembles—lessons from numerical weather prediction. WIREs Clim Change 2012, 3: 409-426.
It’s paywalled, but I can send a copy to individuals if they request it. Liz Stephens is a colleague and friend from my department at Bristol that did a great study with the UK Met Office and David Spiegelhalter on the interpretation of probability-based weather forecasts, using an online game about an ice cream man. I’ve never met David Demeritt, except in one or two Skype video calls. He’s interested in, amongst other things, how people interpret flood forecasts. I haven’t passed this post by them, but hopefully they will comment below if they have things to add or correct.
We noticed there was quite a bit of research on how well people understand and make decisions using weather forecasts, such as the probability of rainfall, and uncertainty in hurricane location, but not much on the equivalents in climate change. There have been quite a few papers, particularly in the run-up to the new IPCC report, that talk in general terms about how people typically interpret probability, uncertainty and risk, and about some of the pitfalls to avoid when presenting this information. But very few actual studies on how people interpret and make decisions from climate change predictions specifically. We thought we’d point this out, and draw some comparisons with other research areas, including forecasting of hurricanes, rain, and flooding.
The ‘ensembles’ in the title are a key part of predicting climate and weather. An ensemble is a group, a sample of different possibilities. Weather forecasts have been made with ensembles for many years, to help deal with the problem of our chaotic atmosphere. The most well-known explanation of chaos is the ‘butterfly effect’. If a butterfly stamps its foot in Brazil, could it cause a tornado in Illinois? Chaos means: small changes can have a big effect. A tiny change in today’s weather could lead to completely different weather next week. And in the same way, a tiny error in our measurements of today’s weather could lead to a completely different forecast of the weather next week. But errors and missing measurements are inevitable. So we try to account for chaotic uncertainty by making forecasts based on several slightly different variations on today’s weather. This is one type of ‘ensemble forecast’. It’s simply a way of dealing with uncertainty. Instead of one prediction, we make many. We hope that the ensemble covers the range of possibilities. Even better, we hope that the most common prediction in the ensemble (say, 70% of them predict a storm) is actually the most likely thing to happen. This gives us an estimate of the probability of different types of weather in the future.
Ensembles are at the heart of our attempts to describe how sure we are about our predictions. They are used to explore an uncertain future: what are the bounds of possibility? What is plausible, and what is implausible? Some climate prediction ensembles, like the weather forecast ensemble above, relate to the information we feed into the model. Others relate to imperfections in the models themselves. Some specific examples are in the footnotes below.**
The question we ask in our paper is: how should we express these big, complex ensemble predictions? There are too many dimensions to this problem to fit on a page or screen. Our world is three dimensional. Add in time, and it becomes four. There are very many aspects of climate to consider, such as air temperature, rainfall, air pressure, wind speed, cloud cover, and ocean temperature. We might have a prediction for each plausible input value, and a prediction for each plausible variation of the model itself. And one of these ensembles is produced for each of the different climate models around the world. Frankly, ensembles are TMI***.
To simplify or not to simplify
Scientists often think that the more information they can give, the better. So they dump all the raw ensemble predictions on the page. It’s a natural instinct: it feels transparent, honest, allows people to draw their own conclusions. The problem is, people are a diverse bunch. Even within climate science, they have different knowledge and experience, which affects their interpretation of the raw data. When you broaden the audience to other scientists, to policymakers, businesses, the general public, you run the risk of generating as many conclusions as there are people. Worse still, some can be overwhelmed by a multitude of predictions and ask “Which one should I believe?”
To avoid these problems, then, it seems the expert should interpret the ensemble of predictions and give them in a simplified form. This is the case in weather forecasting, where a meteorologist looks at an ensemble forecast and translates it based on their past experience. It works well because their interpretations are constantly tested against reality. If a weather forecaster keeps getting it wrong, they’ll be told about it every few hours.
This doesn’t work in climate science. Climate is long-term, a trend over many years, so we can’t keep testing the predictions. If we simplify climate ensembles too much, we risk hiding the extent of our uncertainty.
Our conclusions can be summed up by two sentences:
a) It is difficult to represent the vast quantities of information from climate ensembles in ways that are both useful and accurate.
b) Hardly anyone has done research into what works.
We came up with a diagram to show the different directions in which we’re pulled when putting multi-dimensional ensemble predictions down on paper. These directions are:
- “richness”: how much information we give from the predictions, i.e. whether we simplify or summarise them. For example, we could show a histogram of all results from the ensemble, or we could show just the maximum and minimum.
- “saliency”****: how easy it is to interpret and use the predictions, for a particular target audience. Obviously we always want this to be high, but it doesn’t necessarily happen.
- “robustness”: how much information we give about the limitations of the ensemble. For example, we can list all the uncertainties that aren’t accounted for. We can show maps in their original pixellated (low resolution) form, like the two maps shown below, rather than a more ‘realistic-looking’ smoothed version, like these examples.
Here’s the diagram:
The three ‘dimensions’ are connected with each other, and often in conflict. Where you end up in the diagram depends on the target audience, and the nature of the ensemble itself. Some users might want, or think they want, more information (richness and robustness) but this might overwhelm or confuse them (saliency). On the other hand, climate modellers might reduce the amount of information to give a simpler representation, hoping to improve understanding, but this might not accurately reflect the limitations of the prediction.
In some cases it is clear how to strike a balance. I think it’s important to show the true nature of climate model output (blocky rather than smoothed maps), even if they are slightly harder to interpret (you have to squint to see the overall patterns). Otherwise we run the risk of forgetting that – cough – all models are wrong.
But in other cases it’s more difficult. Giving a map for every individual prediction in the ensemble, like this IPCC multi-model example, shows the extent of the uncertainty. But if this is hundreds or thousands of maps, is this still useful? Here we have to make a compromise: show the average map, and show the uncertainty in other ways. The IPCC deals with this by “stippling” maps in areas where the ensemble predictions are most similar; perhaps the unstippled areas still look quite certain to the hasty or untrained eye. I like the suggestion of Neil Kaye, fading out the areas where the ensemble predictions disagree (examples of both below).
This brings us to the second point of our conclusions. The challenge is to find the right balance between these three dimensions: to understand how the amount of information given, including the limitations of the ensemble, affects the usefulness for various audiences. Do people interpret raw ensemble predictions differently to simplified versions of the same data? Do full ensemble predictions confuse people? Do simplifications lead to overconfidence?
There is very little research on what works. In forecasting rainfall probabilities and hurricanes, there have been specific studies to gather evidence, like workshops to find out how different audiences make decisions when given different representations of uncertainty. People have published recommendations for how to represent climate predictions, but these are based on general findings from social and decision sciences. We need new studies that focus specifically on climate. These might need to be different to those in weather-related areas for two reasons. First, people are given weather forecasts every day and interpret them based on their past experiences. But they are rarely given climate predictions, and have no experience of their successes and failures because climate is so long-term. Second, people’s interpretation of uncertain predictions may be affected by the politicisation of the science.
To sum up: we can learn useful lessons from weather forecasting about the possible options for showing multi-dimensional ensembles on the page, and about ways to measure what works. But the long-term nature of climate creates extra difficulties in representing predictions, just as it does in making them.
* Papers submitted for the IPCC Fifth Assessment Report deadline:
- Ritz, C., Durand, G., Edwards, T.L., Payne, A.J., Peyaud, V. and Hindmarsh, R.C.A. Bimodal probability of the dynamic contribution of Antarctica to future sea level. Submitted to Nature.
- Shannon, S.R., A.J. Payne, I.D. Bartholomew, M.R. van den Broeke, T.L. Edwards, X. Fettweis, O. Gagliardini, F. Gillet-Chaulet, H. Goelzer, M. Hoffman, P. Huybrechts, D. Mair, P. Nienow, M. Perego, S.F. Price, C.J.P.P Smeets, A.J. Sole, R.S.W. van de Wal and T. Zwinger. Enhanced basal lubrication and the contribution of the Greenland ice sheet to future sea level rise. Submitted to PNAS.
- Goelzer, H., P. Huybrechts, J.J. Fürst, M.L. Andersen, T.L. Edwards, X. Fettweis, F.M. Nick, A.J. Payne and S. Shannon. Sensitivity of Greenland ice sheet projections to model formulations. Submitted to Journal of Glaciology.
- Nick, F.M., Vieli, A., Andersen, M.L., Joughin, I., Payne, A.J., Edwards, T.L., Pattyn, F. and Roderik van de Wal. Future sea-level rise from Greenland’s major outlet glaciers in a warming climate. Submitted to Nature.
- Payne, A.J., S.L. Cornford, D.F. Martin, C. Agosta, M.R. van den Broeke, T.L. Edwards, R.M. Gladstone, H.H. Hellmer, G. Krinner, A.M. Le Brocq, S.M. Ligtenberg, W.H. Lipscomb, E.G. Ng, S.R. Shannon , R. Timmerman and D.G. Vaughan. Impact of uncertainty in climate forcing on projections of the West Antarctic ice sheet over the 21st and 22nd centuries. Submitted to Earth and Planetary Science Letters.
- Barrand, N.E., R.C.A. Hindmarsh, R.J. Arthern, C.R. Williams, J. Mouginot, B. Scheuchl, E. Rignot, S. R.M. Ligtenberg, M, R. van den Broeke, T. L. Edwards, A.J. Cook, and S. B. Simonsen. Computing the volume response of the Antarctic Peninsula ice sheet to warming scenarios to 2200. Submitted to Journal of Glaciology.
** Some types of ensemble are:
- ‘initial conditions’: slightly different versions of today’s weather, as in the weather forecasting example above
- ‘scenarios’: different possible future storylines, e.g. of greenhouse gas emissions
- ‘parameters’: different values for the control dials of the climate model, which affect the behaviour of things we can’t include as specific physical laws
- ‘multi-model’: different climate models from the different universities and meteorological institutes around the world
*** Too Much Information
**** Yes, we did reinvent a word, a bit.