Last week, I revealed the results of an experiment into an odd observation while brewing coffee in my Aeropress: why was it that the bubbles formed on the opposite side to the hand I used to pour the water from the kettle? On the face of it, it was an easy experiment, with a simple explanation and a fairly clear set of results. But behind this story is a series of decisions and psychology that can illustrate, on a small level, some of how experimental science is done. It’s not for nothing that there’s the saying, the devil is in the details.
Theory, experiment and the impartial observer
There can be an erroneous idea about the progress of science, repeated even among people who should realise the fallacy. A theory, with testable predictions is proposed, which is subjected to experiment by a series of dispassionate observers in order to provide evidence that either supports the theory or disproves it. We dehumanise the theoreticians and experimentalists to observers who can emotionally disconnect and observe the results from an objective distance.
There are countless examples against this within the history of science (both for theories that have now been rejected but also for theories that we still consider good models) but I want to keep to the example that we can all have in front of us in our kitchen: that of the bubbles in an Aeropress.
With the Aeropress it was an odd experimental result that prompted a theory that then fitted the odd observation. The theory came with some extra ‘predictions’, but theory and experiment evolved together. Again, there are examples of this in the history of science but the experiment prompted the theory that prompted further experimental tests.
The problem then is that the experimenter (in this case me) was well aware of the theoretical predictions. Could I dispassionately, and completely subconsciously, pour a kettle as I had always poured the kettle, or would part of me, however much my conscious was opposed, change how I poured the kettle subsequent to my idea of how the bubbles formed?
For the case of the experiment with the Aeropress, this remains an open question. Generally though, many experimentalists will aim to try to reduce conscious or unconscious biases by putting procedures in place to prevent them coming in. When Isaac Newton and John De Saguliers investigated the role of air resistance on falling masses from inside the dome of St Paul’s Cathedral, London, they dropped them from a trap door system. This meant that the masses (which in the first instance included glass balls filled with mercury) fell at the same time; the quiet suspicions of the experimentalist investigators could not influence the results. It created a mess on the floor of that great Cathedral, but it did eliminate this component of bias from the experiment. You can read more about their experiment here.
A need for peer review
Assuming that we are collecting data in a neutral way, what happens then? On the face of it, seeing if the bubbles appeared on the left hand side or the right hand side should be an easy question to answer. And in some cases, such as the pictures that I chose to illustrate my post about the results last week, the answer is clear. But are those photos representative of the whole data? And, for more ambiguous photos, such as the one shown here, how do you define which bubbles to count?
One problem here is that each photo is very slightly different. Either the angle is different, or there is steam on the lens, or the focus is not there. But even so, sometimes it is harder to see all the bubbles on an image. For this experiment I defined a minimum bubble size (which you can see as the white square in the image) which I used to decide which features on the surface of the coffee to ignore: after all, when viewing the image, it is not clear whether items smaller than this are bubbles or just a different colouration to the coffee crema.
You may notice that I did not mention this detail in last week’s post, but one of the images includes the square. This is one of those things that would (most likely) be picked up in what is known as ‘peer review’. When we write results up and submit it to a journal for publishing, the journal will typically send the paper out to 2 or 3 ‘referees’. These are people, who ideally work on similar experiments, who will read the paper and think “hang on a minute, what if it is not the bubbles but the bubble size that shows an effect, how have these authors counted the bubbles?” The example is admittedly a somewhat trite one, but the point is that the paper is read by someone who also does this sort of experiment and knows where problems can be encountered. The ideal is not to trip the authors up, or to show that they did anything wrong, but to see things from a new angle, a different set of obsessions and so ask the original authors to address points that improves the paper in the sense that we can all start to see what is going on*.
Peer review also of course helps to stop the publication of results that are wrong, or statistically invalid (see below). We therefore need some form of peer review in order that we can be collectively, as a society, happy that this science is being done robustly. So if you see a newspaper report that “the study, which has not yet been peer reviewed…” treat it with a very large pinch of salt and please don’t tweet it (unless you happen to also research that area and so can read the paper as if you are writing it).
Statistics…
We have attempted to eliminate our biases, we have been open and transparent about our methodology, what could possibly go wrong now? It is in not taking enough data. Say I made a coffee pouring from my right hand and the bubbles formed on the left, then with my left hand and the bubbles formed on the right, we can know that this is not enough to be sure that the bubbles ‘always’ form on the alternative side. For that bit of the experiment I made 22 coffees. Not enough to be statistically certain (more on that here), but probably enough for an observation on a coffee blog.
But the bit I want to focus on here is the part of the experiment where I counted the number of bubbles versus the bubble size. I was investigating any similarities with a study that measured thousands of bubbles over 225 images documenting 14 events. I counted the number of bubbles on one small portion of one coffee that may not be representative of the coffee generally. Can we accept that as a valid procedure?
While I may not have counted enough bubbles here, one experiment (that can involve coffee) where there certainly were enough objects counted was in the determination of the mechanism behind Brownian motion. Brownian motion is the phenomenon by which small particles of dust or bits of coffee move in random directions on the surface of your cup. It happens because the molecules within the water of the coffee hit against the dust and impart a small momentum to the particles. Because there are many molecules moving in all sorts of directions, the resultant movement appears random. If we look through a microscope we can see the particles moving but there is no way that we could see the molecules that move them. Back in the nineteenth century this became an exceedingly controversial topic: could you form a scientific theory for a phenomenon (such as Brownian motion) which relied on assuming an underlying reality (molecules) that you could not hope to see or measure directly? The question was (partly) resolved only in the early twentieth century with the very careful experiments of Jean Perrin (you can read more about Perrin’s experiments and their relation to coffee here). When Perrin summarised his results he wrote:
experiment by Isaac Newton and John de Saguliers in 1710.
“I have counted about 11, 000 granules in different regions of different preparations to obtain the figure 21.2 of the first column.”
Which is slightly more than the number of bubbles I counted last week.
A way forward – truth and integrity
What does this mean for science and how science is done and reported, especially in this era of rapid research and in which everyone has an opinion? Is science discredited by the fact that we are humans, and not fully dissociated and objective, when we do it?
Although I ran out of space to discuss Michael Polanyi’s comments on statistics and pattern recognition, he does have something extremely relevant to say about the progress of science. For Polanyi, how we do science and how we behave as a society were (and are) intimately linked. He considered that for science to prosper, we needed “fairness and tolerance” in discussion. By fairness he meant the requirement to state your case, your experimental result or theory, openly, separating fact, from opinion and emotional involvement and openly allowing them each to be critiqued. By tolerance he meant that we needed to listen to the other, even while we disagree, in order to see where they may have a point. He linked this behaviour within science to the behaviour required of the public in listening (and sharing) science. As he said:
“Fairness and tolerance can hardly be maintained in a public contest unless its audience appreciates candour and moderation and can resist false oratory. A judicious public with a quick ear for insincerity of argument is therefore an essential partner in the practise of free controversy…“
Science and society move together.
An invitation
And so an invitation. Keeping in mind the idea of Polanyi about honesty and integrity in discussion, I would like to invite any reader of this blog to become a peer reviewer of the experiment reported last week. Please go and enjoy a coffee, carefully preparing and noticing your brewing technique and then work out how you would have made the experiment and tested any results. Perhaps you have a different theory that would require a slightly different counting method than the one chosen? Perhaps you think that more experiments are necessary? Become my peer reviewer! Feel free to comment below, or on Facebook or Twitter. Or, if you would prefer, email me through the contact form here. Bear in mind I am human, and so I will react to your report. But if you and I keep can Polanyi’s warning in focus, perhaps we can together improve our understanding of the science behind bubbles in an Aeropress. And, by extension, improve our understanding of how science, and society, can work.
I genuinely do look forward to reading your comments.
*I have worked in academia long enough to know that this is not always how the peer review process works in practise. There are many cases where peer review falls short of the ideal, for all sorts of reasons. But it remains a necessary part of the publication process as many referees (and authors) do try to approach the process in this way. Obviously emotion gets in the way when we receive the referee’s report initially, or, on the other side, if we think that the authors have seriously misunderstood their experiment, but if we take a few days to sit with the report/paper, we do try to get towards the ideal.