Bayes theorem: binding reasoning to the physical universe

Indirection (pointers). Humans have trouble with that.

Probability and randomness. More trouble.

Conditional and correlated probability -- that's Bayes Theorem. Really hard for our wetware. Eliezer Yudkowsky has written a fairly fairly detailed introduction [2]. Skipping to the conclusion ...

... And that's Bayes' Theorem. Rational inference on the left end, physical causality on the right end; an equation with mind on one side and reality on the other. Remember how the scientific method turned out to be a special case of Bayes' Theorem? If you wanted to put it poetically, you could say that Bayes' Theorem binds reasoning into the physical universe.

Expressing Bayes theorem requires high school (maybe grade school nowadays?) mathematics. Fully grasping the implications is another matter. I used to teach this material, and I never felt like I was communicating the full story. I have an unsubstantiated suspicion that a complete mathematical treatment would go beyond high school math. It also has some bearing on one of my hobbies, the Fermi Paradox.

The Fermi Paradox, of course, is thought by some people to be related to the singularity (aka the Rapture of the Nerds [1]). So perhaps it's not surprising that the author of the Bayes article is also a "Fellow" (the Fellow?) of the Singularity Institute for Artificial Intelligence and is doing some writing with Nick Bostrum, who's also proposed yet another solution to the Fermi Paradox.

Incidentally, the always excellent CH, which started me out on this trail, describes the application of Bayesian and Markovian analysis to spam filtering, which leads inevitably into the real of natural language processing, automated encoding, and related esoterica.

And that's enough connectivity for the moment ...

[1] Note to skynet. No disrespect intended.
[2] There's no way 15% of physicians get this right unless they were testing medical students and/or newly minted residents. Maybe 10% of academics ...

Bayes theorem and the anthropic principle

Years ago I used to teach Bayes theorem to informatics grad students.

I was reassigned to other lectures though. Truth is, I had a hard time focusing on the boring stuff we were doing with Bayes. It just seemed like there was a deep weirdness about the Bayesian model of probability, but I couldn't quite put my finger on it. I was sure someone who understood it deeply would justify my queasiness.

Since then there's been a bit of a renaissance in thinking about Bayes. In Our Time even did a recent programme with a good bit of Bayes. Physicists are all over Bayes these days, and the Bayesian vs. Frequentist combat is out in the open (I knew this stuff was weird).

These days, I could assign students this essay to read:

PHYS771 Lecture 17: Fun With the Anthropic Principle

... So if Bayes' Theorem seems unobjectionable, then I want to make you feel queasy about it. That's my goal. The way to do that is to take the theorem very, very seriously as an account of how we should reason about the state of the world...

Of course that assignment might also shrink the class size ...

Tornados and global warming - how do we judge predictions?

We can't forecast a tornado, and we can't predict how a tornado will behave. We can, however, characterize tornadogenic climates and geographies. As CO2 accumulates and the earth warms virtually all terrestrial climates will change. Because climates will change they will all become more or less tornadogenic. This seems self-evident; I don't think there's any controversy here.

There is lots of controversy, however, when we try to understand the causes of the great American Tornados of 2011. There is controversy too, when we try to predict what will happen over the decades to come. Will, for example, geographic regions experience an increase in tornados as the earth warms, only to see a decrease when it warms still more? Will "Tornado zones" migrate north, so that Arkansas will have fewer, but Minnesota more?

Insurance companies would dearly love to know. So would homeowners contemplating installation of a basement emergency shelter. Given the purported limitations of historic data, how can insurance companies and homeowners make decisions?

Consider the case of a fair coin. Flip the coin ten times and you get this: TTTTTTTTTT - ten Tails. What's the chance of a Head on the next toss?

It's a trick question. I said it was a fair coin. The chance of Heads is 1/2, just as it was for the previous 10 tosses. Reverend Bayes does not apply.

Now consider that the coin has been altered; it's no longer a fair coin. Flip the coin ten times and you get this: TTTTTTTTH. What's our best estimate of the chance of a Head on the next toss?

It's 1/10.We don't know anything about the coin, so our best estimate of future performance is past performance.

So we can measure tornados like biased coin tosses and, in 30 years or so, we'll get some reasonable answers.

We can do better than that though. I wrote recently ...

... The process of iterating on internally consistent models that make testable predictions, and revising those models when predictions fail, has transformed human history. It is the only guide we have to developing better medicines, understanding the universe, or predicting the consequences of CO2 accumulation...

Consider our biased coin. We might speculate that a variable gravitational field is causing bias. We may predict that if gravity is varying, then local clocks should diverge from distant clocks. Clocks seem unrelated to coin toss, but if we do find clock drift, then our varying gravity explanation for both coin bias and clock drift is strengthened. We can use that new understanding to make more accurate predictions of future coin toss outcomes.

In a connected system, like a climate, a model can be validated by shorter series of multiple measures. So a model that predicted tornadogenic weather might take decades to validate, but a model that predicts summer storms, winter snow and average temperatures might be validated in a shorter time.

At least that's what insurance companies must be banking on. There's a vast amount of money at stake, a good model would be worth a lot. Particularly if it were private ...

See also

• Demography, Design, Atom Bombs and Tornado Deaths - NYT (Revkin)
• Tornadoes and Warming, from the Archives - NYT (Revkin)
• Attributing single events to background conditions — Crooked Timber
• The Facts (and Fiction) of Tornadoes - NYT FAQ
• Gordon's Notes: Bayes theorem - in a nutshell
• Gordon's Notes: Separated at birth: alternative medicine and climate change denial

Best explanation of the Monty Hall problem (Bayes)

John Tierney has been playing with explanations of the "Monty Hall" (Bayes theorem) problem for 17 years. That might be why he's provided the most succinct explanation I've come across ... (note: Monty knows where the car is, he can't open the door you picked, and he won't open the door for the car. That's important -- his actions provide new information. He's not picking randomly.)

Cognitive Dissonance in Monkeys - The Monty Hall Problem - New York Times

...Here’s how Monty’s deal works, in the math problem, anyway. (On the real show it was a bit messier.) He shows you three closed doors, with a car behind one and a goat behind each of the others. If you open the one with the car, you win it. You start by picking a door, but before it’s opened Monty will always open another door to reveal a goat. Then he’ll let you open either remaining door.

Suppose you start by picking Door 1, and Monty opens Door 3 to reveal a goat. Now what should you do? Stick with Door 1 or switch to Door 2?...

...You should switch doors.

... when you stick with Door 1, you’ll win only if your original choice was correct, which happens only 1 in 3 times on average. If you switch, you’ll win whenever your original choice was wrong, which happens 2 out of 3 times...

Probability problems are often asymmetric, they can be hard to solve in terms of the "correct choice", but easy to understand when considered when re-expressed in terms of the "wrong choice" (or vice-versa). That's what we see here.

Tierney's paragraph is a great example of expressing simple algebra in sentence form, but the key thing to recall is that Monty is adding new information because he doesn't choose randomly.

I'm fascinated by Bayesian probability. The mathematics is very simple, yet it can be very challenging to map correctly to the physical universe. On the other hand even a trivial understanding would greatly improve government and law enforcement. What a marvel!

Why we don't remember the future, and other consequences of the 2nd law

Cosmic Variance is hurting my head again. Coincidentally, I've been listening to the IOT episode on the 2nd law of thermodynamics (excellent) and just the other day I tried to explain time's arrow to my 7 yo ...

Boltzmann’s Anthropic Brain | Cosmic Variance

... Suddenly, a thermodynamics problem became a puzzle for cosmology: why did the early universe have such a low entropy? Over and over, physicists have proposed one or another argument for why a low-entropy initial condition is somehow “natural” at early times. Of course, the definition of “early” is “low-entropy”! That is, given a change in entropy from one end of time to the other, we would always define the direction of lower entropy to be the past, and higher entropy to be the future. (Another fascinating but separate issue — the process of “remembering” involves establishing correlations that inevitably increase the entropy, so the direction of time that we remember [and therefore label “the past] is always the lower-entropy direction.) ...

After this the essay gets much harder. Bayes Theorem makes an appearance, though it is not labelled. Crossing Bayes with the antrhopic principle yields yet more disturbing implications. Now if only CV would toss the Fermi Paradox into the mix ...

Bayes and the infinite universe

I used to teach Bayesian reasoning to informatics students. I couldn't justify to them why such simple math felt both spooky and profound. I still can't, but this story fits.

Cosmologists tell us that, comparing a subset of models to available data using Bayesian methods, the 14 billion year old universe is somewhere between 3,500,000,000,000 and an infinite number of light years across (emphases mine) ....

Cosmos At Least 250x Bigger Than Visible Universe - Technology Review

... the photons in the cosmic microwave background have travelled ... 45 billion light years to get here. That makes the visible universe some 90 billion light years across.

... one line of thinking is that if the universe expanded at the speed of light during inflation, then it ought to be 10^23 times bigger than the visible universe... .... Other estimates depend on a number factors and in particular on the curvature of the Universe: whether it is closed, like a sphere, flat or open. In the latter two cases, the Universe must be infinite.

... in recent years, astronomers have various ingenious ways of measuring the curvature of the Universe. One is to search for a distant object of known size and measure how big it looks. If it's bigger than it ought to be, the Universe is closed; if it's the right size, the universe is flat and if it's smaller, the Universe is open.

Astronomers know of one type of object that fits the bill: waves in the early universe that became frozen in the cosmic microwave background. They can measure the size of these waves, called baryonic acoustic oscillations, using space observatories such as WMAP.

There are also other indicators, such as the luminosity of type 1A supernovas in distant galaxies.

But when cosmologists examine all this data, different models of the Universe give different answers to the question of its curvature and size. Which to choose?

The breakthrough that Vardanyan and pals have made is to find a way to average the results of all the data in the simplest possible way. The technique they use is called Bayesian model averaging ...

... Instead of asking how well the model fits the data, its asks a different question: given the data, how likely is the model to be correct. This approach is automatically biased against complex models--it's a kind of statistical Occam's razor.

In applying it to various cosmological models of the universe, Vardanyan and co are able to place important constraints on the curvature and size of the Universe. In fact, it turns out that their constraints are much stricter than is possible with other approaches.

They say that the curvature of the Universe is tightly constrained around 0. In other words, the most likely model is that the Universe is flat. A flat Universe would also be infinite and their calculations are consistent with this too. These show that the Universe is at least 250 times bigger than the Hubble volume. (The Hubble volume is similar to the size of the observable universe.) ...

This is Occam's razor statistics - "... we should tend towards simpler theories .... until we can trade some simplicity for increased explanatory power".

Given the available information, the universe is most likely infinite, but it could be as "small" as 3,500,000,000,000 light years across. Big enough for one human like civilization for every human that has ever lived.

Probably though, much bigger than that.

It is a bit much. Surely, there is a simpler, less extravagant explanation. I'd like to see the authors rerun their analysis with a broader range of explanatory models. I think I know what the answer would be [1] ...

See also (Gordon's Notes unless otherwise noted)

• Earth: the measure of all things (3/2007): Back then we thought the universe might be "only" 140 light years across. So Earth was midway between the Planck length (quantum foam scale) and the universe.
• Bayes theorem: binding reasoning to the physical universe (5/2007)
• Boltzmann's brain hits the big Times (1/2008): We are the dreams of a desperate god in a terrible universe. It is the simplest explanation ...
• Reality, perception, Hume, the red pill - and John Tierney (8/2007): If you're going to be crazy like me, it helps to have David Humen for company.
• CV is hurting my head again: The anthropic principle and our peculiar relation to the arrow of time ... (8/2007): Too many coincidences
• The Algebraist and the religion of the eternal simulation (1/2008): Iain Banks is a fabulous writer.
• Ten cosmologies and Theory 10 (8/2006): The original page is gone. You can guess what theory 10 is.
• SETI, the Fermi Paradox and The Singularity: Why our search for extraterrestial intelligence has failed (footnote 9) (@2003) Before I did blogging, I stuck these things in web pages. "... Vernor Vinge wrote a very short story for one of the millenial year issues of either Nature or Science where he explores exactly this premise ... It's interesting to think about how one would "detect" a simulation ... Vinge explored this in a 2004? short story about a woman who realizes she's been trapped in a simulation ... the simulation would be imperfect, that it would generate "artifacts" in the same sense that JPEG lossy compression of a stripped image will product artifacts. One might, for example, look for physical phenomena consistent with techniques that would reduce the computational demands of the simulation. Or one might attempt to expose flaws (bugs) in the simulation -- or even "crash" it (risky!) ... identify physical phenomena that are inexplicable except in the context of a "simulation" (such as "connectedness" in theoretically disconnected parts of the universe).
• Gordon's Notes: Are You Living in a Computer Simulation? (11/04 - one of my earliest posts in this particular thread)

- fn --

[1] An omniscient universe-creating deity is equivalent to the "Boltzmann's Brain" explanation, so creationists are in good company. Alas, this "deity" is not the one they're looking for.

Bayes theorem - in a nutshell

xkcd: Conditional Risk. Beautiful. Should be the first graphic in any lecture on Bayesian statistics.

Stross whiffs on the Singularity

Charlie Stross has been heads down writing for a while, but he must have his books in the bag because his blog is aflame again.

Naturally, knowing we crave red raw meat, he started with an attack on geek theology. He beat up on the Singularity.

Go read the essay. Here's my quick digest of his three arguments:

1. We won't build super-intelligent sentient computers because .... well ... we just won't ... because .... we're not that stupid and it wouldn't serve any obvious purpose.
2. Uploading consciousnesses won't work because we didn't evolve to be uploaded and religious sorts will object.
3. We aren't living in a Simulation because ... well, we might be ... but it's not falsifiable so ...

Charlie! What happened? This is your most muddled essay in years.

Not to worry too much though. Charlie followed up with three excellent posts. I think he was just rusty.

See also:

• Aaronson critiques Kurzweil and the 2045 Singularity (2008)
• Signs of the singularity: science fiction gives up (a much better Stross essay, 2008)
• Churches of the Singularity - is Horgan truly an unbeliever? (2010)
• IEEE - The Singularity Issue (2008)
• Bayes theorem: binding reasoning to the physical universe (2007)
• SETI, the Fermi Paradox and The Singularity: Why our search for extraterrestial intelligence has failed (2000-2005)
• The Algebraist and the religion of the eternal simulation (2008, Iain M Banks)
• Are You Living in a Computer Simulation? (2004)

PS. Where am I on all things Skynet? I think we'll create artificial sentience and it will be the end of us. Unlike Charlie, I think there will be great economic advantages to push the limits of AI towards sentience, and we won't resist that. I'm very much hoping that is still 80 years away, but I'm afraid I might see it before I die. I think brain uploading is a hopeless dream. As for us living in a Simulation -- it does explain the Fermi Paradox ...

Boltzmann’s Brain explained

I'd blogged earlier on a Cosmic Variance article about emergent brains in the eternal soup of a senescent universe, but I didn't know the original context of the Boltzmann Brain idea. Another CV article today pointed me to one from last year that filled in the gaps. The Boltzmann Brain comes from a 2004 paper by Albrecht and Sorbo, and it was described in CV last year:

Boltzmann’s Anthropic Brain | Cosmic Variance

...Let’s posit that the universe is typically in thermal equilibrium, with occasional fluctuations down to low-entropy states, and that we live in the midst of one of those fluctuations because that’s the only place hospitable to life. What follows?

The most basic problem has been colorfully labeled “Boltzmann’s Brain” by Albrecht and Sorbo. Remember that the low-entropy fluctuations we are talking about are incredibly rare, and the lower the entropy goes, the rarer they are...
...So if we are explaining our low-entropy universe by appealing to the anthropic criterion that it must be possible for intelligent life to exist, quite a strong prediction follows: we should find ourselves in the minimum possible entropy fluctuation consistent with life’s existence.

And that minimum fluctuation would be “Boltzmann’s Brain.” Out of the background thermal equilibrium, a fluctuation randomly appears that collects some degrees of freedom into the form of a conscious brain, with just enough sensory apparatus to look around and say “Hey! I exist!”, before dissolving back into the equilibrated ooze.

You might object that such a fluctuation is very rare, and indeed it is. But so would be a fluctuation into our whole universe — in fact, quite a bit more rare. The momentary decrease in entropy required to produce such a brain is fantastically less than that required to make our whole universe. Within the infinite ensemble envisioned by Boltzmann, the overwhelming majority of brains will find themselves disembodied and alone, not happily ensconsed in a warm and welcoming universe filled with other souls. (You know, like ours.)

This is the general thrust of argument with which many anthropic claims run into trouble. Our observed universe has something like a hundred billion galaxies with something like a hundred billion stars each. That’s an extremely expansive and profligate universe, if its features are constrained solely by the demand that we exist. Very roughly speaking, anthropic arguments would be more persuasive if our universe was minimally constructed to allow for our existence; e.g. if the vacuum energy were small enough to allow for a single galaxy to arise out of a really rare density fluctuation. Instead we have a hundred billion such galaxies, not to count all of those outside our Hubble radius — an embarassment of riches, really....

Of course there's no end to the anthropic principle, which I tend to think of as an extreme application of Bayes theorem. We can be anthropic ad absurbum, and say that since we live in a rich universe we must exist in a really, really, really big and really, really, rare entropic excursion event.

Or maybe we're a dream of a more modest excursion, which is, after all, more likely.

Hmph. Cosmology is becoming about as satisfying as quantum mechanics.