Human reasoning is complex, messy, and approximate. As a result, it has been subject to a millennia-long enterprise to extract principles that are simple, neat, and impeccable. This enterprise is incomplete; humans routinely perform acts of reasoning that remain both poorly understood and beyond the capabilities of modern methods of artificial intelligence. Specifically, humans mentally model the causal structure of the world, and manipulate these models to imagine worlds that could have been but were not, and even worlds that could never exist in reality.  Remarkably, these imaginary worlds, both possible and impossible, shape our beliefs about the real world.

This thesis investigates the proposition that common sense knowledge can be encoded as programs, and that imaginative reasoning constitutes structural manipulations to these programs. Concretely, we introduce probabilistic programming languages – languages in which causal probabilistic models are encoded as programs – with two new forms of inference. The first is distributional inference, which means to reason with statistical information rather than observational data. This allows us to address problems of algorithmic fairness and robustness, and to perform Bayesian parameter estimation using data about probabilities, expectations, and other distributional properties. The second is causal inference, which allows us to reason about counterfactual what-if scenarios and  causation, i.e., whether some event A is the cause of some other event B, in complex simulation models.

We introduce a number of new algorithms for reasoning under uncertainty. Unlike traditional approaches, our methods modify the internal structure of the model or reinterpret how it is executed. We introduce parametric inversion, which inverts the causal structure to literally run programs in reverse from observations to causes, and predicate exchange, which relaxes Boolean operators to make inference more tractable. Collectively, these contributions shrink the gap between human and machine reasoning, and serve as practical tools for scientific modelling and inference.