Generalizably solving complex problems involves decomposing them into simpler components and combining these parts in effective ways to solve new instances. The hippocampal complex has been a rich playground for understanding how the brain constructs and combines modular structures for flexible computation. This is because the hippocampus and associated areas generate strikingly explicit emergent representations of abstract (latent) low-dimensional variables in the domain of spatial navigation that form the elements of spatial inference but are not directly specified by the world. I will summarize mechanistic models of these circuits; describe recent progress in characterizing the rigid nature of these representations through unsupervised discovery of latent low-dimensional structure from population data; and show how these rigid and simple low-dimensional circuits can generate, in a highly flexible way, represenations and memory of different (spatial and non-spatial) variables, as seen in recent experiments. I will conclude with an overview of how understanding these circuits in the realm of navigation gives insights into their potential use in higher-dimensional non-spatial cognitive representations as well.