One of the major functions of our brain is to process spatial information and to make this information available to our motor systems to interact successfully with the environment. Numerous studies over the past decades and even centuries have investigated, how our central nervous system deals with this challenge. Spatial information can be derived from vision. We see, where the cup of coffee stands at the breakfast table or where the un-mute-button of our video-conference tool is. However, this is always just a snapshot, because the location of the projection of the cup or the un-mute-button shifts across the retina by each eye movement, i.e., 2–3 times per second. So, where exactly in space are objects located? And what signals guide self-motion and navigation through our environment? While also other sensory signals (vestibular, tactile, auditory, even smell) can help us localize objects in space and guide our navigation, here, we will focus on the dominant sense in primates: vision. We will review (i) how visual information is processed to eventually result in space perception, (ii) how this perception is modulated by action, especially eye movements, at the behavioral and at the neural level, and (iii) how spatial representations relate to other encodings of magnitude, i.e., time and number.