Musicians with extensive training and playing experience provide an excellent model for studying plasticity of the human brain. The demands placed on the nervous system by music performance are very high and provide a uniquely rich multisensory and motor experience to the player. As confirmed by neuroimaging studies, playing music depends on a strong coupling of perception and action mediated by sensory, motor, and multimodal integration regions distributed throughout the brain. A pianist, for example, must draw on a whole set of complex skills, including translating visual analysis of musical notation into motor movements, coordinating multisensory information with bimanual motor activity, developing fine motor skills in both hands coupled with metric precision, and monitoring auditory feedback to fine-tune a performance as it progresses. This article summarizes research on the effects of musical training on brain function, brain connectivity and brain structure. First we address factors inducing and continuously driving brain plasticity in dedicated musicians, arguing that prolonged goal-directed practice, multi-sensory-motor integration, high arousal, and emotional and social rewards contribute to these plasticity-induced brain adaptations. Subsequently, we briefly review the neuroanatomy and neurophysiology underpinning musical activities. Here we focus on the perception of sound, integration of sound and movement, and the physiology of motor planning and motor control. We then review the literature on functional changes in brain activation and brain connectivity along with the acquisition of musical skills, be they auditory or sensory-motor. In the following section we focus on structural adaptions in the gray matter of the brain and in fiber-tract density associated with music learning. Here we critically discuss the findings that structural changes are mostly seen when starting musical training after age seven, whereas functional optimization is more effective before this age. We then address the phenomenon of de-expertise, reviewing studies which provide evidence that intensive music-making can induce dysfunctional changes which are accompanied by a degradation of skilled motor behavior, also termed “musician’s dystonia”. This condition, which is frequently highly disabling, mainly affects male classical musicians with a history of compulsive working behavior, anxiety disorder or chronic pain. Functional and structural brain changes in these musicians are suggestive of deficient inhibition and excess excitation in the central nervous system, which leads to co-activation of antagonistic pairs of muscles during performance, reducing movement speed and quality. We conclude with a concise summary of the role of brain plasticity, metaplasticity and maladaptive plasticity in the acquisition and loss of musicians’ expertise.