Purpose. The aim of this study was to determine the relationships between physical fitness and various aspects of quality of life in middle-aged adults and the elderly. Methods. The sample included 216 women and 43 men, living in a medium-sized city in Poland aged 50-84 years. Physical fitness was determined using a test battery specifically designed for the elderly (Senior Fitness Test); grip strength by the dominate hand was also measured. The short form of the WHOQOL-BR EF questionnaire was used to evaluate quality of life. Four domains (physical, psychological, social relationships, environment) of quality of life as well as overall quality of life and health were self-assessed. Results. Among the selected components of physical fitness measured in the study, the assessment of overall quality of life in women was related to upper limb strength and, depending on the quality of life domain, also with upper or lower body flexibility. In men, the only factor influencing their overall assessment was upper body flexibility, whereas cardiorespiratory efficiency was the only factor influencing their detailed self-assessment of the physical health domain. Additionally, the self-assessment of general health in men was significantly positively correlated with cardiopulmonary efficiency. In women, the general health assessment was correlated with upper limb strength and upper body flexibility. Conclusions. It was concluded that after the fifth decade of life, physical fitness plays a greater role in improving the self-assessment of quality of life in women than men. Physical activity undertaken by middle-aged and elderly adults should focus on first improving cardiorespiratory efficiency and then strength and flexibility.