There is currently a lack of information adjacent on the influence of sex and age in heart rate variability (HRV), adjusted according to accelerometer-based physical activity (PADL). We hypothesized that the effect of sex and age on the HRV should be reduced or absent in individuals with a suitable PADL level. We aim to evaluate the influence of sex and age on HRV, adjusted for the confounding effects of the PADL level. A total of 485 age-stratified subjects (18-39, 40-59, and ≥60 years) underwent HRV analyses at rest and 7-day assessments of accelerometer-based PADL. Multivariate analyses of covariance were done using log-transformed HRV indices as outcomes, age and sex as fixed factors, and PADL, cardiovascular risk, fat body mass, and heart rate (HR) at rest as covariates. Despite the adjustment for directly measured PADL, women had better indices of vagal tone, whereas men had higher sympathetic influence. Also, compared to middle-aged and older adults, younger individuals (ages 18-39 years) presented better HRV. Multiple regression analyses confirmed that age and sex were the main predictors of HRV indices, even after adjusting for PADL directly assessed by triaxial accelerometer and HR. We also observed that the correlation between some HRV indexes and the different indexes of physical activity directly evaluated was significant, but not very consistent. Thus, HRV indices are influenced by age and sex, regardless of accelerometer-based physical activity. Interventions with physical activity and exercise aimed at improving the autonomic modulation of asymptomatic adults should take such differences into account.