Hole cold expansion and bolt clamping force are usually applied to improve the fatigue performance of bolted joints. In order to investigate the effects of hole cold expansion and bolt clamping force and reveal the mechanism of these two factors on the fatigue damage of bolted joint, a continuum damage mechanics (CDM) based approach in conjunction with the finite element method is used. The damage-coupled Voyiadjis plasticity constitutive model is used to represent the material behavior, which is implemented by user material subroutine in abaqus. The elasticity and plasticity damage evolutions of the material are described by the stress-based and plastic-strain-based equations, respectively. The fatigue damage of joint is calculated using abaqus cycle by cycle. The fatigue lives of double-lap bolted joints with and without clamping force at different levels of hole cold expansion are all obtained. The characteristics of fatigue damage corresponding to the different conditions are presented to unfold the influencing mechanism of these two factors. The predicted fatigue lives and crack initiation locations are in good agreement with the experimental results available in the literature. The beneficial effects of hole cold expansion and bolt clamping force on the fatigue behavior of bolted joint are presented in this work.