Threaded fastening operations are widely used in assembly and are typically time-consuming and costly. In low-volume, high-value manufacturing, fastening operations are carried out manually by skilled workers. The existing approaches are found to be less flexible and robust for performing assembly in a less structured industrial environment. This paper introduces a novel algorithm for detecting the position and orientation of threaded holes and a new method for tightening bolts. First, the elliptic arc fitting method and the three-point method are used to estimate the initial position and orientation of the threaded hole, and the force impact caused by switching from the free space to the constrained space during bolt tightening is solved. Second, by monitoring the deformation of passive compliance, the position information is introduced into the control process to better control the radial force between the bolt and the threaded hole in the tightening process. The constant force controller and orientation compliance controller are designed according to the adaptive control theory. A series of experiments are carried out. The results show that the proposed method can estimate the initial position and orientation of an M24 bolt with an average position error of 0.36 mm, 0.43 mm and 0.46 mm and an orientation error of 0.65°, 0.46° and 0.59°, and it can tighten the bolt with a success rate of 98.5%.