We evaluated the direct release to the environment of a number of potentially toxic elements (PTEs) from various processing nodes at Xikuangshan Antimony Mine in Hunan Province, China. Sampling wastewater, processing dust, and solid waste and characterizing PTE content (major elements Sb, As, Zn, and associated Hg, Pb, and Cd) from processing activities, we extrapolated findings to assess wider environmental significance using the pollution index and the potential ecological risk index. The Sb, As, and Zn in wastewater from the antimony benefication industry and a wider group of PTEs in the fine ore bin were significantly higher than their reference values. The content of Sb, As, and Zn in tailings were relatively high, with the average value being 2674, 1040, and 590 mg·kg−1, respectively. The content of PTEs in the surface soils surrounding the tailings was similar to that in tailings, and much higher than the background values. The results of the pollution index evaluation of the degree of pollution by PTEs showed that while dominated by Sb, some variation in order of significance was seen namely for: (1) The ore processing wastewater Sb > Pb > As > Zn > Hg > Cd, (2) in dust Sb > As > Cd > Pb > Hg > Zn, and (3) surface soil (near tailings) Sb > Hg > Cd > As > Zn > Pb. From the assessment of the potential ecological risk index, the levels were most significant at the three dust generation nodes and in the soil surrounding the tailings reservoir.