Lead-free easy-cutting bismuth brass, which is the preferred material for 5G optical module gaskets, has the characteristics of environmental friendliness, good mechanical properties, good thermal conductivity and easy cutting. A coarse grain size and uneven distribution of the bismuth phase lead to poor cutting consistency, deep burrs and uneven colouration after dyeing black, etc. B, Ti, and Zr are commonly used refining modifiers in industry and can refine grains and microstructures, thus improving the cutting performance of bismuth brass. In this paper, a bismuth brass with a composition of 58.3wt.% Cu, 40.85wt.% Zn, 0.07wt.% P, 0.7wt.% Bi, and 0.08wt.% Sn was taken as the research subject, from which 0.1wt.% B, Ti or Zr was added. Macroscopic metallography, microscopic metallography and SEM were used to analyse the grains, microstructure and bismuth phase particles. The results show that the addition of 0.1wt.% B, Ti, or Zr has a significant refining effect on the grain size of the cast bismuth brass. The modifier-free sample has no equiaxial crystal, while the samples with 0.1wt.% added B, Ti, or Zr have equiaxial crystal sizes of 1.43, 0.95 and 0.71 mm, respectively. Therefore, the refining effect of adding Ti is slightly better than that of B but poorer than that of Zr. Moreover, the addition of B, Ti, and Zr also refines the intracrystalline α+β biphasic structure, with the refining effect of Zr being better than that of Ti and that of Ti being slightly better than that of B. In addition, the addition of B, Ti and Zr helps to eliminate the tendency of the bismuth phase to form film and improve the distribution of the bismuth phase. Among them, the effect of B is better than that of Ti or Zr. The cutting performance of the four samples decreases in the order B>Zr>Ti> modifier-free, the influencing factors of which include the bismuth phase and beta phase number density.