Phonon scattering inhibits simultaneous near-unity efficiency and indistinguishability in semiconductor single-photon sources