My reasoning is that if you can push a sight with a nylon punch, it's not up to specs for an interference fit and could drift on its own with recoil or rough handling.
Sights need to be tough to drift to stay accurate under use.
Sights need to be tough to drift to stay accurate under use.