Some factory sights are driven in with a machine press and are very tightly fit as it sounds like you experienced. I’ve heard that some people will resort to cutting through the sight to remove it from the slide.
XD's are notorious for being extremely tight. When you put it in the sight tool, there is a 50/50 chance that when you hear the "pop" it's either the sight or a slide rail. Jus' sayin'......