Simplest and best advice so far. I had and have the same issue. Mine comes from gripping with my dominant hand too much, and when I press the trigger, my grip torques the gun to the left.
Your non dominant hand should be bearing the majority of the grip. All your shooting hand should be doing is guiding the trigger straight back. I still struggle with this. As Mike said, concentrate on the grouping first, the other is fine tuning.
The egg approach. A good one to follow.