My home is not a very open floor plan so toting a shotty through the house would be rather clumsy at best. Worse having been just woken up by a broken window or a foot hitting the front door, so handgun it is.
Its hard to beat the "chik rack" sound of an 870! But, a long barreled shotgun limits your mobility. Here's an idea...Both. I don't know how to rack a 870 with one hand, fire and still use the pistol, but it's always an option.
I prefer a shotgun with a large magazine and a short barrel (within legal limits of course), loaded with 00 Buck, with the last couple rounds being slugs.
I keep my 150 dollar 18 1/2 in barrel H&R Pardner pump loaded with 00 in the corner behind the door and the glock 32 in the night stand. The AR is in the closet. I keep loaded mags for the Glock and AR in several places
Think about all the news stories you read. When it's handguns: normally ends up with someone shooting at a bad guy. When home owner uses a shotgun: bad guy goes away DOA.