They can't bypass gates, I would recommend using those instead. If you're living in a traditional house, maybe make a wall around it and put a gate somewhere. Banners are mostly meant to protect from other players, not from NPCs. The only protection against NPCs they provide is that they prevent invasions and enemy spawns within the protected zone.

They can still open the doors.