On quickplay, once the map has started, press tab and tick the “public” checkbox to set it to private instead. Despite saying “private”, it is actually “friends only”. Steam friends can join your lobby at any time, even without an invite. Vermintide 2 has no “true private” or “invite only” lobby setting.
Use the Steam block functionality:
Go to Steam → View → Players to see the Steam profiles of players you’ve recently played with, so you can block them.