I find it odd that when I quickplay as a host I often do not make it through the 3 second countdown before a person joins. Their joining then delays me loading into the game, and if they stay I have to wait for them to wander over.
If the game was not available to join until after the countdown expired I could load into the level and so could they, reducing the time spent not actually playing.
Additionally I would be in favor of an auto-leave option within settings to auto drop the group after the scoreboard. That would prevent loading into their keep, leaving, and then loading into my own keep.