I have observed this with myself as the host as well as 2 other friends. Whenever we start Quick Play in a group of two or three, it can take extremely long for others to get joined into our session. Often we end up playing most of a mission with just us and 1-2 bots until near the very end.
Not sure when exactly this started, I think some time between updates 1.3 and 1.4, and it seems to have slowly become worse over time. As I write this we are sitting there two people together in the keep, queue for Champion Quick Play, waiting already more than 10 minutes with no one showing up. It used to take less than a minute to fill this up. (We have experienced the same effect on Veteran Quick Play, although not as often.)
However, when selecting Quick Play alone, it usually doesn’t last long until it puts any of us into another session.
For reference, these are the network settings, and this is in the EU region: https://i.imgur.com/zqqpTvL.png