3v4 games in Team Slayer need to STOP

My 2 friends and I like going into Infinity Slayer together, but we get matched up against 4 people far too often. We will even have a 3v3 and it waits to find one more to make it a 4v3. It is incredibly annoying and not very fair. Does this happen to other people often?