Fail to connect to matchmaking 40% of the time

When I queue to join a matchmade game about 40% of the time it gets to 4/4 players and then says unable to connect. None of my friends have this problem, at least not this frequently and it works fine most of the time.

This means I get kicked to the main menu and any friends that I was matchmaking with have to quit before the game starts and re-join my fireteam.

Has anybody else had this problem or are there any possible solutions?