"Unable to join" message when queuing with friend

We have two Xboxes, same house, same internet, OG Xbox One is wired, Xbox One X is wireless with strong connection. Every single time we boot up Halo 5, party up and search for a match in multiplayer the first two or three attempts the game will partially load and then one of us will get the error message “unable to join” and the other will load into the match. We press home button and force quit the game until it works. After if it works we no longer have the problem again for the remainder of the session. When we turn off the Xboxes and try to play again later then we have to go through the same ritual again. What is going on and how do we fix it?