Cannot play in party with bro

So I’ve been trying to play matchmaking with my brother for some time now. When we first tried it was at the time nobody was able to have some patch. But we tried playing again last week, and every time we get into a party and start searching a playlist he’ll eventually drop out of searching and we can no longer party up. Or we’ll get a game and he’ll lag out and we can’t party up again. We opened his NAT, but it didn’t help at all.

We were finally able to be in an Xbox Live party and just chat while we play on our own, but we want to be able to play together. Something else that is weird is that his search times take forever, I can usually find games right away, but it could take 3-5 minutes or more for him to find a game. We both play over home wifi.

Anyone else have this problem? Fixes?

Thanks!

sometimes specific people cannot join on a specific game host due to connections but simply giving a different person the host makes the player able to stay/join. - but wifi is nowhere near as good as wired connections