Game freezes why trying to join party

I’m not sure if this is the right place to post bugs, please let me know otherwise.

When I’m invited to a game, and I hold down the Xbox button on my controller (the guide button) then I am presented with a loading screen telling me the game is attempting to join the lobby. If I’m sent another invite from the same person for the same game and happen to hold down the guide button again, the game freezes and I’m stuck on the loading screen forever.