Unable to connect to friend lobbies

So, my friend and I have been trying to connect to a friend lobby for a while now, but with no luck, we keep both getting “Unable to connect to game session”. I just want to know if this is an issue on our end or if its a server-side problem.