Never ending "setting up game" lobby

I am having trouble playing a game. The lobby will find players yet we just wait in the lobby setting up game. I can not back out by pressing the B button or anything. The only way to leave is to Dashboard. My NAT is open and I am on Xbox LIVE. What is the problem?