When you are in lobby, if the cards say “incoming” you are ok and it will be a normal match. If it shows actual player names, go to your network settings and select “xbox live” and it will remove you form the lobby, no harm done. It doesn’t count as quitting so you aren’t going to get xp banned or anything weird like that.