Press B to Switch to Game Chat

I love the new functionality IN GAME on Halo 5 to hold the SquareSquare (Back/select) button and press B to toggle between party and game chat. This is extremely helpful since there is no way to communicate with your fireteam while searching for matches.

I have a problem that I don’t know how to solve, though. I am not in a party, but when my girlfriend and I fireteam up in the same room, it forces both of us to a pseudo-party chat. Nobody hears us, but if we hold the scoreboard and press B, the text shows “HOLD B TO SWITCH TO GAME CHAT.” We aren’t in party, so why does this happen?

We are able to switch it out as soon as the match starts, but for the first match, we often don’t realize that this is happening.

TLDR: Scoreboard + HOLD B TO SWITCH TO GAME CHAT is awesome, but seems to be forcing party chat for my fireteam unless we flip the switch ourselves after the match starts.
Is there an option I can toggle somewhere to change this?