Halo 5 Lag Issue Multiple Consoles

So I have a cable modem (in bridge mode), a new AC gigabit Ethernet router and a gigabit Ethernet switch.
When I play Halo 5 it seems pretty good, when my roommate turns on his Xbox and fires up Halo 5, neither of us can play. We can get matches without a problem, it just 20 second spans of lag and then dead, respawn, take a couple steps, repeat.
Our NAT shows open and we have tried wireless ( we keep them wired) with the same result.
Anyone out there have any suggestions?