Losing Connection (2 Xbox's In the Same House)

I have 2 Xbox’s hooked up to 2 TV’s so that friends and I can play together. We party up and play. It was working fine for a few days. Since last night one of us in constantly getting booted out half way through each game, saying connection to server has been lost. The game runs fine until one of us gets booted. The Xbox remains hooked up to Xbox live. I used this setup for Destiny and it worked fine. One xbox is wired and one wifi, this made no difference, it seems pretty random which one gets booted. I have the digital copy. If anyone can help that would be great, thanks.