It has been worse lately. Have you tried a hard reset on the xbox after it happens? if you want to try something else…
In router settings:
Give the xbox a unique assigned IP address. - Turn off automatic UPNP - Create a port forwarding rule for the ports required to use XBL
Port 88 (UDP) - Port 3074 (UDP and TCP) - Port 53 (UDP and TCP) - Port 80 (TCP) - Port 500 (UDP) - Port 3544 (UDP) - Port 4500 (UDP)
Select the XBOX One as the Host for the above portsI am wired, with an open NAT, always been. Even with the above I have to hard reset my xbox once in a while. Good luck.