I think this might be an issue caused by Quick Resume. Restarting your Xbox seems to fix the problem, but I don’t know if you can do a proper restart on the Series consoles (still using a One X). The only method I know of is to unplug and plug it back in.