Won’t let me switch to live in any lobby, my console is banned from matchmaking, not customs
Sounds like your entire console has been banned from Xbox LIVE play across all of Halo 3, not just Matchmaking.
When a console has been banned from online multiplayer, it means from all forms of online play. This includes customs. I’m afraid you are restricted to local play only (unless it is a temporary ban, in which case you will just have to wait it out).