Smart Scope bugs me a Bit.

It would literally bug me to death, whenever I’d sometimes use Smart Scope and go back to aiming from hip (Although it doesn’t not affect your aim) sometimes go back into Smart Scope the moment I get out of it. Has this occurred to anyone else or not?