While this is not a problem normally -- I can just use the menubar item to switch back -- it has happened a few times when I have been at the password entry box for a locked screensaver, from where the menubar isn't reachable.
I remembered that sometimes the input source menu item would hijack my Quicksilver shortcut (the system uses Command-Space for this by default). So I tried hitting Command-Space in the password dialog, and then entered my password. It worked! And even if you are running Quicksilver.

