The function that provides encryption and decryption of smtp passwords CRM/Utils/Crypt::encrypt
wants to use the crypt function included in the mcrypt php extension, but will just fallback to using just base64 encode if it isn't.
That means if you upgrade from a server with mcrypt to one without, or the reverse, or especially if you have it with for your webserver but not with your cli version of php, then you get a very confusing experience (yes, speaking from experience ...).
I'd propose two simple enhancements:
1. Add a warning to the status page if mcrypt is not available.
2. Generate a warning to the system log file every time encrypt runs (or decrypt runs) without mcrypt.
In addition, I think it might be worth checking if we can determine from the stored value itself whether it's been run through mcrypt or not, and fail the decryption if we're using the not of that value.
Somewhat related to CRM-16621. The original encrypt stuff was added in this issue: