That's because there's a glitch in the software. If you assign a custom ringtone to a person, it will ring when the phone's on vibrate. The sms/mms alerts for that person may go off as well.
With the new 5.0 s/w you can now have it vibrate and ring at the same time. Did you check your Vibrate Profile to make sure that the Volume is set to Silent?
If you look under options, Sounds and ring Tone, Play Sound was set to ALWAYS, the other two options are to play only IN the hoslser or play only Out of holster. they really should have another option to not play at all in vibrate mode. (Just saying)