Sideloading apps leaves you and your criteria as the only defense against malware or mailicious code. Download apps from a safer source, such as App World should guarantee a level of quality.
Then again, look at Android and their cesspool called Android Marketplace, now Play. Half the apps are pirated IP and the other half are trojan horses pretending to be something they are not while they hack your info. Also, there's a fart app if you look for it.
Sorry for the bummer. From a technical perspective, there is now downside to sideloading; there is no degradation from the mere act of sideloading or setting your BBPB in developer mode. Performance degradation is possible depending on the app you installed.