I feel like I caused your headache with the Serpae, since I suggested them in this case.
Most sources will say that in groups of 8 or more, the nipping is usually confined to within the group. But your experience shows this is not always so.
I don't know in how many threads I have advised members not to include this species in community tanks, and so often someone will tell me I do not know what i am talking about, they have them and they are fine, blah, blah. Well, this thread shows different. We cannot change the nature of fish by putting them in a closed system and expecting different behaviours. Evolution has made all of us the way we are inherently.
Sidluckman is correct on the numbers aspect of shoaling fish like the characins and cyprinids. Six is usually the minimum suggested for most species, but as I usually suggest, more is always safer and better. And the other aspect is space, and tankmates. The more space, the less trouble. And not including fish that will serve as a red flag to a bull. The Bleeding Hearts in with Angelfish is an example; this can work, perhaps in larger tanks, but it can also backfire. We had another member a while back who was setting up an angelfish display and included Bleeding Hearts that had to be removed within the week because they were tearing the angels to shreds.