Bless your heart, we had this happen too! It is easy to tell the difference between a male and female platy. Unless you do something about this, it will not stop and just get worse, trust me. What we did is ran an ad in our local online Craigslist and sold some but mostly gave away all of our females. The females were all getting chased like crazy anyway and we had a lot more females than we did males. As soon as you can tell the difference in the sex (or sometimes right before then) they are capable of breeding so what them closely. Either way, I would re-home either all of the females or all of the males as soon as you can tell what they are.
This is how you tell the difference between male and female
the Smiley Fish