Soooo... that means they're not fully grown yet.
In a small tank, watch the water quality if you mix male/female, as they breed A LOT!!! It's easy for a small tank to become over populated quickly, which will effect water quality just as quickly.
For this being only a 20 gallon might I suggest either 1 male alone or 2 males with lots of decorations? This would avoid breeding, and help you to keep population and sizes in check with the size of the tank. 2 male swordtails can live fine together so long as they have decorations to use to get away from each other when needed.