My comments will pick up on what others have mentioned. First, the brown algae (diatoms) is nothing to worry about, this is common in new tanks (during first 3 months or so). Light actually has nothing to do with this particular issue, but I won't get into that as there are other more significant issues.
First, the molly. This is likely due to the soft water. If the pH is below 7, it is acidic, and usually this goes along with soft water. For more general background info on hardness and pH you can read here:
http://www.tropicalfishkeeping.com/freshwater-articles/water-hardness-ph-freshwater-aquarium-73276/
Livebearers absolutely must have medium hard or harder water, and the pH must be above 7. As you tap water would appear to be the opposite, I would stay with fish that appreciate that: your tetra will thrive, as will almost every fish except for any livebearers. The molly will die, expect it, as ammonia at any level is toxic to this fish and it is already weakened by the stress of the soft water on its physiology. Stay with soft water fish.
Ammonia above zero is trouble, but this is likely due to the newness of the tank and fish issues. Live plants will solve this. I wold suggest floating plants first, they are fast growing and thus use a lot of nutrients, and plants get their nitrogen from ammonia, so they grab it fast. Some Water Sprite is ideal, but many stem plants can be floating. By the way, you see Water Sprite shades, that means this species is in our profiles, so you can click the name to see info on this species. Similarly for Common Molly, Neon Tetra, etc. Profiles are under th second tab from the left in the blue bar at the top of the page.
Salt, never in a freshwater aquarium. Water changes will reduce the salt bit by bit, so please don't add more. You can read what salt does to freshwater fish here:
http://www.tropicalfishkeeping.com/freshwater-articles/salt-freshwater-aquarium-97842/
On the light, when you have live plants (I really do recommend this, nothing helps your tank as much as live plants) an 8 hour day is sufficient. We can discuss tubes/bulbs for your light later, some are better for plants. For plants and fish, a consistent light period is best, so choose when you are normally home to view the fish and have the 8-hour period within that time.
Your 20g is fully stocked with the tetra and catfish, even without the livebearers. If the store will accept the platy back, even for nothing, I would; the molly too, except it is obviously unhealthy and they likely won't take it.
Hope this helps.
Byron.