Chlorine will burn the gills and if strong enough will kill the fish within minutes. If you are not using anything now, either there is little or no chlorine in your tap water or you are not changing much of the tank volume. I recommend a partial water change of at least 50% every week, although this can depend upon the fish (type and number compared to the tank size).
Many areas also use chloramine to treat water, as it is a bit tougher than chlorine and does not dissipate as chlorine can. There may also be heavy metals in tap water, depending where the water comes from. And some contains ammonia, nitrite or nitrates. All but the last of these is highly toxic to fish and plants.
All of this varies from place to place. I know many aquarists in the UK don't use conditioners, their water is not heavily chlorinated. If you are in North America, you should certainly be using a good water conditioner, such as the ones mrdenim mentioned, or others. One that detoxifies chlorine, chloramine, heavy metals and ammonia is basic.