I wouldn't be very worried about germs or bacteria. For the most part diseases that affect us will have little effect on your fish and vice versa. Lets also be honest, the natural environment of most of these fish is not an anti-septic environment. Things like cleaners and soaps are another story. If I'm doing any tank maintenance I'll avoid using soaps, cleaners, or lotions for several hours before I work on the tank and before I start I rinse them well.
On the whole it's something I don't think you need to overly worry about, just use some common sense.
Originally Posted by watts300
It's funny the way you said that... because your choice of words makes fish sound like a pair of shoes or a new leather jacket that get "damaged" by being out in the rain.
Actually yes, handling fish can damage their slime coat like iamnbatman said. It's a good idea to not try and grab them for fun. Incidental contact happens and is not a concern and occasionally you've got no choice but to physically grab them but you're better off just not doing it unless its a necessity.