It largely depends upon the amount of constant activity around the tank. The cover issues (plants, wood, etc) mentioned are very important, as is the group for shoaling fish; but even in an ideal environment, fish that are not used to activity outside the tank will be prone to run for cover as soon as anyone approaches.
My fish are presently in a fishroom at the end of the house. I only go in there specifically to observe/maintain them. They all disappear fast when I enter the room, always. But once I'm in the chair for a minute, out they all come. But if I make any sudden motion, back they go. This is solely because they spend most of the day on their own. In my former home, I had a tank in my living room and my constant movement in the evenings didn't bother the fish. Fish get accustomed to a situation, and that becomes "the norm." Any variation will be a threat to them.
I recall reading about a discus keeper who had a fishroom, and he spent a lot of time in the room, and the discus were always out and about. He brought in a visitor one day, and the fish suddenly fled behind the plants and would not come out. He worked out that it was because he was wearing dark clothing, whereas he always wore a white labcoat normally in the fishroom. The fish "recognized" the white, but the dark was a threat.