As Jaysee points out, the real harm is done when fish are shocked by sudden dramatic changes in the environment. This applies not only to temperature but nearly all aspects of water chemistry as well. This, for instance, is why it is not recommended to do a significant water change on a neglected tank. Even though it may seem counter intuitive, the dramatic change in chemistry, even though it's 'better', is 'shocking' to the livestock as they need more time to adjust.
Our fish can tolerate
a wider range of temperatures than we might expect, however the fishkeeper is wise not to test this too far or too often!
In nature, fish are exposed to a surprising wide range in temperatures (even in the tropics). From very cool rains to thermal gradients in still pools. The difference is that often the fish can move to regions where they are comfortable.
It is always better for temperature changes to be gradual. Having said that, I think rapid cooling is worse than rapid warming, but I don't know if my fish are in total agreement.
Good luck (I think you'll be okay...but keep a watch on that dern window).