Landscaping and stone retailers sometimes have various gravels and sand. The critical issue is that it be inert, i.e., not affect the water chemistry. Gravel comprised of limestone, dolomite, marble, or coral will add calcium to the water over a long period of time, raising the pH and hardness; these gravls are OK in marine tanks and rift lake cichlid tanks and brackish tanks, but not in aquaria where you want lower pH and hardness. Other ingredients may be mixed in for specific purposes. Know what you're getting before putting it in a fish tank.
The other concern is the sharpness of the grains. Most aquarium gravels made for that purpose are rounded grains that will not cause harm to fish, particularly bottom fish. Some gravels are angular and the sharp edges can perforate fish and also affect barbels on bottom fish like catfish.