There isn't much to worry about on there, that's where the beneficial bacteria will start to colonize. Put it back in the filter and leave it for a while.
Wait until a sponge gets 50% or more clogged or you can see the water flow is slowing a little. Then, when you carry out your part water change, drop the sponge in a bucket with the tank water, give them a squeeze, (don't get them spotlessly clean, leave some of the "mulm" on them) and return them to your filter. The only filter media I replace is floss as it collects all the fine debris and is a bit mushy when squeezed out. Just replace it if you have it, sponges are rinsed and kept to keep the bacteria your tanks needs. Beneficial bacteria lives on pretty much everything in your tank, but very little to none, in the water itself. Hence you can carry out large water changes and as long as you use a good dechlorinator, all will be well.
When my only filter was an internal, I filled it with (from the bottom) coarse sponge, medium sponge, fine sponge, and top it off with floss. In my current tank I have an old Fluval 4 plus internal filled with only floss which I replace when required. My external filter was last checked and rinsed about two months ago. Once your tank is in "balance" your filters need less maintenance.
As to what's killing your fish, well it could be anything. I wouldn't want to give you a completely wrong diagnosis and you lose more fish. An aquarium is such a complex part of nature in your living room and it is a very rewarding hobby. But at the same time it can be infuriating. Parasitic worms such as Callamanus are an absolute nightmare. By the time you see the worms the fish are already close to the end as the worms have already damaged the internal organs. It can be very difficult to treat successfully. Bring in an infected fish, it can be curtains for the rest. Columnaris is another nightmare disease. However, being in the US you have better access to the required treatments than I here in the UK.
With regards to your fish I'm not saying they have these diseases, I'm merely saying diagnosis of anything can be difficult. Keep an eye on water parameters with a good test kit, not just the strips. Water changes are one of the best medicines for a fish. Sounds daft, but it's true.
You could add a little epsom salts (or aquarium salts) to your tank, diluted fully in a separate container before adding it. When a person is admitted to hospital, the first thing in their arm is a saline drip. It replaces electrolytes etc. I use it in my tank very occasionally but only in very small amounts, nowhere near the full dosage. It just acts as a "tonic" or "pick me up" if I've had to medicate my tanks. (Which is always a last resort for me.) Once I've finished medication, carried out a water change and given some activated carbon a few days to leach out the remaining treatment, I add a very weak saline mix to the tank. After a few part water changes it gradually gets diluted to nothing.
Oh, and some carbons can contribute to high phosphates. Worth noting if you have carbon in your filter all the time.
Sorry for such a long winded post. I hope you haven't nodded off.

Obviously these are only my opinions and someone else may tell you different, I don't want to come across as a "know all" because there are far more experienced fish keepers than me out there.
If you have the time, YouTube has some great fish keeping channels. I like..
Prime Time aquatics
Father Fish
KG Tropicals
Aquarium CoOp
Rachael O'Leary
All the best,
Steve.
Steve.