There's 2 factor's I'd consider here that are important IMO:
I know some of this will sound repetitive to you thinking back to your "spotless tank cleaning" thread a while ago.
What you're referring to as dirt getting sucked up is actually what you'd compare to in the gardening word (outdoors that is) as manure. Much like the outdoors gardening world this is very beneficial for your plants.
As most you know by now all my old tanks never gotten liquid ferts or the sticks etc up until I now stared this fert matter in my 55g....So why did these other tank's plants still thrive?
I belive it was the simple combo of my stock; source water and me letting mulm built up. Which initially started simply because I couldn't get inbetween the plants and then in comparison when I built my 55g Yellow tank noticed that my plants in the "dirt" tank were thriving much better then the exact same plant (same mother plants and all own raised by me, so not a store difference there to consider) in the super clean Yellow tank (where I could get inbetween to vacuum).
Nowadays I have 2 spots I gravel vac neatly and the rest is left alone: That is for one the very front glass which for visual reasons and not having plants right up to it I vacuum and the 2nd spot is lil areas I try to keep plant free to feed cory's at.
Now the second part here; and that's the more important part IMO are your shrimp.
Your plants will still thrive because you use all sorts ferts there...The shrimp live off of that mulm built up and there's certain nutrition added to their diet this way (eating what you'd refer to as dirt) that is actually know to help prevent molting issues (just like bog or oak leaf's would).
So while your plants have a secondary source so to say; for the health of your shrimp I'd advise let it built up in the back of the tank at least so they have a snack bar with healthy foods there!
Edit: Upsy...that became kinda long story there didn't it...I'm sorry