You could but bread has been proven time and time again to help with that transition. As they get use to solid foods crush up lab block into smaller pieces and offer them up to the baby. Also with a younger mouse know how to gender. Males will never have nipples and females always will even as newborns. It is the easiest way to tell genders apart since baby males can tuck up their testies and appear female.
Get kitten formula to feed it an an eyedropper. As you wean it fill up a cap with the milk formula and add a chunk of bread, just enough to make it soggy. This makes it soft for baby to eat and the milk will entice it to munch the bread. Babies need to be warm so a heating pad against the cage and a tissuebox stuffed with tissue will make a comfortable next.
How young is it? If it's already weaned I would say go for it. If it's younger it will bound to you faster and have a stronger bond. Also makes them easier to train. If they are still nursing it can be tricky.
I would get a mouse from either a shelter or a breeder but if you don't have either find a petsmart that only carries males. Petco carries males and females and the staff are horrid at telling genders. Nothing worse than taking home a pregnant female they told you was male.
Waterbottle, food dish, shavings (carefresh paper bedding is great), a wheel (make sure it's large enough too), a hut, the plastic ones work but in the hermit crab section they have coconut huts that are nicer since they are dark, and looots of chew toys and blocks. Oh and when feeding mice buy gerbil food. It has the right balance of protein and vitamins for mice compared to rat and mice with sometimes has too much protein which makes petshop mice lose their fur. Oxbow hamster food is good for a main pellet and Tropical Mix is great for treats, full of good things mice can eat but avoid sunflower seed and peanuts. Those are the two most common things mice become allergic too and elarning your mouse is allergic to them is scary o.o Fresh veggies and fruit is a great snack too and yogurt