I see no reason that you can't feed venison to your baby. Out daughters all ate it from the time they were old enough for solids.
The only exception would be the rare instance that there would be an allergic reaction. 

Dave

PS. I also have four daughters and no boys! (Wouldn't trade them for one either.)