Presuming your rooster is old enough to be sexually mature (5 - 6 months or so), he is likely mating your hens when you are not around. Just because you don't see him mating, that doesn't mean it isn't happening. In fact, if he sees you as the proverbial top cock-of-the-walk, he'll want to avoid your seeing him with the girls if he can help it.
A good ratio of hens to roosters in backyard flocks is about 10 hens per rooster. This maintains fertility, but avoids injuries to hens from overbreeding.