For me, I don't think there's a reason to limit yourself to a selected few if you can have more affiliates. Yes, a few quality, engaged affiliates can do the magic but hey, if you can have more why not go for it?
I think the question here is time. If you have the time to sponsor and follow up lots of affiliates, I beg you, please go for it.
On the other hand, if you cannot afford the time needed to follow this through, then simply stick with a few active ones.
