I have about 300 workers in a colony from a Queen last year, so they can grow very fast, it all depends on how you look after them, I wouldnt expect a first ant colony to reach its potential until you have a little experience, but you should have 100 workers at least by next spring
I answered, No, a colony cant be to small, they all start from somewhere!