The integrated division of labor within the insect colony (with the various castes of workers and soldiers) has been achieved by selection of single queens. (In later evolution, the nest acquires such strong entitativity that queen replacement and even multiple queens can occur.) In the ants, the probable initial stage was brood-care help by daughters with postponed fertility. This created an ecology in which only nests with such auxiliary helpers survived. Nest (or colony) selection began at this point. Postponed fertility was probably augmented by a pheromone exuded by the fertile mother.
Initially there was no division of labor or caste specialization. Mother and infertile daughters were all capable of all tasks of food-gathering and brood-care. While it was in the inclusive-fitness interests of each of the sterile workers to become fertile, it was also in the inclusive-fitness interests of each to keep her sisters sterile, as by distributing the queen's fertility-inhibiting pheromone to her sisters (including those in larval stages) and by eating the haploid drone eggs that some of her supposedly sterile sisters might produce. Once worker sterility was dependably achieved, then selection by colony could dominate because the individual-vs.-individual genetic competition had been almost completely eliminated. From this point on, the elaborate division of labor into the several worker and soldier castes could develop (Wilson, 1971; Campbell, 1983).
Note that in none of the social insects are the sterile workers and/or soldiers genetically identical, nor do the castes differ systematically in their genetic composition. The differentiation into worker, soldier, and queen is achieved by differential feeding and pheromone exposure in early development. Note, too, that in the social insects the genetic competition among the cooperators has not been entirely suppressed, and shows itself in many minor ways such as kin-favoritism of workers in caring for larvae, and among the bees, at least, when the queen is removed, disruptive genetic competition among the disinhibited formerly-sterile workers (Seeley, 1985, 1989; Winston and Slessor, 1992).
The anatomical differentiation among the castes is controlled by distributed (rather than centralized) inhibitions. So, too, the communication and coordination among the castes. Some dozen different pheromones are involved, all released and distributed in automatic reflex ways. Quasi-entropic effects of mutations produce discoordination. Again, it is selection by whole nest survival that keeps these dozens of adjustments tuned for whole-nest functionality. Unique nest odors prevent individuals from transferring membership from nest to nest, further ensuring whole-nest selection.
Reference: Heylighen F. & Campbell D.T. (1995): "Selection
of Organization at the Social Level: obstacles and facilitators of metasystem
transitions ", World Futures: the Journal of General Evolution
45, p. 181-212.