In this paper, we present a flexible method for color-based sorting of bulk materials. It is based on semantically meaningful color features that are constructed from a set of training images. First, estimates of color-occurrence frequencies of different materials are derived from the training images and fused into color classes, which are then used to classify individual pixels. An object descriptor is built as count statistic over the color classes appearing in the object image. This descriptor has many advantages: it is compact and very fast to compute, invariant to scale and rotation, has a very clear, intuitive interpretation, and can be used with simple rule-based classifiers. However, tuning the parameters that govern the feature construction process is laborious and requires a lot of experience on part of the system operator. To overcome this shortcoming, we automatically learn the parameters using genetic algorithms. We apply our method to wine grape sorting problems to show that this approach outperforms a human expert. At the same time, it takes considerably less effort on the human part and frees the expert to attend to other tasks. Furthermore, the system allows non-experts to successfully put a sorting machine in operation.