The color features extracted from the image are often expressed as a series of colors, that is, the clustering centers obtained based on the clustering method. In the optimization of color scheme aiming at the reproduction of picture's color image, the clustering dispersion of extracted color is seldom considered. As a result, the colors in application are too concentrated in the cluster center and the diversity is lost. This paper aims to provide more accurate and effective constraints for the optimization process based on the clustering results. Firstly, K-means clustering method is applied to extract several characteristic colors from each image in the library and express them in the visualized 3D color space. Then, user selects colors to construct a series of customized color spaces, which are the constraints for color changes, namely, constraint space. An interactive genetic algorithm for color matching optimization based on constraint space is designed. Taking the color image rebuilding of the intangible cultural heritage library as example, the color optimization task based on graphic design is tested, and performs well in both efficiency and satisfaction. The paper solves the problem of imposing constraints on the reuse of extracted colors, effectively utilizes the distribution information of extracted colors, and enables the optimization process to reproduce the image of the original library more accurately, especially the color distribution features.