Hand shape recognition with a color glove can express many patterns and be used to several applications such as communication and input interface etc. The first step of this scheme is taking hand shape image by web camera and extracting color information of fingers. The recognition performance depends on color information which is affected by environment, especially light condition. This paper proposes a method that removes the hand shape image of color error detection by taking into hand shape characteristics, that is, finger position relation and finger length into account. The effectiveness of the proposed method is shown by comparing the recognition success ratio by conventional methods and with the results by proposed methods.