Image colorization is an interesting yet challenging task due to the descriptive nature of obtaining a natural-looking color image from any grayscale image. To have a fully automatic image colorization procedure, we propose a convolutional neural network (CNN)-based model to benefit from the impressive capabilities of CNN in the image processing tasks. Harnessing from the convolutional-based pre-trained models, we fuse three pre-trained models (VGG16, ResNet50, and Inception-v2) in order to improve the model performance. The average of three model outputs is used to obtain more rich features in the model. We use an encoder-decoder network to obtain a color image from a grayscale input image. To this end, the features obtained from the pre-trained models are fused with the encoder output to input into the decoder network. We perform a step-by-step analysis of different pre-trained models and fusion methodologies to include a more accurate combination of these models in the proposed model. Results on the LFW and ImageNet datasets confirm the effectiveness of our model compared to the state-of-the-art alternatives in the field.