Convolutional Neural Network (CNN) is one of the most efficient Deep Neural Networks. The addition of more layers and neurons to the CNN increases its computational complexity. Even though CNNs are capable of solving many real time image recognition tasks flawlessly, it is also crucial to design optimum neural network architecture by reducing the associated memory and computational costs for resource critical applications. The proposed method optimizes a pre-trained CNN model for Traffic Sign Recognition by identifying and eliminating the redundant channels in fully connected layer of the neural network. The basis of the algorithm is that in a large neural network, the contribution of some of the neurons is negligible and can be eliminated without much effect on the overall performance. After eliminating the channels, the resulting model is retrained to compensate for the performance loss. The process of elimination and retraining is repeated until no more redundant channels are identified. The performance of the models so developed are further compared with the original model and evaluated based on accuracy and inference time. By removing 69% of the neurons in the fully connected layer, a compression rate of 2.85 was achieved and inference time got reduced by 97ms .The model so developed had accuracy slightly higher than the original model due to the retraining performed after each iteration.