Scene Recognition is deeply governed by the semantic context in the scene images. The challenge introduced by diversity in intra-class spatial layouts, and similar object’s existence in inter-classes imposes great difficulties in adapting image characteristics. Today existing ap-proaches either incorporate either object-based features, image-based features, and handcrafted features or a combination of two feature extraction strategies. Therefore, existing models are unable to represent the spatial context, and overlook the distinctiveness of coexisting objects across different scenes. These issues have degraded the performance of scene recognition systems even over a single dataset. The work presented in this article uses distinct features obtained using the scene objects (object-based), complete scene images (spatial layout-based), and eight handcrafted features. A fully connected convolutional neural network (CNN) is trained on cross-domain dataset images obtained from three distinct datasets using the com-bined features. Experimental evaluation of the framework over the test samples showed that the transfer learned-based CNN model was able to obtain a mean classification accuracy of 95.84% (indoor and outdoor scenes) outperforming other better approaches. The sample groups for training, validation, and tests were obtained randomly from the self-generated dataset.