Conventional surveillance for a security robot suffers from severe limitations, perceptual aliasing (e.g., different places/objects can appear identical), occlusion (e.g., place/object appearance changes between visits), illumination changes, significant viewpoint changes, etc. This paper proposes an autonomous robotic system based on CNN (convolutional neural network) to perform visual perception and control tasks. The visual perception aims to identify all objects moving in the scene and to verify whether the target is an authorized person. The visual perception system includes a motion detection module, a tracking module, face detection, and a recognition module. The control system includes motion control and navigation (path planning and obstacle avoidance). The empirical validation includes the evaluation metrics, such as model speed, accuracy, precision, recall, ROC (receiver operating characteristic) curve, P-R (precision−recall) curve, F1-score for AlexNet, VggNet, and GoogLeNet, and RMSE (root-mean-square error) value of mapping errors. The experimental results showed that the average accuracy of VggNet under four different illumination changes is 0.95, and it has the best performance under all unstable factors among three CNN architectures. For the accuracy of building maps in real scenes, the mapping error is 0.222 m.