Purpose: Incorrect patient positioning during radiotherapy can significantly impact treatment efficacy and pose potential risks. This study aims to develop a model that can rapidly and effectively monitor the patient’s postures during radiotherapy sessions using real-time video. Methods: The neural network utilized in this research employed a two-stream architecture, consisting of spatial and temporal streams. For the spatial stream, RGB frames from the videos were directly used as input. In the temporal stream, representative frames were extracted from the video to construct stacked grayscale 3-channel images (SG3I) frames. This approach enabled capturing motion information through a large-scale dataset pre-trained 2D convolutional neural network (CNN), eliminating the need for computationally expensive optical flow calculations. Additionally, an improved lightweight network architecture was employed. The model was trained and tested using volunteer videos collected from a radiotherapy center in a hospital. Results: The results demonstrated that the proposed model outperforms existing methods in terms of detection accuracy while achieving higher efficiency in frame generation. Conclusion: In this study, we introduced a cost-effective and highly accurate method for recognizing patient’s postures during radiotherapy. This approach could be readily deployed in any radiotherapy facility, ensuring treatment precision and patient safety.