Abstract: The quality assessment of sports actions requires providing quality ratings for the completeness, fluency, and difficulty of actions based on the recognition of action types, which is extremely challenging. A network structure based on tube self-attention and spatiotemporal parallel perception was designed to accurately extract key information from action sequences and efficiently assess action quality. Firstly, action information was extracted and background interference was eliminated using a single target tracking and tube self-attention module. Secondly, based on the Uniformer model, a spatial multi-head self-attention mechanism was introduced in the local feature extraction module, and the local and global attention spatiotemporal serial perception network was changed to a spatiotemporal parallel perception network to improve computational efficiency. Finally, the local and global features were fused through a multi-stage fusion module to enhance action representation, and the multi-layer perceptron module was extended to output action recognition and action quality assessment results. Action recognition experiments were conducted on the UCF101 and HMDB51 dataset, respectively. The action recognition Top-1 and Top-5 accuracy of the UCF101 dataset reached 85.2% and 96.7%, respectively, and the computational cost of FLOPs significantly decreased, the action recognition accuracy Top-1 of the HMDB51 dataset reached 77.8%. Action quality assessment experiments were conducted on the AQA-7 and MTL-AQA dataset, respectively, the average Spearman’s rank correlation coefficient for the assessment of the AQA-7 dataset reached 0.808 7, the Spearman’s rank correlation coefficient for the assessment of the MTL-AQA dataset reached 0.948 1. The experimental results show that the model not only has a high action recognition rate, but also can accurately and efficiently complete the task of action quality assessment.