This paper aims to establish a semi-automated usability testing method to provide an idea for solving the problem of high cost when using traditional usability testing methods to identify the usability problems of mobile APP. The user behavior data is automatically captured in the process of users performing test tasks in the form of screen recording by a data acquisition tool CAUX. The actual task execution sequence is extracted from the recorded screen data, through image comparison technology. Then the pages and behavior data with usability problems are automatically extracted by comparing the actual and expected task execution sequence, and a report is generated. Based on the content of the report, usability problems can be quickly identified by researchers. Finally, a comparative experiment with Xiami Music as a case to verify the feasibility of the method. The results show that the majority of pages with usability problems and their usability issues that are difficult to find with traditional methods were discovered by using this method. The usability problems of APP in the daily use environment can be found by automatically capturing and semi-automatically analyzing the screen recording data when users perform test tasks, thereby reducing usability testing costs, benefiting novice researchers and promoting rapid iteration of products.