From the standpoint of signal detection theory, the work aims to explore the impact of transparency design that provides information on system reliability levels on operator trust levels and human-machine collaboration performance under different system reliability levels and risks. In experiment, a within-subjects design of 2 (transparency of the system:low, high) × 2 (risk:low, high) × 3 (reliability level of the system:95%, 70%, 45%) was used, and TNO trust task was adopted to collect the data of 32 subjects. The trust level, judgment standard tendency and human-machine collaboration performance of automation system under various conditions were recorded, and the data were analyzed by repeated measurement of variance. Under different reliability levels and risks, transparency design had different effects. Compared with low transparency design, when automation system reliability was high, the high transparency design could significantly improve the yield rate and subjective trust score of the operator to the system, which was closer to the optimal judgment standard, brought higher correctness gain and smaller optimal performance difference between human and machine. Risk level affected the performance of human-machine collaboration. The performance of human-machine collaboration under high risk was significantly lower than that under low risk, and when the system reliability was high, the operator's yield rate and correctness gain to the system under high risk were significantly reduced under low risk. Transparency design could change the trust behavior and performance of operators under different risks. Under high risk, high transparency could significantly improve the operator's yield rate to the system and reduce the difference with the optimal human-machine performance. Transparency design is beneficial to improving the adaptability of the operator to human-machine trust, optimizing the decision making criteria and promoting the performance of human-machine collaboration, especially under high reliability and high risk of the system.