This paper explores the extent to which Chinese university-level students rely on reading passages in answering multiple-choice reading comprehension tests. It is commonly believed among teachers and students in China that students don not need to read carefully or even comprehend passages to pass the test, and that multiple-choice reading comprehension tests do not accurately indicate students' actual reading comprehension ability. It has been discovered that students can answer some multiple-choice reading comprehension questions correctly without reading or comprehending the associated passages (Powers & Leung, 1995). Another finding has been that even if students read the passages, multiple-choice reading comprehension tests demonstrate more a searching and matching skill than actual reading comprehension (Johnston, 1984). In order to investigate these claims, this study examines the reading test scores of groups of Chinese university-level students taking the standardized multiple-choice reading comprehension test of the College English Test in China under two different sets of testing condition. Our findings show that students perform better when they are not allowed to go back and forth between the questions and passages when compared with a normal reading comprehension test situation; that passage comprehension is relevant to reading comprehension test performance and that students need to read the passages in order to answer most questions correctly. However, even without the associated passages, students still achieved scores above chance level (39.91% and 28.89% respectively for two groups of students), with some test items getting a mean item difficulty of .94 and .88, suggesting that the students' use of test-taking skills such as 'guessing' in such a multiple-choice reading comprehension test was prevalent.