The Reading Comprehension Test, based on NAEP reading framework, was developed for 5^th and 6^th grade students using the Rasch measurement model. The participants of this study were 1,052 5^th and 6^th graders sampled from Kaohsiung and Pingtung counties using stratified random sampling procedures. The 39-item test, including both multiple-choice and construct-response items, measures three cognitive levels (i.e., locate/recall, integrate/interpret, and critique/evaluate) of literary and informational texts. Results show: a) the underlying trait defined by the reading comprehension test holds the characteristics of an objective measurement (i.e., the construct measured by the test is unidimensional and invariant across genders and grades); b) the rating scales for construct-response items are appropriate in terms of their order and distance; c) items spread reasonably well along the latent continuum and are aligned with various ability levels; d) the three reading cognitive levels show hierarchical structure as expected; e) the test converges and discriminates various measures appropriately; f) the test is able to discriminate group differences by gender, grade, and ability level; and g) the test has satisfactory person separation and rater reliability. Discussions and suggestions regarding test application, the application of the Rasch model, and future research are provided.