If you could assign a flavor to the questions on the Uniform CPA Examination, it would undoubtedly be pure vanilla.
Tales of adventure, romance, and violence can be found at the bookstore or in the movie theater, but not on the CPA Exam, which is carefully designed to avoid topics that could unnecessarily distract CPA candidates from the actual purpose of the exam.
“We try to give the candidates the opportunity to provide clear evidence of their mastery of accounting-related knowledge, skills, and abilities (KSAs), and we try to exclude factors that have nothing to do with those KSAs,” said Jessica Lucas, an associate manager for the AICPA Examinations team. “For example, we’re not trying to assess their knowledge of unusual vocabulary words unless those words are commonly used in the field of accounting or any of its specialized fields. So a word like opine, which is part of the everyday vocabulary of auditors, can and does appear on the exam; but a word like blandishment does not. We don’t test advanced reading skills either. Questions are designed to assess essential content only, with no embellishment. We never try to trick the candidates in any way. Each sentence is designed to be as straightforward as possible and to have one and only one meaning in context. The meaning has to be clear to all readers.”
The CPA Exam is one of the most respected professional licensure examinations in the world. What’s not so well known is that the rigor associated with exam development procedures is as intense as the review process that many candidates undertake in their efforts to earn a passing score. Each potential question undergoes scrutiny during multiple rounds of review by specialists evaluating it from different points of view. If the question survives those reviews, the next step is a pretest phase to perform a statistical analysis of actual test takers’ responses to the question. (Pretest items don’t count toward a candidate’s score.) If the question fails to meet statistical standards, it is rejected or revised.
As laws, regulations, and other types of guidance change, the exam also needs to change. It takes about eight months from the time the Examinations team identifies a need for more questions to the point when they are ready to be pretested on the exam; in certain cases, the time elapsed can be as short as four months.
In general, each question is designed to be a successful act of communication. Linguistic philosopher Paul Grice observed that successful, cooperative communications typically follow four basic rules:
Be appropriately informative.
The text of the CPA Exam obeys the rules and conventions of international standard English so that the questions will be easy for English speakers around the world to understand. Likewise, staff members use their knowledge of logic and linguistics to make sure that the questions assess only accounting content, not advanced skills in logic or linguistics.
“We design the text of each question to be easy to process. We avoid needlessly complicated wordings and logical structures,” said Timothy Habick, Ph.D., a test development manager for the AICPA Examinations team. “Now some people might wonder how logic could be an issue relevant to the CPA Exam in the first place, and that is the point. We purposely craft the text so that candidates will not need to use logic to understand the intended meaning of any sentence in context. Our goal is to create clear, coherent, and cooperative communications. This tends to make the text of our questions somewhat unremarkable, but that is our intention.
“Candidates are supposed to focus on the essential accounting content, not on our use of language. For the same reason, we tend to avoid using names for individuals and companies that could attract the candidates’ attention. So whenever possible, we refer to ‘a taxpayer,’ ‘an auditor,’ or ‘a company’ without using realistic names. Proper names can contain allusions to irrelevant historical events and individuals, which could be distracting. We’re here to assess the candidates’ accounting-related skills, not to entertain them. We try to include only content that the candidates need to demonstrate what they know. Everything else is removed in order to create a smooth testing experience.”
The drafting of a CPA Exam question begins with the CPA Exam Blueprints, which contain detailed information on all the subjects that have been specified for use on the exam, as well as the level of understanding of the material that may be tested. The Blueprints are based on surveys of practicing CPAs regarding the knowledge that newly licensed CPAs should possess. The AICPA maintains a bank of over 10,000 approved questions, classified according to topic and various statistical features. The bank is kept full by external subject-matter experts. The volunteers are experienced CPAs who have been trained to write exam questions in their areas of expertise. All volunteers are carefully screened and are bound by a confidentiality agreement.
As these experts write questions, they work with AICPA staff and mentors who review questions for fairness and test development principles (see the sidebar “Exam Questions: What Are the Rules?”). Once satisfied, they send the questions to a review group composed of content experts with knowledge of specialized areas. The review group votes to accept, reject, or revise each question. The questions that are accepted then receive additional extensive internal reviews from AICPA staff members, who ensure that questions assess topics and skills specified in the Blueprints and review the questions for technical accuracy, readability, clarity, logic, fairness, and simplicity of wording.
“When you consider how much time, research, and care goes into every single question on the exam, it is quite amazing,” Lucas said.
Once the questions pass the internal review, they are ready to be tested by actual candidates. They are first included in the exam as pretest questions, which have no effect on candidate scoring. There is no noticeable difference to candidates taking the exam between pretest questions and questions that count toward the calculation of scores. The statistical performance of a pretest question on the exam provides test developers with valuable information, which is used to further ensure the fairness of the exam.
“We might have thought that a question was perfect in terms of its carefully constructed wordings and content, but if you put it in front of 1,000 candidates and there’s still some problem with it, the statistical results of that trial run will generally highlight the issue,” said Renée Warias, senior manager for the AICPA Examinations team.
An ideal question will be answered correctly by a large percentage of candidates of high ability (as determined by their performance on the test as a whole) and will be answered incorrectly by a large percentage of candidates with lower demonstrated ability. When that situation occurs, the question plays a useful role in distinguishing between candidates who have the required accounting skills and those who do not.
“If 99% of the candidates answer a particular question correctly, the question does not provide useful information about the candidates’ relative abilities,” Warias said. “The same situation occurs if only 5% answer correctly. The question is either too easy or too difficult to provide useful information.”
If the metrics on a pretest question meet statistical standards, the question is sent to a committee of CPAs for a final vote on whether it merits a place on the exam. Questions that receive final approval are added to the bank of questions that can be selected for use as a scored item on the CPA Exam.
Throughout the development process, the Examinations team treats exam questions as “high-stakes communications” because there is a lot on the line with professional certification exams. Unclear sentences or wordings in questions could prevent qualified candidates from passing the exam. That’s why the development and review process for CPA Exam questions is so painstaking.
“The high stakes of the CPA Exam mean that we have a duty to do our jobs perfectly,” Habick said. “We are not aiming for 99%. No. It has to be 100% perfect, and there can be nothing wrong with the questions by the time they’re ready for the candidates. That’s our duty, and we take it seriously.”
Managing the whole question-development process is a complicated endeavor. It is a huge job to recruit, train, and manage the external subject-matter experts — whose expertise and willingness to help are extremely valuable. Managing the “bank” of test questions is complicated, too. It’s not a good idea to keep a reserve of too many questions because the ever-changing regulations and accounting standards could render them obsolete before they actually get used on the exam. At the same time, moving questions through a multistep review process can be time-consuming.
Nonetheless, getting it right can be extremely rewarding — providing a pathway for qualified candidates to serve the public as CPAs.