| Performance-based testing has been around ever since primitive people showed their children critical skills, then watched the kids try to duplicate the adults' actions.
|
| What is Authentic Testing?
|
| Authentic Testing is testing that requires the same type of response that you’re going to give when you use the skill in the real world.
|
| In more formal terms, the response modality for the item and the response modality for application of the skill are identical, or nearly so.
|
| Conventional multiple-choice testing makes the assumption that because you can make a correct choice on an item, you can respond appropriately when you need to apply the knowledge assessed by the item. This is a difficult assumption to verify.
|
| Do I need to have a degree in testing to run a successful testing department?
|
| No. The tools presented on this site and the plain-English explanations should enable you to run a rigorous testing program.
|
| How come most of the stuff on this site is free?
|
| We have been doing authentic testing since 1989, and over that period have written a lot, and created a lot of spreadsheets to implement various points. It’s better to have that information available than to hoard it.
|
| Also, the static spreadsheets (without macros) illustrate points and are good for training. We’d like the training to be low-cost or free. Many are available now under Static Spreadsheets.
|
| In the coming months, we’ll offer complex spreadsheets that configure themselves and are adjusted automatically to your data. These auto spreadsheets will be low-cost, so you can run a professional testing organization without consulting help.
|
| Then we will offer online tools for Job Task Analysis, remote testing, and test analysis. These will be available for one-time or annual license fees.
|
| We do not sell our mailing list, nor will it be used for spam.
|
| What about the statistics required to run a testing program?
|
| The statistical analyses are built into a number of the Excel programs available on this site. The results are numbers you can evaluate without knowing the details of derivation or computational processes.
|
| For the mathematically inclined or the intellectually curious, statistical processes underlying the analyses presented on Authentic Testing are referenced.
|
| What about standards: ANSI, NCCA, and the APA/AERA/NCME Testing Standards?
|
| That’s a complex subject, one we address in a one-day workshop.
|
| But the short answer is this:
|
| The Testing Standards are advisory, representing defensible best practices.
|
| The NCCA standards require rigorous organizational and reporting requirements, but depend on self-report for enforcement.
|
| The ANSI guidelines have a stronger management component than NCCA, and are enforced by on-site surveillance visits.
|
| What do the Equal Employment Opportunity Guidelines require?
|
| We have salient excerpts from the guidelines for you to download and read if you wish.
|
| The entire text of the revised guidelines is available at the
|
| U.S. Government EEOC Website.
|
| If you use testing for pre-employment screening or promotion, it must be unbiased against protected minorities. It must also be valid as assessed by any of several criteria. If you find your testing is biased, then you must show that you are remedying the bias by actively recruiting and training minorities to pass your employment exams.
|
| Work samples which are a direct and representative sample of the type of work to be performed are acceptable under the revised guidelines.
|
| A more detailed evaluation of bias and how calculate compliance is found on this website under Bias.
|
| What if I find an item was scored incorrectly?
|
| Dicey subject. Immediately remove that form of the exam from administration. Then, see how many exams the item was scored incorrectly on. Rescore each of the exams and see which exams the rescoring would have changed the pass/fail decision. Then meet with your board to see what the policy is for rescoring exams, and follow that policy. Often, the policy is something like the following: If someone formerly failed the exam and now has achieved a passing score, notify them that they have been certified based on review of the exam. If someone formerly passed the exam and would now fail the exam, notify them that their certification is in suspense, and offer them a retake of the exam at no charge.
|
| The best way to eliminate incorrect scoring is to evaluate item point-biserial coefficients after 50 to 100 exams have been administered. If the point-biserial is a large negative number (say 0.2 to 0.7), then it’s quite likely the item is being scored incorrectly, or that there’s an error in the item content.
|
| How do I pick subject matter experts to write items?
|
| Pick recognized experts in the field: teachers, authors, conference presenters, anyone recognized as preeminent in the field. Make sure you keep their bios in case anyone challenges the content of the exam. And make sure you record the attendance at item development workshops and at exam reviews. These are the best ways to ward off a content appeal if it ever goes to court.
|
| How long should I archive scores?
|
| Make sure you archive both scores and responses. If someone appeals, you want to be able to show the exact items presented and the responses the candidate made to each item. With this information, you should be able to justify the scoring of each response.
|
| Can I use Social Security numbers as a unique candidate identifier?
|
| Yes, but it’s not a good idea.
|
| Congress has made it illegal to require social security numbers. You may request them, but can’t require them, so you must have provision for an alternative. In addition, you’ll have trouble if your exam goes overseas, since they don’t use Social Security numbers in the same format in other countries.
|
| Create your own unique identifiers and use them.
|
| What about translation?
|
| Translation in the testing industry is usually referred to as localization. That’s because converting an exam to another culture typically involves much more than just translating the words from one language to another.
|
| Some critical issues are
|
| Ø how the domain is practiced differently in the target culture
|
| Ø how different roles impact practice in the target culture
|
| Ø what correct practices and procedures are in the target culture
|
| Ø the level of acceptable expertise required to practice in the target culture.
|
| All of the above contribute to significant differences in item presentation, item scoring, content distribution, and the passing score in the target culture.
|
| How can I get my questions answered?
|
| Just go to the Authentic Testing Forum, sign up, post your question, and it will be answered.
|
