As practitioners who frequently use psychometric instruments, we are often in search for newer and better instruments to refine our assessment processes. We are constantly looking for tools to measure that which we have deemed to be important in a particular context. When we come across new instruments that appear to be promising, we sometimes accept it enthusiastically without asking all the right questions. This might be particularly difficult when the assessment is put forward with sweeping statements and extravagant claims. It is imperative for us to take a step back and to evaluate information critically. We all need to be reminded by considering the following: In order for the psychometric testing industry to remain credible over the long term, it is important for practitioners to distinguish between good instruments and poor instruments. This serves as a filter and has an important function to ensure consistent and high standards of usage. It is therefore imperative to ask the right questions when faced with new or unknown instruments, or even when evaluating those that seem familiar to us. As a start, it is important to know for what purpose an instrument was developed. It is important to understand whether a tool is designed for selection or development purposes, or both, and to use it appropriately. It is also important to know what theoretical or empirical research gave rise to the development of the tool. What model is the current research based on, and how is it related to the broader research base of the relevant subject? Instruments developed in a seeming vacuum should always be approached with caution. It should also be clear what exactly an instrument purports to measure.
Thus, how are the constructs defined? This should provide a rational and meaningful framework within which the content of an instrument can be understood. It is imperative to know how constructs are defined and operationalised for a particular instrument, to ensure that we know what behaviour is and is not part of a measured construct. We cannot just assume that semantically similar constructs are homogenous in meaning across different instruments. We all know that the reliability and validity evidence should be evaluated. The overall reliability, as well as the reliabilities for each scale on multi-scale assessments should be checked. Make sure this is acceptable for your purposes! Unlike reliability, validity is not simply a number. It is an ongoing process determined by the converging of evidence. Thus, validity studies can come in many forms but the onus is on the practitioner to judge the comprehensiveness thereof. Remember to ask for published proof of these claims. International research is a useful source of information but remember to ask for South African data. If the instrument is norm based, ask whether it has South African norms or where there is research to demonstrate that it functions similarly in South Africa. Very important also is research that shows that an instrument can be used across different cultures, language and gender groups in an unbiased way. Research on the psychometric properties of an instrument is an ongoing process, which means that research should constantly be updated. It is important to understand that when changes are made, for instance to the items of an assessment, that new research is required since the old research then becomes outdated. Therefore, new and updated research should always be available for older instruments. The above information just serves as a reminder that when choosing a psychometric instrument, it is not what the tool measures that is of primary importance, but rather that you are able to make a case for why it is an appropriate instrument to use. This information should be available in the relevant test manual, which you should scrutinise carefully before making your purchase. In the case of psychometric assessment, ignorance is not bliss, so choose your assessments carefully. For help in choosing the right instrument for your purposes, you can contact any one of our psychometric advisors. You can also contribute to local research studies by offering to collect data in your organisation.
If you would like to participate in any of JvR’s research studies you can contact the research department at firstname.lastname@example.org.