In 1959 Campbell and Fiske introduced the multi trait multi method (MTMM) matrix as a statistical tool that researchers can use to evaluate construct validity. The MTMM matrix examines how different measures of the same underlying construct correlate when assessed with methods that are maximally different. It identifies correlations that are common across all measured traits, and separates them into those that are related to the trait and those that are associated with the measurement method (i.e., method effects). Campbell and Fiske proposed four criteria to evaluate convergent and discriminant validity in their article.
The first criterion is that the same-trait, different-method mt-mm correlations should be high. This means that the correlations between a self-rating and a staff rating or between a teammate’s rating and a self-rating should be similar to those between the other pair of ratings. This criterion is not a strict requirement, but it does mean that researchers should strive for the highest possible consistency among ratings.
Method specific variance is another important consideration when evaluating construct validity. A measure’s variance is a mixture of its true score variance, which represents the variance inherent in the trait, and its error variance, which represents the random variation that occurs during measurement. It is desirable to have low method-specific variance and high trait-related variance in a measure.
Fortunately, there are several ways to reduce method-specific variance in an MTMM matrix. One approach is to use a regression model to estimate the relationship between the underlying trait and the individual method factor. The estimated coefficients in the regression model can be interpreted as an indication of the proportion of true-score variance that is due to each method.
A more sophisticated approach is to use a confirmatory factor analysis. This technique uses a set of covariance and mean structures to decompose the observed correlations into trait, method and error terms. The analysis identifies manifest variables that share variance with a latent trait variable and removes the proportion of variance in each of the manifest variables that is not shared by the latent trait variable. Error terms can then be correlated with each other and the remainder of the manifest variables.
This technique is also known as latent structure modeling (LSM). The results of the analyses can be interpreted in a manner very similar to that of LD models. This may be particularly useful when a clear reference method is not available or the reference method is not clearly distinguishable.
The Mplus software program supports the estimation of both LD and LM models for a MTMM matrix. The following example will demonstrate how to run the LM model using data from a study of multi-rater ratings of children’s inattention symptoms. The results will provide an indication of how much variance in the true score is due to each of the methods and an estimate of the magnitude of the trait-method effect. The resulting correlations can be used to test for construct validity and to assess whether or not the measures have truly independent meaning.
