[Stata] Convergent and discriminant validity and reliability of subscales: validscale command

When we use a questionnaire to measure a concept that is not directly observable, such as quality of life, personality traits, or mental health, we need to ensure that the questionnaire is valid and reliable.

In this blog post, we will focus on three types of validity: convergent validity, discriminant validity, and concurrent validity, and one type of reliability: internal consistency. We will also introduce a Stata command that can perform all the necessary analyses to assess these properties: validscale.

📒 Perrot, B., Bataille, E., & Hardouin, J. B. (2018). validscale: A command to validate measurement scales. The Stata Journal, 18(1), 29-50.

Definition

Validity means that the questionnaire measures what it is supposed to measure, and reliability means that the questionnaire measures it consistently.

• Convergent validity is the degree to which the items that are supposed to measure the same concept are correlated with each other.
  • For example, if we have a questionnaire that measures anxiety, we would expect that the items that belong to the anxiety subscale are positively correlated with each other.

• Conversely, discriminant validity is the degree to which the items that are supposed to measure different concepts are not correlated with each other.
  • For example, if we have a questionnaire that measures both anxiety and depression, we would expect that the items that belong to the anxiety subscale are not correlated with the items that belong to the depression subscale.
• Concurrent validity is another type of validity that can be used to evaluate a questionnaire. It shows how well a new test agrees with an existing test that has already been proven to be valid. The new test and the existing test should measure the same or similar concepts, and they should be taken at the same time.
  • For example, if we have a new test that measures anxiety, we can compare it with an existing test that also measures anxiety and has been validated before. If the scores of the two tests are similar, we can say that the new test has concurrent validity.
• Internal consistency is the degree to which the items that belong to the same subscale produce similar scores.
  • For example, if we have a questionnaire that measures anxiety, we would expect that the scores obtained from the items that belong to the anxiety subscale are consistent among respondents.

To assess convergent validity, discriminant validity, and internal consistency, we can use various methods such as factor analysis, correlation analysis, Cronbach’s alpha, and omega coefficients. Fortunately, there is a Stata command that can do all these analyses for us in one step: validscale.

validscale command

validscale is a user-written command that was developed by Perrot et al. (2018) to validate subjective measurement scales using classical test theory. It can perform factor analysis, correlation analysis, Cronbach’s alpha, omega coefficients, and other tests to assess the validity and reliability of a questionnaire. It can also produce a summary table and a graphical output that display the results of these analyses in an easy-to-read format. validscale can handle both unidimensional and multidimensional questionnaires, as well as binary and ordinal items.

You can first install packages in one line.

ssc install validscale // install package 

Then, you can start browsing the features of the command with dialog box.

db validscale

The dialog boxes show that you have a variety of options to adapt for your specific case.

Here is the most common command that you can use. What I do in general is copy this command and change the item-item20 for the list of variables in the scale, and then put the part for the subscale. Remember that this is used for the case that subscales or multidimensional scales are already known, which is confirmatory factor analysis (CFA) rather than exploratory factor analysis (EFA).

validscale item1-item20, part(5 4 6 5) graphs cfa cfastand convdiv convdivboxplots kgv(factor_variable) kgvboxplots

• graphs: graphs displays graphs for items and dimensions descriptive analyses. It provides histograms of scores, a biplot of the scores and a graph showing the correlations between the items.
• cfa: cfa performs a Confirmatory Factor Analysis (CFA) using sem command. It displays estimations of parameters and several goodness-of-fit indices.
• cfastand: cfastand displays standardized coefficients.
• convdiv: convdiv assesses convergent and divergent validities. The option displays the matrix of correlations between items and rest-scores. If scores(varlist) is used, then the correlations coefficients are computed between items and scores of scores(varlist).
• convdivboxplots: convdivboxplots displays boxplots for assessing convergent and divergent validities. The boxes represent the correlation coefficients between the items of a given dimension and all scores. Thus the box of correlation coefficients between items of a given dimension and the corresponding score must be higher than other boxes. There are as many boxplots as dimensions.
• kgv: kgv(varlist) assesses known-groups validity according to the grouping variables defined in varlist. The option performs an ANOVA which compares the scores between groups of individuals, constructed with variables in varlist lue based on a Kruskal-Wallis test is also given.
• kgbvboxplots: kgvboxplots draws boxplots of the scores split into groups of individuals.

It will return the results like this 🙂

Tip. Exploratory Factor Analysis

factor item1-item10
factor item1-item10, factor(2) // you can force the number of factors 
rotate, orthogonal varimax blanks(.5) // and then rotate it to interpret the results easily 

Another highly recommended command to explore factors is loadingplot! You can see how factor loadings are grouped together like this.

loadingplot, norotated

Reporting the results

The following table provides an example of reporting the results from validscale, with Cronbach’s alpha (reliability), convergent validity, and divergent validity across two subgroups in the sample.

Slides from command authors

You can also learn from the slides from authors; it might be much easier to understand than reading help file 🙂