[Stata] Chi-square test & Cramer’s V effect size, post-hoc analysis (tab and tabchi)

The chi-square test is an analysis used when both the independent and dependent variables are categorical variables.

In STATA, the relationship between the two categorical variables can be checked through cross-tabulation, and the chi-square test can be conveniently performed as an option here.

tab yvar xvar, chi col
tab xvar yvar, chi row // either way  

As above, after the tab command, list two categorical variables and put “chi” as an option to perform the chi-square test. Here, categorical variables also include binary variables.

Application

Here, x is the independent variable and y is the dependent variable. For example, let’s say I want to analyze the difference in health status according to sex in the dataset. Here, the sex is defined as a categorical variable with 1= Male 2 = Female. Meanwhile, health status was defined with 6 categories. Therefore, it is appropriate to perform the chi-square test at this time.

tab yvar xvar, chi col

If you add an column option after the chi option, the percentage based on the column is also provided.

According to the results, it can be concluded that there is a significant difference in the ratio of health status according to the biological sex at the significance level of 99.9% (p <.001).

Calculating Effect Size: Phi or Cramer’s V

When conducting a chi-square test, understanding the strength of association between two categorical variables is key. The chi-square test tells us if there is a significant relationship, but to understand how strong that relationship is, we calculate the effect size. The two common measures of effect size for chi-square tests are Phi and Cramer’s V. Before calculating effect size, you must run the chi-square test.

The formula for calculating Phi and Cramer’s V depends on the size of the contingency table:

• Phi is used when the table is a 2×2 table.
• Cramer’s V is used for larger tables.

To calculate Phi in Stata, extract the chi-square statistic from the tabulate command output and manually compute Phi:

display sqrt(r(chi2) / r(N))

In Stata, you can directly calculate Cramer’s V using the tabulate command with the V option. Here’s how you can do it:

tabulate y x, chi V

Both Phi and Cramer’s V range from 0 to 1, where:

• 0 indicates no association.
• 1 indicates a perfect association.

Guidelines for interpreting the effect size vary depending on the field, but a common approach is:

• 0.1 = small effect
• 0.3 = medium effect
• 0.5 = large effect

Advanced: Post-hoc analysis

You may want to explore specifically how the differences are significant within subgroups here. This is called post hoc analysis. Since STATA does not provide this as a default option, you must use a user-created package called tab_chi (developed by Nicholas Cox). This command performs the same test that is available in SPSS (See this video for SPSS) based on the adjusted residuals for interpreting the contingency table. Please note that performing post-hoc on the chi-square test is not widely recommended and is debatable (See this discussion post).

ssc install tab_chi
tabchi yvar xvar, adj noo noe

Afterward, posthoc analysis for the chi-square test can be performed using the above syntax. Here, the adj option is an abbreviation for adjusted, which means “adjusted residuals, Pearson residuals divided by an estimate of their standard error.” Adjusted residuals refer to Pearson residuals that are adjusted for errors in estimating the standard error, considering the sample size.

As the p-value is not provided here, you need to interpret the significance by using “Z-distribution” for the two-tailed tests.

You can interpret the significance level through the Z-score distribution table above.

• If the test statistic is greater than or equal to 1.645 or less than -1.645, it is significant at the p<0.1 level.
• If the test statistic is greater than or equal to 1.96 or less than -1.96, it is significant at the p<0.05 level.
• If the test statistic is greater than or equal to 2.576 or less than -2.576, it is significant at the p<0.01 level.
• If the test statistic is greater than or equal to 3.291 or less than -3.291, it is significant at the p<0.001 level.

Please find this post if you are unfamiliar with the concept of z-score and p-value: https://pro.arcgis.com/en/pro-app/latest/tool-reference/spatial-statistics/what-is-a-z-score-what-is-a-p-value.htm

Also, you can get the exact p-value by using this by using two-tailed hypothesis: https://www.socscistatistics.com/pvalues/normaldistribution.aspx

In the result table, the absolute value of the adjusted residual in East Asian cells is 2.576 or more, and in South Asian cells, it is 1.96 or more. The adjusted residuals can be reported as shown in the table above. Significance means “they are more extreme than what would be expected if the null hypothesis of independence was true.”

Please refer to the below two papers for more details on how to report in the tables and how to interpret them.

To examine the relationship between the provider-trained group and scoring on the knowledge items, chi-square tests were conducted to test for independence of the accuracy of responses to knowledge questions and whether a respondent had received training or counseling in FABM from a provider.
Chi-square tests were then used to test for independence of the groupings and how users rated attraction and functionality and ranked types of evidence. The adjusted residuals were calculated to determine where the largest differences between observed and expected counts arose, while accounting for sample size of each of the three respondent groups. Statistical analyses were conducted in Stata v. 15 (StataCorp LLC; College Station, TX).

Starling, M. S., Kandel, Z., Haile, L., & Simmons, R. G. (2018). User profile and preferences in fertility apps for preventing pregnancy: an exploratory pilot study. Mhealth, 4.