Consultation3 : GLMM/GEE & Kappa for ACS Preference Data

1. Recommended statistical approach

• The outcomes (acs_instrumental_1 to acs_instrumental_33) are binary (0/1), and each subject provides 33 repeated responses, so observations within a subject are correlated.
• Therefore, instead of running 33 separate logistic regressions, it is recommended to use:
  • Logistic GLMM (Generalized Linear Mixed Model) with a random intercept for subject, or
  • Logistic GEE (Generalized Estimating Equations) to account for within-subject correlation.

2. Why restructuring to long format is needed

• To fit GLMM/GEE properly, the data should be converted from wide format (33 columns of outcomes) to long format, where each row represents one (subject, activity) observation.

Long-format example (key idea)

• In long format:

  • activity_id = 1 corresponds to acs_instrumental_1
  • response is the value of that variable (0 or 1)

Example rows:

3. Variable coding advice (demographics)

• The current coding (e.g., sex as 1/2, education as 1–5, job as 1–7, hand as 1/2) is generally fine as long as:

  • In SPSS, these are explicitly treated as categorical variables (Nominal/Ordinal as appropriate, and specified as “categorical predictors” in the model).

• For easier interpretation (optional but recommended), binary variables can be recoded to 0/1:

  • sex: male = 0, female = 1
  • hand: right = 0, left = 1 This is not strictly required, but coefficients become more intuitive to interpret.

0. Intraclass Correlation Coefficient (ICC) — Reliability Assessment

The Intraclass Correlation Coefficient (ICC) is a statistic that quantifies the degree of agreement (or consistency) between measurements of the same targets made at different times or by different raters. ICC is typically used for continuous measurements and evaluates how much of the total variance is attributable to differences between subjects rather than measurement error.

• Model

  • Two-way random: Both subjects and raters (or time points) are treated as random effects (i.e., raters/timepoints are considered a random sample from a larger population).
  • Two-way mixed: Subjects are random but raters/time points are fixed (e.g., specifically Time1 and Time2).

• Type

  • Consistency: Focuses on whether subjects keep the same relative ranking across measurements (ignores systematic differences in mean levels between raters/timepoints).
  • Absolute agreement: Focuses on exact agreement of measurement values (penalizes systematic differences in means).

A conceptual form of the ICC is:

\[ \mathrm{ICC} \;=\; \frac{\sigma^2_{\text{subject}}} {\sigma^2_{\text{subject}} + \sigma^2_{\text{error}} + (\text{if applicable } \sigma^2_{\text{rater}})} \]

where \(\sigma^2_{\text{subject}}\) is the between-subject variance, \(\sigma^2_{\text{error}}\) is the residual/error variance, and \(\sigma^2_{\text{rater}}\) is the variance due to raters/timepoints when included.

Common, rule-of-thumb ranges (e.g., Landis & Koch or similar) are:

• \(\mathrm{ICC} \ge 0.75\): Excellent
  
• \(0.60 \le \mathrm{ICC} < 0.75\): Good
  
• \(0.40 \le \mathrm{ICC} < 0.60\): Fair
  
• \(\mathrm{ICC} < 0.40\): Poor

1. Appropriateness of ICC for the Current Data

❌ Not appropriate. ❌

For nominal data such as binary preference selection (0/1) and categorical activity numbers, the Intraclass Correlation Coefficient (ICC) is not an appropriate reliability measure.

2. Cohen’s Kappa

Cohen’s Kappa measures how consistently two assessments agree beyond chance. In simple terms, it quantifies how well the results of the first and second assessments match.

Example:

• A participant selects “Visiting a hospital” in the first assessment
• The same activity is selected again in the second assessment
  → This is counted as agreement.

Kappa summarizes such agreements across all activities. Cohen’s Kappa (\(\kappa\)) is defined as \[ \kappa = \frac{P_o - P_e}{1 - P_e}, \] where:

• \(P_o\) is the observed proportion of agreement between the first and second assessments.
• \(P_e\) is the expected proportion of agreement by chance, calculated from the marginal distributions of the two assessments.

The observed agreement \(P_o\) is given by \[ P_o = \frac{\sum_{i=1}^{K} n_{ii}}{N}, \] where:

• \(K\) is the number of categories,
• \(n_{ii}\) is the number of cases classified into category \(i\) in both assessments,
• \(N\) is the total number of observations.

The expected agreement \(P_e\) is calculated as \[ P_e = \sum_{i=1}^{K} \left( \frac{n_{i+}}{N} \cdot \frac{n_{+i}}{N} \right), \] where:

• \(n_{i+}\) is the total number of cases classified into category \(i\) in the first assessment,
• \(n_{+i}\) is the total number of cases classified into category \(i\) in the second assessment.

How Is Kappa Different from Simple Agreement Rate?

• Simple agreement rate:
  Number of matching responses ÷ total responses

• Problem:
  Some agreement may occur by chance.

Example: Guessing heads or tails yields about 50% agreement by chance.

Kappa adjusts for this by subtracting chance agreement, providing a more accurate reliability estimate.

Interpretation of Kappa Values

Example :

Suppose four participants responded as follows:

Simple agreement rate = 3 / 4 = 75%

Kappa evaluates this agreement after accounting for chance.

3. Agreement of Top 1 Preferred Activities by Domain

The goal is to examine whether the Top 1 preferred activity selected in each domain is consistent between the first and second assessments. For each domain, Cohen’s Kappa can be calculated to assess intra-rater reliability between the two time points.

Example Variables ”

Instrumental Activities

• acs_instrumental_top1 (first assessment)
• 2acs_instrumental_top1 (second assessment)

Leisure Activities

• acs_leisure_top1
• 2acs_leisure_top1

Social Activities

• acs_social_top1
• 2acs_social_top1

Each pair represents the Top 1 selection from the first and second assessments.
A separate Kappa value is computed for each domain.

Although the Top 1 variables are coded numerically, they represent categorical labels, not ordered or quantitative values.

Examples:

• A value of 5 does not mean “greater” than 7
• It simply refers to “Activity #5” versus “Activity #7”
• Therefore, during analysis (e.g., in SPSS), these variables must be treated as Nominal.

4. Conclusion

Comparing the Top 1 preferred activities using Cohen’s Kappa for each domain is an appropriate way to assess intra-rater reliability. Three Kappa values will be obtained (instrumental, leisure, social), and higher values (commonly ≥ 0.6) indicate more stable and reliable assessments.