Jekyll2026-03-08T04:42:28+00:00https://jason1ko.github.io/feed.xmlHan Jun KoAn amazing website.Han Jun KoIntroduction2025-03-01T23:59:59+00:002025-03-01T23:59:59+00:00https://jason1ko.github.io/data%20science%20institute/DSI-intro

Data Science Institute

Introduction

The Yonsei University Data Science Institute is a center where graduate students in the Department of Statistics and Data Science provide statistical consulting to graduate students both within and outside Yonsei. When students encounter questions about statistical methodology while writing their theses, the institute helps them clarify their research design and suggests appropriate analytical solutions.

I participated in this institute as a consulting researcher for one semester. During that time, I handled several consultations, and I have summarized some of the main cases and lessons learned on this blog. Right now, this page serves as the Introduction under the Data Science Institute category on the left sidebar. If you would like to see the specific consulting cases, please read the posts below in order, starting with Consultation 1, then Consultation 2, and so on.

]]>Han Jun KoConsultation 12025-03-01T23:59:00+00:002025-03-01T23:59:00+00:00https://jason1ko.github.io/data%20science%20institute/DSI-c1

Consultation 1: Repeated-Measures ANOVA for Office-Image Effects on Stress

I. Client’s Inquiry

This study aims to examine the effects of indoor office environment images on stress levels. To achieve this, three spatial variables were defined: color (cool/warm), ceiling height (low/high), and window view (urban/natural).

The experiment consists of two stages (Test 1 and Test 2) conducted on the same group of 30 participants.

Test 1:

To investigate the individual effects of each spatial variable, images representing each condition (a total of 6 images) will be presented separately.
Paired-sample t-tests will be used to compare stress indices across conditions in order to determine whether there are significant differences between them.

Test 2:

In the second stage, combinations of the three variables will be used to create 2 × 2 × 2 conditions (color × ceiling height × view), resulting in a total of 8 image conditions.
Stress levels will be measured under each condition, with stress index as the dependent variable and environmental condition (8 combinations) as the independent variable.
A One-way Repeated Measures ANOVA will be conducted to examine whether there are significant differences in mean stress levels among the combined environmental conditions.

We would like to ask for your review regarding whether this statistical design and analysis approach is appropriate, and whether there might be a simpler or clearer method for analyzing the data.
In particular, we would like to confirm whether the use of One-way ANOVA or One-way Repeated Measures ANOVA is suitable in this case, and whether the sample size of 30 participants is sufficient or should be increased.

II. Answer

1. Test 1

The model is given as \[Y_{ij} = \mu + \tau_j + s_i + \epsilon_{ij}\] where

  • \(Y_{ij}\) : ith subject’s stress figure when exposed to jth condition (color, …)
  • \(\mu\) : mean
  • \(\tau_j\) : the effect of color condition ( Cool colors vs. warm colors)
  • \(s_i\)​: the effect of subjects (random effect)
  • \(\epsilon_{ij}\)​: error

the way to adjust the significance level (Bonferroni) : \(\alpha_B = \alpha/n\) That is: each test is considered significant only if its p-value is less than \(\alpha/n\). This ensures that the overall significance level remains at \(\alpha\).

If there are more than 3 levels? => One-way Repeated Measures ANOVA. ex) 3 colors.

Idendical model, but the number of \(\tau_j\) is 3: \(j=1,2,3\). Then, the null hypothesis is

\[H_0 : \tau_1 = \tau_2 = \tau_3\]

If the null hypothesis is rejected, it means that at least one of the levels has a different effect. The specific levels that differ can be identified through pairwise t-tests. For instance,

  • cool color vs neutral color
  • cool color vs warm color
  • neutral color vs warm color

At this point, a Bonferroni correction (or similar adjustment) should be applied.

We may have the following question : > If we conduct a Three-way Repeated Measures ANOVA from the start, can we still check the significance of each factor in the same way?

The Three-way RM ANOVA can encompass the t-test results from Test 1, but it does not guarantee completely identical outcomes. Test 1 is designed to evaluate each factor in isolation, whereas the Three-way RM ANOVA analyzes the effect of each factor while all factors act simultaneously.

If all interaction effects are not significant, the main effects of each factor can be interpreted independently. In this case, the main effect results from the Three-way RM ANOVA and the results from analyzing each factor separately using t-tests can provide practically similar or even identical conclusions.

However, the main effects in a Three-way RM ANOVA represent the average effects while controlling for the mean influence of other factors, whereas t-tests examine the effects with other factors held constant. Therefore, due to the differences in design and interpretation, the numerical results may differ.

2. Test 2

(1) one-way RM anova

model : \[Y_{ij} = \mu + \tau_j + s_i + \epsilon_{ij}\]

  • \(\tau_j\) : jth effect (j = 1,…,8)
  • \(s_i\) : ith subject effect, random effect

Then,

null hypothesis : tau_1 = … = tau_8, alternative hypothesis : At least one of them has a different stress index.

If the null hypothesis is rejected, post-hoc tests can be used to determine which conditions differ.

In conclusion, this approach is suitable if the goal is simply to test whether the three factors have no overall effect. However, it is not appropriate if you want to understand the individual effects of each factor or their interaction effects.

(2) three-way rm anova

\[Y_{ijkl} = \mu + color_j + level_k + view_l + ... + s_i + \epsilon_{ijkl}\] - \(s_i\) : subject effect (random effect)

It also allows you to identify both the main effects of each factor and the presence of interaction effects. Through this, you can determine: - whether each factor (e.g., color) has a significant effect, and - whether there are interaction effects between two or among all three factors.
(This is done by checking whether p < 0.05 ⇒ significant.)

3. Better model

(1) Client’s Analysis Objective:

  • To determine whether each design element (color, ceiling height, and window view) has an individual effect.

  • To see whether stress levels differ depending on the combination of conditions.

  • If possible, to further identify which specific combinations are particularly effective.

(2) Comparison:

  • Interpretability: One-way RM ANOVA shows only the mean differences among the 8 conditions,
    whereas Three-way RM ANOVA reveals the main effects of color, ceiling height, and window view, as well as their interaction effects.

  • Clarity of interpretation: One-way RM ANOVA allows statements like “Which combination resulted in lower stress,”
    while Three-way RM ANOVA allows statements such as “What is the average effect of color on stress.”

(3) Conclusion: Three-way RM ANOVA

The client’s main interest is not just in comparing combinations,
→ but in understanding how much each design factor (color, ceiling height, view) influences stress.

A One-way RM ANOVA only shows whether there are differences among the 8 combinations,
→ but it can never tell whether those differences are due to color, ceiling height, or their interaction.

In contrast, a Three-way RM ANOVA can analyze
→ the main effects of color, ceiling height, and window view,
→ as well as all possible interaction effects among them.

Given that the core purpose of the experiment is to understand how these design factors affect stress,
→ the Three-way RM ANOVA is a more suitable approach than the One-way RM ANOVA.

4. [30] participants

Appropriate Minimum Benchmark

When the sample size is around 30 or more, the sampling distribution of the mean approximately follows a normal distribution.
→ Therefore, under the assumption of normality, the use of t-tests and ANOVA becomes valid.
→ For this reason, “30 participants” is widely used as a practical benchmark in research.

cf) Advantage of Repeated Measures Design:
It provides higher statistical power with a smaller sample size compared to an independent-sample design.

]]>Han Jun KoConsultation 22025-03-01T23:50:00+00:002025-03-01T23:50:00+00:00https://jason1ko.github.io/data%20science%20institute/DSI-c2

Consultation 2: Repeated-Measures ANOVA and Multilevel Mediation for a Four-Type Persuasion Study

I. Client’s Inquiry

1. Context

The client seeks statistical consulting to (a) choose appropriate analysis methods for their study and (b) potentially outsource analyses if they are too difficult to perform independently.

2. Key Questions

Q1. Appropriateness of H1 & H2 Analyses

  • Design: within-subjects (repeated measures); all participants experienced four persuasion types (A, B, C, D).
  • Sample size: n = 286.
  • Assumptions: normality violated, sphericity violated.
  • Question: Is it acceptable to use One-Way Repeated Measures ANOVA with Greenhouse–Geisser correction under these violations?

Q2. Possibility of Mediation Analysis for H3

  • Independent variable structure: 4 equally weighted types; there is no single “reference” type, so standard dummy-coding with one baseline is not conceptually ideal.
  • Design issue: All participants saw all four types (repeated measures); classic mediation frameworks often assume between-subjects manipulations.
  • Current fallback: Simple regressions testing whether perceived autonomy significantly predicts behavioral intention, interaction satisfaction, and reuse intention.
  • Question: Despite the within-subjects design and four-level factor, is there a valid mediation approach for H3?

Q3 Appropriate Analysis for Manipulation-Check Items

  • Example (Type A): show that emotional appeal / gain framing scores are high, whereas rational appeal / loss framing scores are low.
  • Question: What statistical tests should be used to demonstrate that the manipulations worked as intended?

Q4. Balanced Latin Square & Whether an Independent-Groups Design Is Preferable

3. Proposed/Considered Analysis Techniques

  1. Repeated-Measures ANOVA (primary for H1/H2).
  2. Mediation analysis / Simple regression (for H3, as feasible).

4. Hypotheses

  • H1: Do behavioral intention, interaction satisfaction, and reuse intention differ by persuasion type?
  • H2: Does perceived autonomy differ by persuasion type?
  • H3 (Version 1 – Mediation): Perceived autonomy mediates the relationship between persuasion type and each outcome (behavioral intention, interaction satisfaction, reuse intention).
  • H3 (Version 2 – Simple Regression): Within each persuasion type, perceived autonomy significantly predicts behavioral intention, interaction satisfaction, and reuse intention.

5. Data Description

  • N = 286 survey respondents.
  • Scales: 5-point Likert, multiple-choice questionnaire items.
  • Mediator: Perceived autonomy.
  • Outcomes (DVs): Behavioral intention, interaction satisfaction, reuse intention.
  • Independent variable (IV): Four persuasion types (operationalized via image stimuli).

II. Answer of Q1

1. Model

Conclusion: A one-way repeated-measures (RM) ANOVA is the appropriate analysis.

  • Design: Within-subjects. All participants evaluated four persuasion types (A, B, C, D).
  • Dependent variables:
    • For H2: Perceived autonomy
    • For H1: Behavioral intention, interaction satisfaction, reuse intention
  • The same 286 participants experienced all four conditions and provided repeated responses for the same outcomes under each condition. Because observations are not independent across conditions within participants, a repeated-measures approach is required.

Model: \[ Y_{ij} = \mu + \alpha_j + s_i + \epsilon_{ij} \] where \(Y_{ij}\) is the outcome for participant \(i\) in condition \(j\), \(\mu\) is the grand mean, \(\alpha_j\) is the fixed effect of persuasion type \(j\), \(s_i\) is the (random) subject effect, and \(\epsilon_{ij}\) is the residual.

cf) Note on Two-Way Repeated-Measures ANOVA - Technically feasible, but here the client did not separate appeal (emotional vs. rational) and framing (gain vs. loss) as two orthogonal factors.
- Instead, these were combined into a single factor (“persuasion type”).
- Since the goal is to compare differences among the four types, one-way RM ANOVA better matches the research objective.

2. Normality

(1) Kolmogorov–Smirnov (K–S) Normality Test

Definition. Tests whether the sample empirical CDF (ECDF) differs significantly from the CDF of a normal distribution.

  • For sample \(X_1, X_2, \ldots, X_n\), the ECDF is
    \[ F_n(x) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}(X_i \le x). \]
  • Test statistic:
    \[ D = \sup_x \lvert F_n(x) - F(x) \rvert, \] the maximum absolute gap between the ECDF and the target normal CDF.
  • Hypotheses:
    \(H_0\): the sample follows a normal distribution.
    \(H_1\): the sample does not follow a normal distribution.
  • Decision rule: if \(p\)-value \(< 0.05\), reject \(H_0\) (i.e., normality is violated).

Procedure 1. Compute the ECDF at each observation. 2. Compute the normal CDF (with appropriate mean/SD). 3. Compute \(D\). 4. Compare to the critical value (or obtain a \(p\)-value) at significance level \(\alpha\) and test \(H_0\).

(2) Shapiro–Wilk Normality Test

For sample \(x_1, x_2, \ldots, x_n\) (with order statistics \(x_{(1)} \le \cdots \le x_{(n)}\)), the statistic is \[ W = \frac{\left(\sum_{i=1}^n a_i\, x_{(i)}\right)^2}{\sum_{i=1}^n (x_i - \bar{x})^2}, \] where \(a_i\) are weights derived under normality.

Interpretation

  • Numerator: squared linear combination aligning observed order statistics with those expected under normality.
  • Denominator: sample variance (total variation).
  • \(W \approx 1\) indicates strong concordance with normality; smaller \(W\) indicates departures from normality.

Hypotheses

  • \(H_0\): the sample follows a normal distribution.
  • \(H_1\): the sample does not follow a normal distribution.
    Reject \(H_0\) when \(p\)-value \(< 0.05\).

(3) Appropriateness of Normality Testing in This Study

1) On averaging item scores within each condition

Averaging two items per condition can attenuate non-normality. Therefore, it is preferable to test normality on the original item-level scores rather than on the averaged composites.

2) Accounting for the repeated-measures structure

Because the design is within-subjects, measurements are correlated across conditions (A/B/C/D). Testing normality separately on each condition’s raw scores ignores this correlation and can bias interpretation.

A more appropriate approach is to examine the residuals from the repeated-measures model: \[ Y_{ij} = \mu + \alpha_j + s_i + \epsilon_{ij}, \] where \(\alpha_j\) is the fixed effect of condition \(j\), \(s_i\) is the (random) subject effect, and \(\epsilon_{ij} \sim \mathcal{N}(0,\sigma^2)\).

  • Pool all residuals across A, B, C, D and assess their normality (e.g., Shapiro–Wilk or K–S on the residuals).
  • Why not per-condition tests? The subject-specific effect \(s_i\) influences all conditions; evaluating a single condition in isolation incompletely accounts for this shared effect.

Visual diagnostics are recommended alongside tests (e.g., QQ-plots of residuals).

3. Other Models (Beyond Simple One-Way ANOVA)

(1) One-Way ANOVA (Between-Subjects)

  • Appropriate when different participant groups each experience only one condition.
  • Assumes independent observations across groups.
  • Not applicable here because all participants experienced all four conditions (A/B/C/D).

(2) Repeated-Measures MANOVA

  • Participants respond to multiple dependent variables across multiple within-subject conditions (e.g., A/B/C/D).
  • Analyzes several outcomes jointly (e.g., behavioral intention, interaction satisfaction, reuse intention).
  • Useful when you want to test the common effect of condition across correlated outcomes simultaneously.
    • Example: If those three outcomes are correlated and you want to know how they jointly change across A/B/C/D, RM-MANOVA is appropriate.

(3) Most Suitable Approach for This Study

Focus Do we analyze correlations among DVs? Recommended analysis
Compare each DV separately across A/B/C/D No (no explicit need stated) Separate one-way RM ANOVA per DV (current approach is appropriate)

4. Sphericity

  • Greenhouse–Geisser correction is a standard and appropriate remedy when sphericity is violated.
  • No issue using it in this context.

III. Answer of Q2

1. Mediation Analysis

Mediation analysis tests whether the effect of an independent variable X on a dependent variable Y occurs through a mediator M.

Structure:

X (persuasion type) → M (perceived autonomy) → Y (behavioral outcome)
↘────────────────────────────────────────→ Y (direct path)

Key effects:

  • a path: X → M
  • b path: M → Y (controlling for X)
  • c’ path: X → Y (direct effect)
  • Indirect effect: a · b

2. Why Simple Regression Is Insufficient

The client proposes to:

  • Run simple regressions within each persuasion type (A, B, C, D) to test if perceived autonomy → behavioral outcomes (behavioral intention, satisfaction, reuse intention).

But this only examines the “b path” in isolation.
It does not consider:

  • The impact of persuasion type X on perceived autonomy (a path)
  • Nor the full mediation structure
  • Nor the repeated-measures (within-subject) nature of the data

Furthermore, regression assumes independence of observations, which is violated here: the same participants rated all 4 persuasion types.

3. Correct Method: Repeated Measures Mediation (Multilevel Mediation)

Because the data involve:

  • All participants rating all 4 persuasion types (within-subjects factor)
  • Responses nested within participants

→ The correct approach is Multilevel Mediation Analysis (also called Repeated Measures Mediation).

4. Model Structure

Level 1 (within participants):

\[\begin{aligned} M_{ij} &= \alpha_{0i} + a\,X_{ij} + r_{1ij} \\ Y_{ij} &= \beta_{0i} + c' \, X_{ij} + b \, M_{ij} + r_{2ij} \end{aligned}\]

  • \(i\): participant
  • \(j\): condition (A, B, C, D)

Level 2 (between participants):

\[ \alpha_{0i} \sim \mathcal{N}(\gamma_{0}, \tau_{\alpha}^{2}), \quad \beta_{0i} \sim \mathcal{N}(\delta_{0}, \tau_{\beta}^{2}) \]

→ This models both the fixed effects (mediation paths) and random intercepts per participant.

5. Conclusion (Key Message for Client)

  • Yes, H3 mediation analysis is possible despite the repeated-measures design.
  • But it must be done using a Multilevel Mediation model that accounts for the within-subject structure of the data.
  • The simple regression approach may offer partial insight but cannot test for mediation and violates statistical assumptions (independence of residuals).
  • We strongly recommend using tools such as lme4 and mediation (or brms for Bayesian estimation) to perform proper repeated-measures mediation.

IV. Answer of Q3

1. What is a Manipulation Check?

A manipulation check verifies that the experimental manipulation (the intended effect of the independent variable) was perceived by participants as designed.
Example: If you showed participants a “emotional appeal” chatbot vs. a “rational appeal” chatbot, you might ask, “How emotional was the chatbot you just saw?” (1 = not at all, 5 = very much).

3. Non-Normality: Robust Alternatives

  • Paired t-test → Wilcoxon signed-rank test
  • RM ANOVA → Friedman test

However, with \(n=286\), modest deviations from normality are typically not consequential due to large-sample robustness (CLT), especially for paired mean differences and RM ANOVA residuals—provided there are no extreme outliers or severe skewness. Complement tests with QQ-plots and residual diagnostics.

V. Answer of Q4

1. Appropriateness of a Balanced Latin Square

(1) Example

A typical Balanced Latin Square for four conditions:

Order group 1st 2nd 3rd 4th
Group 1 A B D C
Group 2 B C A D
Group 3 C D B A
Group 4 D A C B

Each of A, B, C, D appears exactly once in each serial position.

(2) Is it suitable? (re: “minimizing reviewer concerns about order effects”)

  • In within-subjects designs, a Balanced Latin Square is a gold-standard method for order control—more systematic than simple randomization.
  • In repeated-measures experiments, responses can depend on presentation order (e.g., novelty/primacy for the first stimulus; fatigue for the last). The balanced scheme distributes such effects evenly across conditions.

(3) Must group sizes be identical?

  • Not strictly. Minor imbalances (±1–2 participants) are not problematic.
  • Large imbalances (e.g., 50 vs 15) can distort results; avoid pronounced disparities.

(4) Are there “better” alternatives?

  • For repeated-measures order control, the Balanced Latin Square is already authoritative and appropriate.
  • Reporting that a Latin Square was used typically conveys high design rigor to reviewers.

Sample size sufficiency - Minimum needed ≈ (# of orders) × (minimum per order). - Here: 4 orders and N = 286, i.e., about 70 per order → ample power and validity.

Optional model-based check for order effects

\[ Y_{ijk} = \mu + \alpha_j + s_i + \gamma_k + \epsilon_{ijk} \]

  • \(\alpha_j\): fixed effect of condition \(j\) (A/B/C/D)
  • \(s_i\): random subject effect
  • \(\gamma_k\): fixed effect of order group \(k=1,\dots,4\)

If \(\gamma_k\) is not significant, order effects are negligible—evidence that the Latin Square successfully controlled order.

In short, the Latin Square is a validated method for order control.

2. Is an Independent-Groups (Between-Subjects) Design More Appropriate?

(1) Summary

  1. Is the current repeated-measures design appropriate? ✅ Yes—very appropriate and aligned with the study goals.
  2. Would between-subjects be better? ❌ No—likely reduces accuracy (more noise from individual differences).

(2) Side-by-side comparison

Aspect Repeated-Measures (current) Independent-Groups (alternative)
Statistical power ✅ Higher (shared variance removed) ❌ Lower (more error from individual differences)
Individual differences ✅ Controlled (same person rates all conditions) ❌ Uncontrolled
Order effects ⚠️ Potential → mitigate with Balanced Latin Square ✅ Absent by design
Fit to research aim (compare persuasion types) ✅ Excellent ❌ Risk that group composition (traits) confounds condition effects

(3) Answers to common concerns

“Isn’t it more valid to assign each condition to a different random group?”
No. Individual differences can obscure or distort true condition effects.
Example: If Group A is more sensitive to emotional content and Group D is less so, the outcomes may reflect participant traits, not chatbot persuasion.

“What about fatigue or order effects in repeated measures?”
Valid concern, but Balanced Latin Square minimizes order effects. Some fatigue may remain, but it’s a typical and acceptable tradeoff for cleaner comparisons. Many psychological experiments use repeated-measures for this reason.

“Do we need to recollect data?”
Absolutely not. Your current design is ideal for the research questions.

]]>Han Jun KoConsultation 32025-03-01T23:45:59+00:002025-03-01T23:45:59+00:00https://jason1ko.github.io/data%20science%20institute/DSI-c3

Consultation3 : GLMM/GEE & Kappa for ACS Preference Data

I. Client’s Inquiry 1

Fifty patients with aphasia were asked to select their preferred activities using the ACS activity cards.
The ACS consists of 67 cards in total, including 33 instrumental activities, 18 leisure activities, and 16 social activities, each represented by a printed photograph.

First, the researcher presented the 33 instrumental activity cards to each patient one by one and asked whether the patient would like to perform the activity.

  • If the patient did not want to do the activity, it was coded as 0.
  • If the patient wanted to do the activity, it was coded as 1.
  • For example, if acs_instrumental_1 = 1 in the Excel coding file, it indicates that the patient prefers instrumental activity #1 (visiting a hospital).

After all 33 instrumental activities were classified, only the activities selected as preferred were gathered, and the patient was asked to choose their Top 1 preferred activity among them. The activity number was recorded in acs_instrumental_top1. For example, if acs_instrumental_top1 = 5, it indicates that instrumental activity #5 (preparing a meal) was selected as the top preference.

After completing the instrumental activity classification, the same procedure was applied to:

  • Leisure activities (acs_leisure_34–acs_leisure_51, acs_leisure_top1)
  • Social activities (acs_social_52–acs_social_67, acs_social_top1)

To evaluate intra-rater reliability of the ACS, a second assessment was conducted for 30 out of the 50 patients, 2 weeks to 4 months after the first assessment.

Demographic variables

  1. sex: 1 = male, 2 = female

  2. age: numeric

  3. education:

    • 1 = no schooling
    • 2 = elementary school
    • 3 = middle school
    • 4 = high school
    • 5 = college

  4. job:

    • 1 = unemployed
    • 2 = production
    • 3 = service
    • 4 = sales
    • 5 = office
    • 6 = professional
    • 7 = managerial

  5. hand: 1 = right-handed, 2 = left-handed

What statistical methods are appropriate for this analysis, and how should the variables be coded?

Answer of Inquiry 1

2. Why restructuring to long format is needed

  • To fit GLMM/GEE properly, the data should be converted from wide format (33 columns of outcomes) to long format, where each row represents one (subject, activity) observation.

Long-format example (key idea)

  • In long format:

    • activity_id = 1 corresponds to acs_instrumental_1
    • response is the value of that variable (0 or 1)

Example rows:

subject_id activity_id response sex age education job hand
1 1 value of acs_instrumental_1 1/2 … … … 1/2
1 2 value of acs_instrumental_2 1/2 … … … 1/2
… … … … … … … …

3. Variable coding advice (demographics)

  • The current coding (e.g., sex as 1/2, education as 1–5, job as 1–7, hand as 1/2) is generally fine as long as:

    • In SPSS, these are explicitly treated as categorical variables (Nominal/Ordinal as appropriate, and specified as “categorical predictors” in the model).

  • For easier interpretation (optional but recommended), binary variables can be recoded to 0/1:

    • sex: male = 0, female = 1
    • hand: right = 0, left = 1 This is not strictly required, but coefficients become more intuitive to interpret.

I. Client’s Inquiry 2

The objective of this study is to examine the intra-rater reliability of the ACS assessment. The dataset includes:

  • Preference selections for activities
    (acs_instrumental_1–33, acs_leisure_34–51, acs_social_52–67)
  • The activity number selected as Top 1 preference in each domain

for both the first and second assessments.

I would like to ask:

  1. Which statistical methods are appropriate for assessing the reliability of
    • the selected preferred activities, and
    • the Top 1 preferred activity?
  2. Whether the current variable coding scheme is appropriate for conducting reliability analysis.

For example:

  • acs_instrumental_1 indicates whether Instrumental Activity #1 was selected in the first assessment.
  • 2acs_instrumental_1 indicates whether the same activity was selected in the second assessment.

Answer of Inquiry 2

0. Intraclass Correlation Coefficient (ICC) — Reliability Assessment

The Intraclass Correlation Coefficient (ICC) is a statistic that quantifies the degree of agreement (or consistency) between measurements of the same targets made at different times or by different raters. ICC is typically used for continuous measurements and evaluates how much of the total variance is attributable to differences between subjects rather than measurement error.

  • Model

    • Two-way random: Both subjects and raters (or time points) are treated as random effects (i.e., raters/timepoints are considered a random sample from a larger population).
    • Two-way mixed: Subjects are random but raters/time points are fixed (e.g., specifically Time1 and Time2).

  • Type

    • Consistency: Focuses on whether subjects keep the same relative ranking across measurements (ignores systematic differences in mean levels between raters/timepoints).
    • Absolute agreement: Focuses on exact agreement of measurement values (penalizes systematic differences in means).

A conceptual form of the ICC is:

\[ \mathrm{ICC} \;=\; \frac{\sigma^2_{\text{subject}}} {\sigma^2_{\text{subject}} + \sigma^2_{\text{error}} + (\text{if applicable } \sigma^2_{\text{rater}})} \]

where \(\sigma^2_{\text{subject}}\) is the between-subject variance, \(\sigma^2_{\text{error}}\) is the residual/error variance, and \(\sigma^2_{\text{rater}}\) is the variance due to raters/timepoints when included.

Common, rule-of-thumb ranges (e.g., Landis & Koch or similar) are:

  • \(\mathrm{ICC} \ge 0.75\): Excellent
  • \(0.60 \le \mathrm{ICC} < 0.75\): Good
  • \(0.40 \le \mathrm{ICC} < 0.60\): Fair
  • \(\mathrm{ICC} < 0.40\): Poor

1. Appropriateness of ICC for the Current Data

❌ Not appropriate. ❌

For nominal data such as binary preference selection (0/1) and categorical activity numbers, the Intraclass Correlation Coefficient (ICC) is not an appropriate reliability measure.

(1) Why Is ICC Inappropriate?

  1. ICC is designed for continuous (numerical) data ICC is used when the magnitude and distance between values are meaningful. Examples:

    • Blood pressure values
    • Test scores
    • Speed, distance
    • Mean scores of questionnaire items

    → ICC is meaningful only when the numerical value itself carries quantitative meaning.

  2. The current data are nominal (categorical) Example:

    • acs_instrumental_top1 = 5 → “Preparing meals”
    • acs_instrumental_top1 = 8 → “Doing laundry”

    The numerical difference between 5 and 8 has no quantitative meaning.
    Treating these category labels as numerical values is a logical error. ICC relies on variance and covariance between numerical values, so applying ICC to activity codes produces meaningless results.

(2) What Should Be Used Instead?

Data type Appropriate reliability measure
Nominal (0/1 selection, Top 1 activity code) ✅ Cohen’s Kappa
Ordinal (e.g., ranked satisfaction levels) 🔸 Weighted Kappa
Continuous (scores, measurements) ✅ ICC

(3) When Is ICC Appropriate?

If an assessment produces continuous scores that are repeatedly measured in the same way, ICC is appropriate.

Example : Treatment effect rated on a 0–100 scale

However, activity codes and binary selections are categorical, so ICC should not be used here.

2. Cohen’s Kappa

Cohen’s Kappa measures how consistently two assessments agree beyond chance. In simple terms, it quantifies how well the results of the first and second assessments match.

Example:

  • A participant selects “Visiting a hospital” in the first assessment
  • The same activity is selected again in the second assessment
    → This is counted as agreement.

Kappa summarizes such agreements across all activities. Cohen’s Kappa (\(\kappa\)) is defined as \[ \kappa = \frac{P_o - P_e}{1 - P_e}, \] where:

  • \(P_o\) is the observed proportion of agreement between the first and second assessments.
  • \(P_e\) is the expected proportion of agreement by chance, calculated from the marginal distributions of the two assessments.

The observed agreement \(P_o\) is given by \[ P_o = \frac{\sum_{i=1}^{K} n_{ii}}{N}, \] where:

  • \(K\) is the number of categories,
  • \(n_{ii}\) is the number of cases classified into category \(i\) in both assessments,
  • \(N\) is the total number of observations.

The expected agreement \(P_e\) is calculated as \[ P_e = \sum_{i=1}^{K} \left( \frac{n_{i+}}{N} \cdot \frac{n_{+i}}{N} \right), \] where:

  • \(n_{i+}\) is the total number of cases classified into category \(i\) in the first assessment,
  • \(n_{+i}\) is the total number of cases classified into category \(i\) in the second assessment.

How Is Kappa Different from Simple Agreement Rate?

  • Simple agreement rate:
    Number of matching responses ÷ total responses

  • Problem:
    Some agreement may occur by chance.

Example: Guessing heads or tails yields about 50% agreement by chance.

Kappa adjusts for this by subtracting chance agreement, providing a more accurate reliability estimate.

Interpretation of Kappa Values

Kappa value Interpretation
1.00 Perfect agreement
0.81–0.99 Almost perfect agreement
0.61–0.80 Substantial agreement
0.41–0.60 Moderate agreement
0.21–0.40 Fair agreement
0.00–0.20 Slight agreement
< 0.00 Worse than chance

Example :

Suppose four participants responded as follows:

Participant First Second
A Want Want
B Do not want Want
C Want Want
D Do not want Do not want

Simple agreement rate = 3 / 4 = 75%

Kappa evaluates this agreement after accounting for chance.

3. Agreement of Top 1 Preferred Activities by Domain

The goal is to examine whether the Top 1 preferred activity selected in each domain is consistent between the first and second assessments. For each domain, Cohen’s Kappa can be calculated to assess intra-rater reliability between the two time points.

Example Variables ”

Instrumental Activities

  • acs_instrumental_top1 (first assessment)
  • 2acs_instrumental_top1 (second assessment)

Leisure Activities

  • acs_leisure_top1
  • 2acs_leisure_top1

Social Activities

  • acs_social_top1
  • 2acs_social_top1

Each pair represents the Top 1 selection from the first and second assessments.
A separate Kappa value is computed for each domain.

Although the Top 1 variables are coded numerically, they represent categorical labels, not ordered or quantitative values.

Examples:

  • A value of 5 does not mean “greater” than 7
  • It simply refers to “Activity #5” versus “Activity #7”
  • Therefore, during analysis (e.g., in SPSS), these variables must be treated as Nominal.

4. Conclusion

Comparing the Top 1 preferred activities using Cohen’s Kappa for each domain is an appropriate way to assess intra-rater reliability. Three Kappa values will be obtained (instrumental, leisure, social), and higher values (commonly ≥ 0.6) indicate more stable and reliable assessments.

]]>Han Jun KoConsultation 42025-03-01T23:45:50+00:002025-03-01T23:45:50+00:00https://jason1ko.github.io/data%20science%20institute/DSI-c4

Consultation 4: Multilevel SEM for Logic Model Evaluation with Small Cluster Sizes

I. Client’s Inquiry

1. Analysis goal

Based on a logic model framework, the study aims to empirically test a sequential causal pathway that links public library performance outcomes.

2. Planned statistical methodology

  • Use Structural Equation Modeling (SEM) to test the logic-model causal pathway, including both direct and indirect (mediated) effects.

  • Conceptual chain: Inputs → Activities → Outputs → Outcomes

  • Outcomes are further structured as:

    • Short-term outcomes

    • Intermediate outcomes

    • Long-term outcomes

3. Data description

  • Variables collected:

    • Inputs

    • Activities

    • Outputs

    • Short-term outcomes

    • Intermediate outcomes

    • Long-term outcomes

  • Note: A conceptual research model figure has been prepared (attached by the researcher).

4. Key issue (mixed unit of analysis)

Some variables are measured at the library level, while others are measured at the individual user level.

  • Inputs, Activities, Outputs: quantitative data at the library level

  • Short-/Intermediate-/Long-term outcomes: user-level survey responses (Likert scale)

5. Intended analysis level

  • The researcher wants to perform the analysis at the library level.

  • In the survey, each user selected the library they mainly use and answered about that library, so survey responses can be grouped by library.

6. Main question

  • Many libraries have fewer than 30 respondents (about 46% of all libraries).

  • Is SEM still feasible when some groups (libraries) have n < 30? (Excluding all libraries with fewer than 30 respondents would remove too much data.)

  • Since small group sizes are believed to reduce reliability, the researcher asks whether there are practical analytic approaches that still allow SEM (or related modeling) without dropping most of these libraries.

II. Background

0. What is Structural Equation Modeling (SEM)?

Structural Equation Modeling (SEM) is a statistical framework that allows researchers to test multiple causal (direct and indirect) relationships among variables simultaneously. SEM is typically composed of two main parts: a measurement model and a structural model.

0.1 Measurement model

The measurement model specifies how observed indicators (measured variables) represent latent variables (unobserved constructs).

\[ \begin{aligned} x & = \Lambda_x \xi + \delta \quad \text{(measurement model for exogenous latent variables)} \\ y & = \Lambda_y \eta + \epsilon \quad \text{(measurement model for endogenous latent variables)} \end{aligned} \]

  • \(x\): vector of observed indicators for exogenous latent variables

    • In this study, \(x\) can correspond to the library-level measured variables that are treated as exogenous inputs to the causal chain.

    • Inputs / Activities / Outputs (e.g., budget, staffing, number of programs, number of loans, number of events, etc., depending on the dataset).

  • \(y\): vector of observed indicators for endogenous latent variables

    • In this study, \(y\) can correspond to user-survey items (Likert-scale indicators) that measure outcome-related constructs.

    • Short-term / Intermediate / Long-term outcomes (e.g., satisfaction items, perceived usefulness, learning/skill improvement items, intention to revisit, community engagement, etc.).

  • \(\xi\): vector of exogenous latent variables (latent predictors)

    • Example interpretation: a latent construct such as “Library Resources/Capacity” or “Service Provision Intensity”, measured by library-level indicators in \(x\).

  • \(\eta\): vector of endogenous latent variables (latent outcomes/mediators)

    • Example interpretation: latent constructs such as “Short-term Outcome”, “Intermediate Outcome”, and “Long-term Outcome”, each measured by multiple survey items in \(y\).

  • \(\Lambda_x, \Lambda_y\): factor loading matrices

  • \(\delta, \epsilon\): measurement error terms

0.2 Structural model

The structural model defines the causal paths among latent variables, capturing both direct and mediated effects.

\[ \eta = B \eta + \Gamma \xi + \zeta \]

  • \(B\): matrix of path coefficients among endogenous latent variables (\(\eta \rightarrow \eta\))

  • \(\Gamma\): matrix of path coefficients from exogenous to endogenous latent variables (\(\xi \rightarrow \eta\))

  • \(\zeta\): structural disturbance (residual) term

1. Variable definitions

Following the logic-model framework, you aim to test a sequential causal pathway (including direct and indirect effects):

\[ \text{Inputs} \rightarrow \text{Activities} \rightarrow \text{Outputs} \rightarrow \text{Short-term Outcomes} \rightarrow \text{Intermediate Outcomes} \rightarrow \text{Long-term Outcomes}. \]

Exogenous latent variable

\[ \xi_1 = \text{Inputs}. \]

Endogenous latent variables

\[ \begin{aligned} \eta_1 & = \text{Activities}, \\ \eta_2 & = \text{Outputs}, \\ \eta_3 & = \text{Short-term Outcomes}, \\ \eta_4 & = \text{Intermediate Outcomes}, \\ \eta_5 & = \text{Long-term Outcomes}. \end{aligned} \]

2. Measurement model

Let the observed indicators (survey items, administrative metrics, etc.) that measure each latent construct be denoted by \(x\) and \(y\). For example:

\[ \begin{aligned} x_1 & = \Lambda_{x_1}\,\xi_1 + \delta_1, \\ y_i & = \Lambda_{y_i}\,\eta_i + \epsilon_i,\quad i=1,\ldots,5. \end{aligned} \]

  • \(\Lambda_{\cdot}\): factor loading matrices

  • \(\delta_1,\epsilon_i\): measurement errors

The measurement model links each latent construct to its observed indicators. Under your research framework, you can substitute the actual indicator names from the attached model into the notation below.

2.1 Measurement model for the exogenous latent variable: Inputs (\(\xi_1\))

Let the observed variables \(x_{1i}\) \((i=1,\ldots,p)\) be indicators that measure Inputs. Then:

\[ x_{1i} = \lambda_{x_{1i}}\,\xi_1 + \delta_{1i},\quad i=1,\ldots,p. \]

In vector/matrix form:

\[ x_1 = \begin{pmatrix} x_{11}\\ x_{12}\\ \vdots\\ x_{1p} \end{pmatrix} = \underbrace{ \begin{pmatrix} \lambda_{x_{11}}\\ \lambda_{x_{12}}\\ \vdots\\ \lambda_{x_{1p}} \end{pmatrix}}_{\Lambda_{x_1}} \xi_1 + \underbrace{ \begin{pmatrix} \delta_{11}\\ \delta_{12}\\ \vdots\\ \delta_{1p} \end{pmatrix}}_{\delta_1}. \]

2.2 Measurement models for endogenous latent variables (\(\eta_j,\ j=1,\ldots,5\))

For each stage-specific latent variable \(\eta_j\) \((j=1,\ldots,5)\), let the observed indicators be \(y_{ji}\) \((i=1,\ldots,q_j)\). Then:

\[ y_{ji} = \lambda_{y_{ji}}\,\eta_j + \epsilon_{ji},\quad i=1,\ldots,q_j,\quad j=1,\ldots,5. \]

Stage interpretation:

  • \(j=1\): Activities (\(\eta_1\)), indicators \(y_{1i}\) \((i=1,\ldots,q_1)\)

  • \(j=2\): Outputs (\(\eta_2\)), indicators \(y_{2i}\) \((i=1,\ldots,q_2)\)

  • \(j=3\): Short-term Outcomes (\(\eta_3\)), indicators \(y_{3i}\) \((i=1,\ldots,q_3)\)

  • \(j=4\): Intermediate Outcomes (\(\eta_4\)), indicators \(y_{4i}\) \((i=1,\ldots,q_4)\)

  • \(j=5\): Long-term Outcomes (\(\eta_5\)), indicators \(y_{5i}\) \((i=1,\ldots,q_5)\)

In vector/matrix form (for each \(j\)):

\[ y_j = \begin{pmatrix} y_{j1}\\ y_{j2}\\ \vdots\\ y_{jq_j} \end{pmatrix}= \Lambda_{y_j}\,\eta_j + \epsilon_j,\quad j=1,\ldots,5. \]

2.3 Summary of the full measurement model (block-diagonal form)

Stacking all observed indicator vectors into one vector:

\[ \begin{pmatrix} x_1\\ y_1\\ \vdots\\ y_5 \end{pmatrix}= \underbrace{ \begin{pmatrix} \Lambda_{x_1} & 0 & 0 & 0 & 0 & 0\\ 0 & \Lambda_{y_1} & 0 & 0 & 0 & 0\\ \vdots & & \ddots & & & \vdots\\ 0 & 0 & 0 & 0 & 0 & \Lambda_{y_5} \end{pmatrix}}_{\Lambda} \begin{pmatrix} \xi_1\\ \eta_1\\ \vdots\\ \eta_5 \end{pmatrix} + \begin{pmatrix} \delta_1\\ \epsilon_1\\ \vdots\\ \epsilon_5 \end{pmatrix}. \]

  • \(\Lambda\): a block-diagonal loading matrix that places each factor loading matrix in the appropriate block

  • \(\delta_1, \epsilon_j\): measurement error vectors

3. Structural model

3.1 Comprehensive structural equations (sequential + all earlier-to-later direct paths)

Below is a more comprehensive specification that includes:

  • the sequential path (e.g., \(\eta_1 \to \eta_2 \to \cdots \to \eta_5\)), and

  • direct paths from every earlier stage to every later stage, including \(\xi_1\) to all \(\eta_j\).

\[ \begin{aligned} \eta_1 &= \gamma_{11}\xi_1 + \zeta_1,\\ \eta_2 &= \beta_{21}\eta_1 + \gamma_{21}\xi_1 + \zeta_2,\\ \eta_3 &= \beta_{32}\eta_2 + \beta_{31}\eta_1 + \gamma_{31}\xi_1 + \zeta_3,\\ \eta_4 &= \beta_{43}\eta_3 + \beta_{42}\eta_2 + \beta_{41}\eta_1 + \gamma_{41}\xi_1 + \zeta_4,\\ \eta_5 &= \beta_{54}\eta_4 + \beta_{53}\eta_3 + \beta_{52}\eta_2 + \beta_{51}\eta_1 + \gamma_{51}\xi_1 + \zeta_5. \end{aligned} \]

Interpretation examples:

  • \(\gamma_{11}\): Inputs \(\rightarrow\) Activities

  • \(\beta_{21}\): Activities \(\rightarrow\) Outputs

  • \(\beta_{31}\): Activities \(\rightarrow\) Short-term Outcomes (direct shortcut)

  • \(\beta_{42}\): Outputs \(\rightarrow\) Intermediate Outcomes (direct shortcut)

  • \(\gamma_{51}\): Inputs \(\rightarrow\) Long-term Outcomes (direct shortcut)

  • \(\zeta_i\): structural disturbances (residuals)

3.2 Matrix form

Let

\[ \eta = (\eta_1,\eta_2,\eta_3,\eta_4,\eta_5)^\prime,\quad \xi = (\xi_1)^\prime. \]

Then the structural model can be written as:

\[ \eta = B\eta + \Gamma\xi + \zeta, \]

where \(\zeta=(\zeta_1,\ldots,\zeta_5)^\prime\), and (one possible representation)

\[ B= \begin{pmatrix} 0 & 0 & 0 & 0 & 0\\ \beta_{21} & 0 & 0 & 0 & 0\\ \beta_{31} & \beta_{32} & 0 & 0 & 0\\ \beta_{41} & \beta_{42} & \beta_{43} & 0 & 0\\ \beta_{51} & \beta_{52} & \beta_{53} & \beta_{54} & 0 \end{pmatrix}, \quad \Gamma= \begin{pmatrix} \gamma_{11}\\ \gamma_{21}\\ \gamma_{31}\\ \gamma_{41}\\ \gamma_{51} \end{pmatrix}. \]

3.3 Examples of direct, indirect, and total effects

Direct effect (example)

  • Inputs \(\rightarrow\) Long-term Outcomes (if included): \(\gamma_{51}\)

One specific indirect effect (pure sequential chain)

  • Inputs \(\rightarrow\) Activities \(\rightarrow\) Outputs \(\rightarrow\) Short-term \(\rightarrow\) Intermediate \(\rightarrow\) Long-term:

\[ \gamma_{11}\times \beta_{21}\times \beta_{32}\times \beta_{43}\times \beta_{54}. \]

Total effect

  • The total effect equals the direct effect plus the sum of all possible indirect paths implied by the model.

4. Structural model using only the sequential (logic-model) paths

4.1 Overview

Even though a comprehensive structural model can include direct paths from every earlier stage to every later stage, it is also valid to specify a parsimonious model that includes only the sequential causal chain implied by the logic model:

\[ \xi_1 \rightarrow \eta_1 \rightarrow \eta_2 \rightarrow \eta_3 \rightarrow \eta_4 \rightarrow \eta_5. \]

In equations (sequential paths only):

\[ \begin{aligned} \eta_1 \;&=\; \gamma_{11}\xi_1 + \zeta_1, \\ \eta_2 \;&=\; \beta_{21}\eta_1 + \zeta_2, \\ \eta_3 \;&=\; \beta_{32}\eta_2 + \zeta_3, \\ \eta_4 \;&=\; \beta_{43}\eta_3 + \zeta_4, \\ \eta_5 \;&=\; \beta_{54}\eta_4 + \zeta_5. \end{aligned} \]

In matrix form, letting

\[ \eta = (\eta_1,\eta_2,\eta_3,\eta_4,\eta_5)^\prime,\quad \xi = (\xi_1)^\prime, \]

the model can be written as:

\[ \eta = B\eta + \Gamma\xi + \zeta, \]

with

\[ B= \begin{pmatrix} 0 & 0 & 0 & 0 & 0\\ \beta_{21} & 0 & 0 & 0 & 0\\ 0 & \beta_{32} & 0 & 0 & 0\\ 0 & 0 & \beta_{43} & 0 & 0\\ 0 & 0 & 0 & \beta_{54} & 0 \end{pmatrix}, \quad \Gamma= \begin{pmatrix} \gamma_{11}\\ 0\\ 0\\ 0\\ 0 \end{pmatrix}. \]

4.2 Why start with a parsimonious model? (Theoretical parsimony)

In SEM, adding every possible path “just in case” can make the model unnecessarily complex. When the model becomes too complex relative to the sample size, it may lead to:

  • loss of degrees of freedom,

  • unstable estimation, and

  • potential identification problems.

Therefore, a recommended strategy is to first fit a baseline SEM that reflects the core theoretical assumption of the logic model (the sequential stage-to-stage causality), and then consider adding additional paths only when there is strong theoretical and empirical justification.

With sequential paths only, it is easier to interpret mediation because direct and indirect effects are less entangled.

For example, the sequential-only model allows you to test the mediated pathway step by step, such as:

\[ \xi_1 \rightarrow \eta_3 \rightarrow \eta_4 \rightarrow \eta_5, \]

and later examine whether adding an extra direct path (e.g., \(\xi_1 \rightarrow \eta_4\)) is theoretically meaningful and empirically supported.

4.3 A standard stepwise modeling strategy

A common approach is:

  • Model 1 (baseline): test the sequential logic-model paths only.

  • Model 2 (expanded): add one theoretically motivated additional path at a time, and evaluate improvement in fit and statistical significance.

This stepwise strategy helps prevent overfitting and supports a more defensible final model by keeping only the “truly important” paths.

III. Answer

0. Addressing small within-library sample sizes (\(n_k<30\)) in library-level SEM

If you aggregate user surveys into simple library-level means and treat those means as observed indicators, libraries with small \(n_k\) can have large measurement error in those means. This can bias or destabilize the SEM measurement model. However, the issue can be mitigated while keeping those libraries in the analysis by using the following approach.

1. Multilevel SEM using individual responses (no aggregation)

Idea: Do not aggregate survey responses. Instead, model users at Level-1 and libraries at Level-2. This naturally yields partial pooling, which stabilizes estimation even when some libraries have small \(n_k\).

Advantages:

  • Even libraries with small \(n_k\) benefit from information sharing (partial pooling), improving stability.

  • Separates within-library (individual) variation from between-library variation.

  • Possible tools: Mplus (Multilevel SEM), and multilevel / complex-sample approaches in R (e.g., lavaan.survey with survey), depending on design and feasibility.

2. Multilevel SEM formulation for this consultation

Index libraries by \(k=1,\ldots,K\), individuals within library \(k\) by \(i=1,\ldots,n_k\), and (outcome) survey items or indicators by \(j\).

2.1 Level-1 (individual level): outcomes-only measurement

At the individual level, the dataset contains only outcome-related survey responses. Inputs / Activities / Outputs are library-level administrative variables, so they do not appear in Level-1.

A simple random-intercept-only specification is:

\[ \begin{aligned} y_{kij} \;&=\; \pi_{0kj} + \epsilon_{kij}, \\ \epsilon_{kij} \;&\sim\; N(0,\sigma_{\epsilon j}^2). \end{aligned} \]

  • \(y_{kij}\): response of individual \(i\) in library \(k\) to outcome indicator/item \(j\)

  • \(\pi_{0kj}\): library-specific mean (random intercept) for item \(j\)

  • \(\epsilon_{kij}\): within-library individual-level deviation (treated as Level-1 error)

In this setup, small \(n_k\) is not automatically fatal because the multilevel model yields partial pooling at Level-2.

2.2 Level-2 (library level): measurement model including exogenous Inputs

At Level-2, both exogenous latent variables (\(\xi\)) and endogenous latent variables (\(\eta\)) are modeled.

(A) Exogenous latent variable (Inputs) measurement model

Let \(x_{k\ell}\) denote library-level administrative indicators (e.g., budget, staff size, collection size). Then:

\[ x_{k\ell} = \lambda_{x\ell}\,\xi_k + \delta_{k\ell},\quad \delta_{k\ell}\sim N(0,\theta_{x\ell}). \]

  • \(\xi_k\): latent Inputs for library \(k\)

  • \(x_{k\ell}\): administrative indicator \(\ell\) for library \(k\)

  • \(\delta_{k\ell}\): measurement error at the library level

This component is essential because Inputs are library-level and should enter the SEM as a Level-2 latent construct.

(B) Endogenous latent variables (Activities through Outcomes) measurement model

Use the Level-1 random intercepts \(\pi_{0kj}\) (library-level item means) as the measurement layer for library-level latent constructs:

\[ \pi_{0kj} = \tau_j + \lambda_{yj}\,\eta_{kj} + \epsilon_{kj},\quad \epsilon_{kj}\sim N(0,\theta_{yj}). \]

  • \(\eta_{kj}\): library-level latent variable for stage \(j\) (Activities, Outputs, Short-term, Intermediate, Long-term)

  • \(\epsilon_{kj}\): library-level measurement error, distinct from Level-1 individual error \(\epsilon_{kij}\)

2.3 Level-2 structural model (logic-model causal paths)

The logic-model causal chain is modeled at Level-2. A comprehensive version can allow sequential effects plus selected direct paths:

\[ \begin{aligned} \eta_{k1} \;&=\; \gamma_{11}\,\xi_k + \zeta_{k1}, \\ \eta_{k2} \;&=\; \beta_{21}\,\eta_{k1} + \gamma_{21}\,\xi_k + \zeta_{k2}, \\ \eta_{k3} \;&=\; \beta_{32}\,\eta_{k2} + \beta_{31}\,\eta_{k1} + \gamma_{31}\,\xi_k + \zeta_{k3}, \\ \eta_{k4} \;&=\; \beta_{43}\,\eta_{k3} + \beta_{42}\,\eta_{k2} + \gamma_{41}\,\xi_k + \zeta_{k4}, \\ \eta_{k5} \;&=\; \beta_{54}\,\eta_{k4} + \beta_{53}\,\eta_{k3} + \gamma_{51}\,\xi_k + \zeta_{k5}. \end{aligned} \]

  • \(\xi_k\): Inputs (exogenous latent variable)

  • \(\eta_{k1},\ldots,\eta_{k5}\): Activities \(\rightarrow\) Outputs \(\rightarrow\) Outcomes (endogenous latent variables)

  • \(\zeta_{kj}\): library-level structural disturbances (residuals)

3. Why this is a multilevel SEM (and why it helps when \(n_k<30\))

  • Level-1: absorbs individual survey noise and separates within-library variability

  • Level-2: estimates library-level latent constructs and the causal logic-model pathway

  • Inputs measurement: models library administrative indicators as an exogenous latent factor

  • Outcomes measurement: accounts for measurement error in library-level outcome constructs without naïvely treating small-\(n_k\) means as error-free

4. Conclusion (for consultation response)

A multilevel SEM can be applied by modeling individual survey responses at Level-1 and libraries at Level-2. This approach allows stable estimation of library-level latent constructs even when some libraries have fewer than 30 respondents, because the model separates individual-level variability and uses partial pooling at the library level. Moreover, library-level Inputs can be incorporated as an exogenous latent variable via administrative indicators, enabling a theoretically consistent test of the logic-model causal pathway without dropping a large portion of libraries.

]]>Han Jun KoConsultation 52025-03-01T23:40:00+00:002025-03-01T23:40:00+00:00https://jason1ko.github.io/data%20science%20institute/DSI-c5

Consultation 5: Complex Survey Methods for Age-Group Comparisons in KNHANES

I. Client’s Inquiry

Study Objective

  • Thesis title: Nutritional Recommendations for Muscle Health in the Ageing Population of South Korea: Exploring Dietary Intake and Patterns, Activity Levels, and Socio-cultural Influences on Sarcopenia Risk.
  • Hypothesis: Intake of specific nutrients/foods in older adults positively affects arm muscle mass (Lean Body Mass) and handgrip strength (HGS), which may in turn improve quality of life (QoL), particularly in relation to physical activity.

Current Issue

  • I used:
    • Complex samples chi-square tests, and
    • Post hoc logistic regression with Bonferroni adjustment to compare categorical variables across age groups.
  • I am unsure whether this approach is appropriate and would like professional guidance on whether the methods are correct (and, if not, what alternatives are recommended).

Data Description

  • Data source: Korea National Health and Nutrition Examination Survey (KNHANES), 2015–2023.
  • Exclusion: 2021–2022 were excluded due to missing HGS measurements.
  • Variables included: socioeconomic status (SES), anthropometric/body composition (InBody) data, nutrient intake, QoL, and HGS.
  • Sample size: 30,354 participants
    • Men: 13,149
    • Women: 17,205
  • Age groups (5 categories):
    • Young Adults: 19–29 (n = 3,398)
    • Middle-aged Adults: 30–49 (n = 9,422)
    • Older Adults: 50–64 (n = 8,174)
    • Elderly: 65–74 (n = 6,132)
    • Older Elderly: ≥ 75 (n = 3,228)
  • Key derived/classified measures:
    • Physical activity levels and HGS classified using:
      • quintiles, and
      • Asian Working Group for Sarcopenia (AWGS) criteria (low vs high strength)
    • Estimated energy requirements calculated
    • Macronutrient intake computed as:
      • absolute intake, and
      • percentage contribution to total caloric intake

II. Background

1. Categorical variables

When the client says:

“to get the difference between age groups for different categorical variables,”

the items in the Data Description that most directly connect to “categorical variables” (especially for comparing distributions across the 5 age groups) are the following statements:

  • “age was categorized into five groups …”
  • “Physical activity levels and HGS were classified using both quintiles and the AWGS criteria …”
  • “The dataset includes information on socioeconomic status (SES) … quality of life (QoL) …”

From these, the most natural interpretation is that “different categorical variables” refers primarily to the variables that were explicitly classified into categories, and secondarily to other variables that are commonly categorical in KNHANES.

1) Physical activity (PA) level — explicitly “classified”

Likely categorical forms include:

  • Quintiles (5 categories) of physical activity level, and/or
  • A binary grouping (e.g., low vs high) if the quintiles were collapsed

This is a very typical setup for using a chi-square test to check whether the distribution of PA categories differs by age group.

2) Handgrip strength (HGS) — explicitly categorized

Likely categorical forms include:

  • AWGS criteria: low vs high strength (binary), and/or
  • Quintiles (5 categories) if HGS was also divided into quintiles

Again, comparing low/high proportions (or quintile distributions) across age groups via chi-square is common.

3) Socioeconomic status (SES) variables — a categorical “bundle”

The client only wrote “SES,” so specific variable names are not given. However, in KNHANES, SES-related variables are often categorical, such as:

  • income quantile group
  • education level
  • occupation class
  • marital status

It is very common to test whether these SES category distributions differ across age groups.

4) Quality of life (QoL) — likely categorical depending on how it was used

QoL can be continuous (e.g., an index score), but survey-based QoL measures (e.g., EQ-5D-type items) often include categorical responses (e.g., no problems / some problems / severe problems). Since the client explicitly referred to “categorical variables,” it is plausible that QoL was used in a categorical form (individual items or categorized scores).

Summary

In the client’s wording, “different categorical variables” most plausibly refers to:

  1. Physical activity classification variables (quintiles or low/high), and
  2. HGS classification variables (AWGS low/high or quintiles),

as the primary targets, and additionally:

  1. SES variables (commonly categorical in KNHANES), and
  2. QoL variables if represented as categorical items or categorized scores.

2. AWGS criteria (Asian Working Group for Sarcopenia)

AWGS (Asian Working Group for Sarcopenia) is a consensus guideline used to classify/diagnose sarcopenia in Asian populations (especially older adults).
Its core idea is to take continuous measurements (e.g., handgrip strength, gait speed, muscle mass) and apply clinical cut-offs to create categories such as low vs normal.

Typical AWGS 2019 cut-offs (examples)

  • Low muscle strength (Handgrip strength, HGS)
    • Men: < 28 kg
    • Women: < 18 kg
  • Low physical performance
    • 6m gait speed < 1.0 m/s, or
    • SPPB ≤ 9, or
    • 5-times chair stand ≥ 12 s
  • Low muscle mass also has DXA/BIA-based cut-offs (used for full diagnosis).

The sentence:

“Physical activity levels and HGS were classified using both quintiles and the AWGS criteria …”

is most naturally read as:

The same variables were categorized in two different ways.

(1) Quintiles (Q1–Q5): distribution-based categorization

  • Split the sample into five equal-sized groups (20% each) based on the variable’s distribution.
  • Example: HGS quintiles, or physical activity (e.g., MET-min/week) quintiles.

Pros: easy “relative low vs high” comparisons within the sample
Cons: weaker direct clinical meaning (“at risk” threshold)

(2) AWGS criteria: guideline cut-off categorization

  • Use clinical cut-offs to create categories like low vs normal.
  • Example: HGS low/high using AWGS thresholds (men < 28 kg, women < 18 kg).

Pros: clinically interpretable (“below risk threshold or not”)
Cons: only possible if the needed measures exist (or if proxies are used)

3. Complex Samples Chi-square Test

A complex samples chi-square test is a design-based test of association (independence) between two categorical variables when the data come from a complex survey design (e.g., stratification, clustering/PSUs, and sampling weights), not from simple random sampling (SRS).

Core ideas (why it differs from the usual Pearson chi-square)

  • In complex surveys, observations are not i.i.d. due to weights and within-cluster correlation.
    → Using the usual Pearson chi-square test can produce incorrect p-values.

  • The solution is:

    • build a weighted contingency table, but

    • compute inference accounting for the survey design, typically via a Rao–Scott adjusted chi-square (often with an F transformation).

1) Starting point: weighted contingency table and Pearson chi-square

Let:

  • \(A \in \{1,\dots,R\}\), \(B \in \{1,\dots,C\}\) be categorical variables,

  • \(w_i\) be the survey weight for individual \(i\).

Weighted cell counts

\[ \widehat{N}_{rc} =\sum_{i=1}^{n} w_i \, \mathbf{1}(A_i=r, B_i=c). \]

Marginal totals: \[ \widehat{N}_{r+}=\sum_{c=1}^{C}\widehat{N}_{rc},\quad \widehat{N}_{+c}=\sum_{r=1}^{R}\widehat{N}_{rc},\quad \widehat{N}_{++}=\sum_{r=1}^{R}\sum_{c=1}^{C}\widehat{N}_{rc}. \]

Under the independence null hypothesis \(H_0: A \perp B\), the expected weighted counts are:

\[ \widehat{E}_{rc} =\frac{\widehat{N}_{r+}\widehat{N}_{+c}}{\widehat{N}_{++}}. \]

The (weighted) Pearson chi-square statistic is:

\[ X_P^2 =\sum_{r=1}^{R}\sum_{c=1}^{C} \frac{\left(\widehat{N}_{rc}-\widehat{E}_{rc}\right)^2}{\widehat{E}_{rc}}. \]

Why \(X_P^2\) is not enough in complex surveys

In complex samples, \(X_P^2\) does not follow a standard chi-square distribution well because:

  • clustering induces correlation (reducing the effective sample size),

  • unequal weights increase variance.

2) Key adjustment: Rao–Scott correction (design effect)

To get valid inference, we estimate the covariance of estimated proportions (or cell proportions) reflecting stratification/clustering/weights, e.g. using:

  • Taylor linearization, or

  • replicate weights (jackknife, BRR, bootstrap).

Rao–Scott adjustment modifies the Pearson statistic to reflect the design effect (variance inflation due to the survey design). A common first-order intuition is:

\[ X_{RS}^2 =\frac{X_P^2}{\widehat{c}}, \] where \(\widehat{c}\) acts like an “average design effect / inflation factor,” shrinking \(X_P^2\) because the true variance is larger under complex sampling.

Many software packages (e.g., SPSS Complex Samples, R survey) often convert the adjusted statistic into an F statistic for p-values:

  • Numerator df: \((R-1)(C-1)\)

  • Denominator df: approximately “#PSUs − #strata” (design-based degrees of freedom; exact details depend on implementation)

3) One-line summary

A complex samples chi-square test is an independence test on a weighted contingency table, where the p-value is computed using a design-based adjustment (typically Rao–Scott adjusted chi-square or its F-equivalent) to account for stratification, clustering, and weights.

4. Bonferroni test in logistic regression

In this context, “post hoc logistic regression Bonferroni test” does not mean that logistic regression has a special standalone test called “the Bonferroni test.”
It means:

  • the researcher ran multiple post hoc comparisons using logistic regression, and
  • they adjusted the resulting p-values (or significance level / confidence intervals) using the Bonferroni correction to control inflation of Type I error due to multiple testing.

1) Why it is needed: the multiple-comparisons problem

Suppose there are 5 age groups. If we want to find which groups differ, we typically run several comparisons:

  • Reference-group comparisons: choose one reference group and compare it to the other 4
    • number of tests: \(m=4\)
  • All pairwise comparisons: compare every pair of groups
    • number of tests: \(m=\binom{5}{2}=10\)

If we test each comparison at level \(\alpha\) without adjustment, the probability of getting at least one false positive (family-wise error rate, FWER) increases as \(m\) grows.

2) Bonferroni correction: the key idea

Bonferroni controls FWER at \(\alpha\) by making each individual test more stringent.

(a) Adjust the significance level

Use: \[ \alpha^* = \frac{\alpha}{m}. \]

(b) Equivalent p-value adjustment

If the original p-value for comparison \(j\) is \(p_j\), the Bonferroni-adjusted p-value is: \[ p_j^{(\mathrm{Bonf})} = \min\{m\,p_j,\,1\}. \]

Declare significance if: \[ p_j^{(\mathrm{Bonf})} \le \alpha. \]

3) What is being tested multiple times in logistic regression?

In logistic regression, post hoc comparisons are typically tests of:

  • regression coefficients, or
  • linear contrasts (differences between coefficients), which correspond to group comparisons.

Example: let \(Y_i\) indicate whether a person has low HGS
(e.g., \(Y_i=1\) for “low”, \(Y_i=0\) for “normal”).
With age group \(G_i \in \{1,\dots,5\}\) and group 1 as reference:

\[ \mathrm{logit}\{\Pr(Y_i=1)\} =\beta_0 + \beta_2\,\mathbf{1}(G_i=2)+\cdots+\beta_5\,\mathbf{1}(G_i=5). \]

Then each \(\beta_k\) represents the log-odds difference between group \(k\) and the reference group 1, and the odds ratio is: \[ \mathrm{OR}_{k:1} = \exp(\beta_k). \]

Typical post hoc logistic + Bonferroni workflows are:

  • Reference vs others: test \(H_{0k}:\beta_k=0\) for \(k=2,\dots,5\)
    • \(m=4\) tests
  • All pairwise comparisons: test contrasts such as \[ H_{0(a,b)}:\beta_a - \beta_b = 0 \] for all pairs \((a,b)\)
    • \(m=10\) tests

Bonferroni adjustment is then applied to these multiple p-values (or contrasts).

4) Bonferroni-adjusted confidence intervals (optional but common)

Bonferroni can also adjust confidence intervals.

Instead of using \(z_{1-\alpha/2}\) for each interval, to achieve simultaneous coverage across \(m\) comparisons we use: \[ \widehat{\beta}_j \pm z_{1-\alpha/(2m)}\,\mathrm{SE}(\widehat{\beta}_j). \]

For odds ratios, exponentiate the endpoints: \[ \exp\Bigl(\widehat{\beta}_j \pm z_{1-\alpha/(2m)}\,\mathrm{SE}(\widehat{\beta}_j)\Bigr). \]

One-sentence summary

“post hoc logistic regression Bonferroni test” means the researcher performed multiple age-group comparisons using logistic regression (via coefficients or contrasts) and controlled the inflated false-positive risk by applying the Bonferroni correction (either \(\alpha/m\) or \(m p_j\) adjustments).

III. Answer

1) Bottom line first: their procedure can be valid, conditionally

What the client did:

  1. Run a complex-samples chi-square test (Rao–Scott family) to test the overall association between age group (5 levels) and a categorical outcome \(Y\).

  2. If significant, run post hoc comparisons using logistic-regression-based methods (to see which age groups differ).

  3. Apply Bonferroni (or Holm, etc.) to adjust for multiple comparisons.

Conceptually, this is a reasonable workflow.

However, for it to be truly correct, both of the following must hold:

  • The post hoc logistic regression must also account for the complex survey design (weights, strata, clusters/PSUs).
    Otherwise, step (1) is design-based but step (2) ignores the design → inconsistent SEs / p-values.

  • The regression model must match the number of categories / ordinality of \(Y\).

    • If \(Y\) is binary → logistic regression
    • If \(Y\) has \(3+\) categories → multinomial or ordinal logistic regression (as appropriate)

2) Step 1: What does the complex-samples chi-square test actually test?

Let age group be \(G \in \{1,\dots,5\}\) and a categorical variable be \(Y \in \{1,\dots,C\}\).

Null hypothesis (independence):

\[ H_0:\Pr(Y=c \mid G=g)=\Pr(Y=c)\quad \forall g,c. \]

With survey weights \(w_i\), form the weighted contingency table:

\[ \widehat{N}_{gc}=\sum_{i=1}^{n} w_i\,\mathbf{1}(G_i=g,\;Y_i=c). \]

Compute expected counts under independence:

\[ \widehat{E}_{gc}=\frac{\widehat{N}_{g+}\widehat{N}_{+c}}{\widehat{N}_{++}}. \]

Instead of treating the usual Pearson statistic as chi-square, complex-sample software uses a Rao–Scott adjusted chi-square (or an F-transformation) so that:

  • the variance / standard errors reflect stratification + clustering + weighting, and
  • the reference distribution / degrees of freedom are design-based.

3) Step 2 (key): What is the “proper” way to do post hoc logistic regression + Bonferroni?

In practice, the cleanest and easiest-to-defend approach is often to use one survey-weighted regression framework that handles:

  • a global test (overall age-group effect), and then
  • post hoc contrasts (pairwise comparisons),

all within the same design-based model (rather than “chi-square first, then regression”).

3.1 If \(Y\) is binary (e.g., low HGS vs normal): survey-weighted logistic regression

Model:

\[ \mathrm{logit}\{\Pr(Y_i=1 \mid G_i)\} =\beta_0+\sum_{g=2}^{5}\beta_g\,\mathbf{1}(G_i=g). \]

If group 1 is the reference, then \(\exp(\beta_g)\) is the odds ratio for “group \(g\) vs group 1”.

(A) Global test (overall age effect)

\[ H_0:\beta_2=\beta_3=\beta_4=\beta_5=0. \]

A typical design-based Wald form is:

\[ W=\widehat{\beta}^{\top}\Bigl(\widehat{\mathrm{Var}}(\widehat{\beta})\Bigr)^{-1}\widehat{\beta}, \]

where \(\widehat{\mathrm{Var}}(\widehat{\beta})\) is the survey-design-based (sandwich) variance.

(B) Post hoc comparisons (pairwise contrasts)

To compare age groups \(a\) and \(b\), test a contrast such as:

\[ H_0:\beta_a-\beta_b=0. \]

You obtain p-values \(p_j\) for \(j=1,\dots,m\) comparisons, then adjust for multiplicity.

Bonferroni-adjusted p-values:

\[ p_j^{(\mathrm{Bonf})}=\min\{m\,p_j,\,1\}. \]

Equivalently, adjust the per-test significance level:

\[ \alpha^{\*}=\frac{\alpha}{m}. \]

3.2 If \(Y\) has \(3+\) categories: multinomial or ordinal survey regression

Running separate binary logistic models for each category is often messy and can complicate error control.

  • If categories are nominal (no order): multinomial logistic (with baseline category \(c=1\))

\[ \log\frac{\Pr(Y_i=c \mid G_i)}{\Pr(Y_i=1 \mid G_i)} =\alpha_c+\sum_{g=2}^{5}\beta_{cg}\,\mathbf{1}(G_i=g),\quad c=2,\dots,C. \]

  • If categories are ordered: consider an ordinal (cumulative logit) model.

Global tests are still “all age effects are zero” (a multivariate Wald test), and post hoc comparisons are still contrast tests of the form \(L\beta=0\), followed by Bonferroni/Holm adjustments.

4) Conclusion

  • “Your overall workflow—testing the age-group association first using a complex-samples chi-square test and then conducting post hoc comparisons—is conceptually acceptable.”

  • “However, the post hoc logistic regression must also reflect the complex survey design (weights, strata, clusters). If step 1 is design-based but step 2 ignores the design, standard errors and p-values may not be consistent.”

  • “Also, logistic regression is appropriate only when the outcome is binary. If the outcome has three or more categories, multinomial or ordinal logistic regression is typically the standard approach.”

  • “Bonferroni adjustment is a valid (but conservative) way to control family-wise error. If the number of comparisons is large, Holm or FDR-based options can be considered depending on the study goal.”

]]>Han Jun KoIntroduction2025-01-01T23:59:59+00:002025-01-01T23:59:59+00:00https://jason1ko.github.io/industrial%20academic%20cooperation%20big%20data%20analysis/Intro-to-IAC

Industrial Academic Cooperation Big Data Analysis

Introduction

“Industrial Academic Cooperation Big Data Analysis” is a course series offered by the Department of Statistics and Data Science at Yonsei University in collaboration with external institutions and companies. It is designed to give students experience applying what they learn in class to real-world data from fields such as engineering, marketing, and finance, and to derive statistical insights and practical, data-driven solutions.

In the first course (STA6810), we focused on reviewing fundamental theory, attending invited talks from various companies, and investigating what kinds of statistical perspectives are currently needed in industry. In the second course (STA6850), in addition to industry talks, each team selected a specific domain and carried out a Kaggle project that closely reflected the types of tasks required in that field. In the third course (STA7510), a real marketing company gave a lecture, shared its in-house data, and requested an analysis; students then analyzed the data and proposed ideas back to the company.

In this blog, I summarize two of the projects in which we conducted full analyses and illustrate the step-by-step process of how we approached and solved the problems.Right now, this page serves as the Introduction under the Industrial Academic Cooperation Big Data Analysis category on the left sidebar. If you would like to explore further, please look for the project-specific posts titled Finance Data Analysis and Marketing Data Analysis under the same category.

]]>Han Jun KoFinance Data Analysis2025-01-01T23:59:00+00:002025-01-01T23:59:00+00:00https://jason1ko.github.io/industrial%20academic%20cooperation%20big%20data%20analysis/IAC-finance
]]>Han Jun KoMarketing Data Analysis2025-01-01T23:50:00+00:002025-01-01T23:50:00+00:00https://jason1ko.github.io/industrial%20academic%20cooperation%20big%20data%20analysis/IAC-market
]]>Han Jun KoMCMC HW12024-09-01T23:59:00+00:002024-09-01T23:59:00+00:00https://jason1ko.github.io/monte%20carlo%20methods/MCMC-HW1

1. Normal from Laplace distribution

\[\begin{aligned} f(x) &= \frac{1}{\sqrt{2\pi}}\exp\Bigl\{-\frac{x^2}{2}\Bigr\} \\ q(x) &= \frac{\beta}{2}\exp\Bigl\{-\beta|x|\Bigr\} \end{aligned}\]

(a) Calculate the value of \(M\) when \(\beta = 1\)

Recall that \(M = \sup_{x}\frac{f(x)}{q(x)}\). Both the target function \(f\) and proposal density \(q\) are symmetric at \(x=0\), so we can only consider the case \(x \ge 0\) for achieving the supremum of \(f(x)/q(x)\) for all \(x\in\mathbb R\).

\[\begin{aligned} f(x)/q(x) &= (1/\sqrt{2\pi})/(\beta/2)\cdot\exp\Bigl\{-\frac{x^2}{2}+\beta x \Bigr\} \\ &= \frac{2}{\sqrt{2\pi}\beta}\exp\Bigl\{-\frac{1}{2}(x-\beta)^2 + \frac{\beta^2}{2} \Bigr\} \\ &\le \frac{2}{\sqrt{2\pi}\beta}\exp\Bigl\{ \frac{\beta^2}{2} \Bigr\}, \quad\text{when } x = \beta \end{aligned}\]

So, \(\sup_{x\ge0}\frac{f(x)}{q(x)} = \sqrt{\frac{2}{\pi}}\exp\{1/2\}\) when \(\beta = 1\), and thus \(M = \sqrt{\frac{2}{\pi}}\exp\{1/2\}\). This constant works as a constant that is multiplied by the proposal density \(q\) so that the envelope function \(e(x) := M\cdot q(x)\) is always greater than target density \(f\) and there exist some \(x\in\mathbb R\) such that \(e(x) = f(x)\).

(b) Sampling from Standard normal distribution using rejection sampling.

We import the “VGAM” package to sample from Laplace distribution.

rm(list = ls())
set.seed(2023311161)

#install.packages("VGAM") 
library(VGAM) # rlaplace
## Warning: 패키지 'VGAM'는 R 버전 4.3.3에서 작성되었습니다
## 필요한 패키지를 로딩중입니다: stats4
## 필요한 패키지를 로딩중입니다: splines
library(ggplot2)
## Warning: 패키지 'ggplot2'는 R 버전 4.3.3에서 작성되었습니다

As obtained from (a), we set \(M = \sqrt{\frac{2}{\pi}}\exp\{1/2\}\). Also, we will generate 100,000 samplings from rejection sampling, which will be saved in vector X.

M <- sqrt(2/pi)*exp(1/2)

num <- 100000;i<-1
X <- c()

The rejection algorithm for this problem is as follows:

  1. Sample from proposal density; \(Y \sim \text{Laplace}(0,1)\).
  2. Sample from Uniform distribution; \(U \sim \text{Unif}(0,1)\).
  3. If \(U \le f(Y)/q(Y)\), keep the value \(Y\) and set \(X=Y\).
  4. otherwise, reject \(Y\) and return to step 1 until 100,000 number of X is generated.
while(i <= num){
  y <- rlaplace(1)
  u <- runif(1)
  
  if(u < dnorm(y)/(M*dlaplace(y))){
    X[i] <- y
    i <- i+1
  }

}

The following histogram shows how the result of rejection sampling (blue) are similar to target density (red), the standard normal distribution.

data_X <- as.data.frame(X)

ggplot(data_X, aes(x=X)) + 
  geom_histogram(aes(y = after_stat(density)), binwidth = 0.1, fill = 'cyan', color = 'black') +
  stat_function(fun = dnorm, args = list(mean = 0, sd = 1), color = 'red', linewidth = 1) + 
  labs(title = "Histogram of X using rejection sampling")

2. Importance Sampling

(a) For a standard normal random variable \(Z\), calculate \(\mathbb P(Z > 2.5)\) using Monte Carlo sums based on indicator functions. How many simulated random variables are needed to obtain three digits of accuracy?

The given probability can be expressed as an expectation of an indicator function: \[\mathbb P(Z>2.5) = \mathbb E[\mathbb{1}(Z > 2.5)].\] The value of the probability is around 0.00621, So the purpose is to check whether the difference between the Monte Carlo result and the real value is less than 0.000005.

rm(list = ls())
set.seed(2023311161)

1-pnorm(2.5) # 0.006209665, 0.00621
## [1] 0.006209665
ind <- c()
i <- 1

The ‘ind’ vector contains the value 0 and 1, indicating whether the ‘z’ is greater than 2.5. We continue these steps until the difference between the average of all the ‘ind’ vector elements and 0.00621 are less than 0.000005.

while(1){
  z <- rnorm(1)
  ind[i] <- as.numeric(z>2.5)
  
  if(abs(mean(ind)-(0.00621)) < 0.000005){break}
  i <- i+1
}

The Monte Carlo estimates are 0.00621118, and the number of interation is 322.

mean(ind) # 0.00621118
## [1] 0.00621118
length(ind) # 322
## [1] 322

(b) Compare the number of variables needed to obtain three digits of accuracy with importance sampling to the answers obtained from above.

Given the target and proposal density \(f\) and \(g\), the Importance Sampling algorithm for estimating \(\mathbb E_{X\sim f}[h(X)]\) is as follows:

  1. Sample \(X_i\) from \(g\)
  2. Calculate the weight \(w(X_i) = f(X_i)/g(X_i)\).
  3. Repeat 1 and 2 for desired number of iteration (\(n\)), and take an average of all \(h(X)w(X)\). That is, \[\hat{\mathbb E_f[h(X)]} = \frac{1}{n}\sum_{i=1}^nh(X_i)w(X_i).\]

The pdf value of target density \(f\) at \(x=2.5\) is extremely low; about 0.0175. So. we need a proposal density \(g\) so that the tail probability of \(g\) is greater than \(f\). If the proposal density is \(\text{Exp}(1)\) truncated at \(x=2.5\), \(g(x) = \exp\{-(x-2.5)\}, x>2.5\) and \[\begin{aligned} \frac{f(x)}{g(x)} &= \frac{1}{\sqrt{2\pi}}\exp\Bigl\{-\frac{1}{2}x^2 + x - 2.5 \Bigr\} \\ &= \frac{1}{\sqrt{2\pi}}\exp\Bigl\{-\frac{1}{2}(x-1)^2-2\Bigr\} \\ &\le \frac{1}{\sqrt{2\pi}}\exp\{-2\} \\ &\approx 0.05399,\qquad \forall x>2.5 \end{aligned}\] This shows \(g\) satisfies the condition of the proposal density of importance sampling.

set.seed(2023311161)

dnorm(2.5) # 0.0175283
## [1] 0.0175283
1/sqrt(2*pi)*exp(-2) # 0.05399097
## [1] 0.05399097

The \(h(X)\) for this problem is the indicator function \(\mathbb{1}(X > 2.5)\). So, the importance sampling algorithm is written as follows:

ind_is <- c()
i <- 1

while(1){
  y <- rexp(1)+2.5 # truncated at 2.5
  ind_is[i] <- dnorm(y)/dexp(y-2.5)*as.numeric(y > 2.5)
  
  if(abs(mean(ind_is) - 0.00621) < 0.000005){break}
  i <- i+1
}

The estimated value of the probability using importance sampling is 0.006210526 and the number of iteration is 206.

mean(ind_is) # 0.006210526
## [1] 0.006210526
length(ind_is) # 206
## [1] 206

3. Compute \(\int_0^1 \frac{e^{-x}}{1+x^2}dx\) using control variate method. (hint. \(c(y) = \frac{e^{-0.5}}{1+y^2}\))

The integral \(\int_0^1 \frac{e^{-x}}{1+x^2}dx\) can be viewed as an expectation as follows: \[\int_0^1 \frac{e^{-x}}{1+x^2}\cdot1dx = \mathbb E_{X}\biggl[\frac{e^{-X}}{1+X^2}\biggr],\] where \(X \sim \text{Unif}(0,1)\). The theoretical value of the integral is 0.5247971.

rm(list = ls())
set.seed(2023311161)

n <- 100000

h <- function(x) {
  exp(-x)/(1+x^2)
}

integrate(h, 0, 1)$value # 0.5247971
## [1] 0.5247971

Instead of simple Monte Carlo method, we may use another random variable \(c(X) = \frac{e^{-0.5}}{1+X^2}\) so that \(\theta = \int_0^1 \frac{e^{-0.5}}{1+x^2}dx \approx 0.476\).

c_y <- function(y) {
  exp(-0.5)/(1+y^2)
}

theta <- integrate(c_y, 0, 1)$value # 0.4763681

Instead of using distinct random variable \(Y\) as written in the lecture note (page 21), we will use the same \(X\) to generate \(c(X)\) and corresponding \(\hat\theta_{MC}\). Then, the control variate estimator is \[\hat\mu_{CV} = \hat\mu_{MC} + \lambda(\hat\theta_{MC} - \theta),\] where \(\hat\mu_{MC}\) and \(\hat\theta_{MC}\) are Monte Carlo estimators of \(\int_0^1 \frac{e^{-x}}{1+x^2}dx\) and \(\int_0^1 \frac{e^{-0.5}}{1+x^2}dx\) respectively, and \[\lambda = -\frac{\text{Cov}[\hat\mu_{MC}, \hat\theta_{MC}]}{\text{Var}[\hat\theta_{MC} ] }.\] We may estimate the covariance and variance using vectors of \(h(x) := \frac{e^{-x}}{1+x^2}\) and \(c(x)\) respectively.

xs <- runif(n)
hs <- h(xs)
c_ys <- c_y(xs)

cov_h_c <- cov(hs, c_ys) # 0.02337966
lambda <- -cov_h_c/var(c_ys)

mu_mc <- mean(hs)
theta_mc <- mean(c_ys)

mu_cv <- mu_mc + lambda*(theta_mc - theta)

Thus, the estimated integral using control variate is 0.5251. For reference, the estimates using Monte Carlo is 0.5260.

mu_cv # 0.5251436
## [1] 0.5251436
mu_mc # 0.5260371
## [1] 0.5260371
]]>Han Jun Ko