Consultation 4: Multilevel SEM for Logic Model Evaluation with Small Cluster Sizes

0.1 Measurement model

The measurement model specifies how observed indicators (measured variables) represent latent variables (unobserved constructs).

\[ \begin{aligned} x & = \Lambda_x \xi + \delta \quad \text{(measurement model for exogenous latent variables)} \\ y & = \Lambda_y \eta + \epsilon \quad \text{(measurement model for endogenous latent variables)} \end{aligned} \]

• \(x\): vector of observed indicators for exogenous latent variables

  • In this study, \(x\) can correspond to the library-level measured variables that are treated as exogenous inputs to the causal chain.

  • Inputs / Activities / Outputs (e.g., budget, staffing, number of programs, number of loans, number of events, etc., depending on the dataset).

• \(y\): vector of observed indicators for endogenous latent variables

  • In this study, \(y\) can correspond to user-survey items (Likert-scale indicators) that measure outcome-related constructs.

  • Short-term / Intermediate / Long-term outcomes (e.g., satisfaction items, perceived usefulness, learning/skill improvement items, intention to revisit, community engagement, etc.).

• \(\xi\): vector of exogenous latent variables (latent predictors)

  • Example interpretation: a latent construct such as “Library Resources/Capacity” or “Service Provision Intensity”, measured by library-level indicators in \(x\).

• \(\eta\): vector of endogenous latent variables (latent outcomes/mediators)

  • Example interpretation: latent constructs such as “Short-term Outcome”, “Intermediate Outcome”, and “Long-term Outcome”, each measured by multiple survey items in \(y\).

• \(\Lambda_x, \Lambda_y\): factor loading matrices

• \(\delta, \epsilon\): measurement error terms

0.2 Structural model

The structural model defines the causal paths among latent variables, capturing both direct and mediated effects.

\[ \eta = B \eta + \Gamma \xi + \zeta \]

• \(B\): matrix of path coefficients among endogenous latent variables (\(\eta \rightarrow \eta\))

• \(\Gamma\): matrix of path coefficients from exogenous to endogenous latent variables (\(\xi \rightarrow \eta\))

• \(\zeta\): structural disturbance (residual) term

2.1 Measurement model for the exogenous latent variable: Inputs (\(\xi_1\))

Let the observed variables \(x_{1i}\) \((i=1,\ldots,p)\) be indicators that measure Inputs. Then:

\[ x_{1i} = \lambda_{x_{1i}}\,\xi_1 + \delta_{1i},\quad i=1,\ldots,p. \]

In vector/matrix form:

\[ x_1 = \begin{pmatrix} x_{11}\\ x_{12}\\ \vdots\\ x_{1p} \end{pmatrix} = \underbrace{ \begin{pmatrix} \lambda_{x_{11}}\\ \lambda_{x_{12}}\\ \vdots\\ \lambda_{x_{1p}} \end{pmatrix}}_{\Lambda_{x_1}} \xi_1 + \underbrace{ \begin{pmatrix} \delta_{11}\\ \delta_{12}\\ \vdots\\ \delta_{1p} \end{pmatrix}}_{\delta_1}. \]

2.2 Measurement models for endogenous latent variables (\(\eta_j,\ j=1,\ldots,5\))

For each stage-specific latent variable \(\eta_j\) \((j=1,\ldots,5)\), let the observed indicators be \(y_{ji}\) \((i=1,\ldots,q_j)\). Then:

\[ y_{ji} = \lambda_{y_{ji}}\,\eta_j + \epsilon_{ji},\quad i=1,\ldots,q_j,\quad j=1,\ldots,5. \]

Stage interpretation:

• \(j=1\): Activities (\(\eta_1\)), indicators \(y_{1i}\) \((i=1,\ldots,q_1)\)

• \(j=2\): Outputs (\(\eta_2\)), indicators \(y_{2i}\) \((i=1,\ldots,q_2)\)

• \(j=3\): Short-term Outcomes (\(\eta_3\)), indicators \(y_{3i}\) \((i=1,\ldots,q_3)\)

• \(j=4\): Intermediate Outcomes (\(\eta_4\)), indicators \(y_{4i}\) \((i=1,\ldots,q_4)\)

• \(j=5\): Long-term Outcomes (\(\eta_5\)), indicators \(y_{5i}\) \((i=1,\ldots,q_5)\)

In vector/matrix form (for each \(j\)):

\[ y_j = \begin{pmatrix} y_{j1}\\ y_{j2}\\ \vdots\\ y_{jq_j} \end{pmatrix}= \Lambda_{y_j}\,\eta_j + \epsilon_j,\quad j=1,\ldots,5. \]

2.3 Summary of the full measurement model (block-diagonal form)

Stacking all observed indicator vectors into one vector:

\[ \begin{pmatrix} x_1\\ y_1\\ \vdots\\ y_5 \end{pmatrix}= \underbrace{ \begin{pmatrix} \Lambda_{x_1} & 0 & 0 & 0 & 0 & 0\\ 0 & \Lambda_{y_1} & 0 & 0 & 0 & 0\\ \vdots & & \ddots & & & \vdots\\ 0 & 0 & 0 & 0 & 0 & \Lambda_{y_5} \end{pmatrix}}_{\Lambda} \begin{pmatrix} \xi_1\\ \eta_1\\ \vdots\\ \eta_5 \end{pmatrix} + \begin{pmatrix} \delta_1\\ \epsilon_1\\ \vdots\\ \epsilon_5 \end{pmatrix}. \]

• \(\Lambda\): a block-diagonal loading matrix that places each factor loading matrix in the appropriate block

• \(\delta_1, \epsilon_j\): measurement error vectors

3.1 Comprehensive structural equations (sequential + all earlier-to-later direct paths)

Below is a more comprehensive specification that includes:

• the sequential path (e.g., \(\eta_1 \to \eta_2 \to \cdots \to \eta_5\)), and

• direct paths from every earlier stage to every later stage, including \(\xi_1\) to all \(\eta_j\).

\[ \begin{aligned} \eta_1 &= \gamma_{11}\xi_1 + \zeta_1,\\ \eta_2 &= \beta_{21}\eta_1 + \gamma_{21}\xi_1 + \zeta_2,\\ \eta_3 &= \beta_{32}\eta_2 + \beta_{31}\eta_1 + \gamma_{31}\xi_1 + \zeta_3,\\ \eta_4 &= \beta_{43}\eta_3 + \beta_{42}\eta_2 + \beta_{41}\eta_1 + \gamma_{41}\xi_1 + \zeta_4,\\ \eta_5 &= \beta_{54}\eta_4 + \beta_{53}\eta_3 + \beta_{52}\eta_2 + \beta_{51}\eta_1 + \gamma_{51}\xi_1 + \zeta_5. \end{aligned} \]

Interpretation examples:

• \(\gamma_{11}\): Inputs \(\rightarrow\) Activities

• \(\beta_{21}\): Activities \(\rightarrow\) Outputs

• \(\beta_{31}\): Activities \(\rightarrow\) Short-term Outcomes (direct shortcut)

• \(\beta_{42}\): Outputs \(\rightarrow\) Intermediate Outcomes (direct shortcut)

• \(\gamma_{51}\): Inputs \(\rightarrow\) Long-term Outcomes (direct shortcut)

• \(\zeta_i\): structural disturbances (residuals)

3.2 Matrix form

Let

\[ \eta = (\eta_1,\eta_2,\eta_3,\eta_4,\eta_5)^\prime,\quad \xi = (\xi_1)^\prime. \]

Then the structural model can be written as:

\[ \eta = B\eta + \Gamma\xi + \zeta, \]

where \(\zeta=(\zeta_1,\ldots,\zeta_5)^\prime\), and (one possible representation)

\[ B= \begin{pmatrix} 0 & 0 & 0 & 0 & 0\\ \beta_{21} & 0 & 0 & 0 & 0\\ \beta_{31} & \beta_{32} & 0 & 0 & 0\\ \beta_{41} & \beta_{42} & \beta_{43} & 0 & 0\\ \beta_{51} & \beta_{52} & \beta_{53} & \beta_{54} & 0 \end{pmatrix}, \quad \Gamma= \begin{pmatrix} \gamma_{11}\\ \gamma_{21}\\ \gamma_{31}\\ \gamma_{41}\\ \gamma_{51} \end{pmatrix}. \]

3.3 Examples of direct, indirect, and total effects

Direct effect (example)

• Inputs \(\rightarrow\) Long-term Outcomes (if included): \(\gamma_{51}\)

One specific indirect effect (pure sequential chain)

• Inputs \(\rightarrow\) Activities \(\rightarrow\) Outputs \(\rightarrow\) Short-term \(\rightarrow\) Intermediate \(\rightarrow\) Long-term:

\[ \gamma_{11}\times \beta_{21}\times \beta_{32}\times \beta_{43}\times \beta_{54}. \]

Total effect

• The total effect equals the direct effect plus the sum of all possible indirect paths implied by the model.

4.1 Overview

Even though a comprehensive structural model can include direct paths from every earlier stage to every later stage, it is also valid to specify a parsimonious model that includes only the sequential causal chain implied by the logic model:

\[ \xi_1 \rightarrow \eta_1 \rightarrow \eta_2 \rightarrow \eta_3 \rightarrow \eta_4 \rightarrow \eta_5. \]

In equations (sequential paths only):

\[ \begin{aligned} \eta_1 \;&=\; \gamma_{11}\xi_1 + \zeta_1, \\ \eta_2 \;&=\; \beta_{21}\eta_1 + \zeta_2, \\ \eta_3 \;&=\; \beta_{32}\eta_2 + \zeta_3, \\ \eta_4 \;&=\; \beta_{43}\eta_3 + \zeta_4, \\ \eta_5 \;&=\; \beta_{54}\eta_4 + \zeta_5. \end{aligned} \]

In matrix form, letting

\[ \eta = (\eta_1,\eta_2,\eta_3,\eta_4,\eta_5)^\prime,\quad \xi = (\xi_1)^\prime, \]

the model can be written as:

\[ \eta = B\eta + \Gamma\xi + \zeta, \]

with

\[ B= \begin{pmatrix} 0 & 0 & 0 & 0 & 0\\ \beta_{21} & 0 & 0 & 0 & 0\\ 0 & \beta_{32} & 0 & 0 & 0\\ 0 & 0 & \beta_{43} & 0 & 0\\ 0 & 0 & 0 & \beta_{54} & 0 \end{pmatrix}, \quad \Gamma= \begin{pmatrix} \gamma_{11}\\ 0\\ 0\\ 0\\ 0 \end{pmatrix}. \]

4.2 Why start with a parsimonious model? (Theoretical parsimony)

In SEM, adding every possible path “just in case” can make the model unnecessarily complex. When the model becomes too complex relative to the sample size, it may lead to:

• loss of degrees of freedom,

• unstable estimation, and

• potential identification problems.

Therefore, a recommended strategy is to first fit a baseline SEM that reflects the core theoretical assumption of the logic model (the sequential stage-to-stage causality), and then consider adding additional paths only when there is strong theoretical and empirical justification.

With sequential paths only, it is easier to interpret mediation because direct and indirect effects are less entangled.

For example, the sequential-only model allows you to test the mediated pathway step by step, such as:

\[ \xi_1 \rightarrow \eta_3 \rightarrow \eta_4 \rightarrow \eta_5, \]

and later examine whether adding an extra direct path (e.g., \(\xi_1 \rightarrow \eta_4\)) is theoretically meaningful and empirically supported.

4.3 A standard stepwise modeling strategy

A common approach is:

• Model 1 (baseline): test the sequential logic-model paths only.

• Model 2 (expanded): add one theoretically motivated additional path at a time, and evaluate improvement in fit and statistical significance.

This stepwise strategy helps prevent overfitting and supports a more defensible final model by keeping only the “truly important” paths.

2.1 Level-1 (individual level): outcomes-only measurement

At the individual level, the dataset contains only outcome-related survey responses. Inputs / Activities / Outputs are library-level administrative variables, so they do not appear in Level-1.

A simple random-intercept-only specification is:

\[ \begin{aligned} y_{kij} \;&=\; \pi_{0kj} + \epsilon_{kij}, \\ \epsilon_{kij} \;&\sim\; N(0,\sigma_{\epsilon j}^2). \end{aligned} \]

• \(y_{kij}\): response of individual \(i\) in library \(k\) to outcome indicator/item \(j\)

• \(\pi_{0kj}\): library-specific mean (random intercept) for item \(j\)

• \(\epsilon_{kij}\): within-library individual-level deviation (treated as Level-1 error)

In this setup, small \(n_k\) is not automatically fatal because the multilevel model yields partial pooling at Level-2.

2.2 Level-2 (library level): measurement model including exogenous Inputs

At Level-2, both exogenous latent variables (\(\xi\)) and endogenous latent variables (\(\eta\)) are modeled.

2.3 Level-2 structural model (logic-model causal paths)

The logic-model causal chain is modeled at Level-2. A comprehensive version can allow sequential effects plus selected direct paths:

\[ \begin{aligned} \eta_{k1} \;&=\; \gamma_{11}\,\xi_k + \zeta_{k1}, \\ \eta_{k2} \;&=\; \beta_{21}\,\eta_{k1} + \gamma_{21}\,\xi_k + \zeta_{k2}, \\ \eta_{k3} \;&=\; \beta_{32}\,\eta_{k2} + \beta_{31}\,\eta_{k1} + \gamma_{31}\,\xi_k + \zeta_{k3}, \\ \eta_{k4} \;&=\; \beta_{43}\,\eta_{k3} + \beta_{42}\,\eta_{k2} + \gamma_{41}\,\xi_k + \zeta_{k4}, \\ \eta_{k5} \;&=\; \beta_{54}\,\eta_{k4} + \beta_{53}\,\eta_{k3} + \gamma_{51}\,\xi_k + \zeta_{k5}. \end{aligned} \]

• \(\xi_k\): Inputs (exogenous latent variable)

• \(\eta_{k1},\ldots,\eta_{k5}\): Activities \(\rightarrow\) Outputs \(\rightarrow\) Outcomes (endogenous latent variables)

• \(\zeta_{kj}\): library-level structural disturbances (residuals)