The conclusiveness of trial sequential analysis varies with estimation of between-study variance: a case study

Abstract

Background

Trial sequential methods have been introduced to address issues related to increased likelihood of incorrectly rejecting the null hypothesis in meta-analyses due to repeated significance testing. Between-study variance (τ²) and its estimate ($\widehat{\tau }$ ²) play a crucial role in both meta-analysis and trial sequential analysis with the random-effects model. Therefore, we investigated how different $\widehat{\tau }$ ² impact the results of and quantities used in trial sequential analysis.

Methods

This case study was grounded in a Cochrane review that provides data for smaller (< 10 randomized clinical trials, RCTs) and larger (> 20 RCTs) meta-analyses. The review compared various outcomes between video-laryngoscopy and direct laryngoscopy for tracheal intubation, and we used outcomes including hypoxemia and failed intubation, stratified by difficulty, expertise, and obesity. We calculated odds ratios using inverse variance method with six estimators for τ², including DerSimonian-Laird, restricted maximum-likelihood, Paule-Mandel, maximum-likelihood, Sidik-Jonkman, and Hunter-Schmidt. Then we depicted the relationships between $\widehat{\tau }$ ² and quantities in trial sequential analysis including diversity, adjustment factor, required information size (RIS), and α-spending boundaries.

Results

We found that diversity increases logarithmically with $\widehat{\tau }$ ², and that the adjustment factor, RIS, and α-spending boundaries increase linearly with $\widehat{\tau }$ ². Also, the conclusions of trial sequential analysis can differ depending on the estimator used for between-study variance.

Conclusion

This study highlights the importance of $\widehat{\tau }$ ² in trial sequential analysis and underscores the need to align the meta-analysis and the trial sequential analysis by choosing estimators to avoid introducing biases and discrepancies in effect size estimates and uncertainty assessments.

Graphical Abstract

What is new?

What is already known on this topic

Diversity, a quantity in trial sequential analysis, is related to between-study variance.

What this study adds

Between-study variance estimates are consistently associated with all quantities in trial sequential analysis, regardless of whether the meta-analysis is small or large.

How this study might affect research, practice or policy

Consistency between meta-analysis and trial sequential analysis in variance estimators is crucial for integrity and validity.

Introduction

Cumulative meta-analysis has been introduced as a useful method of identifying intervention benefits early and determining the statistical significance of evidence [1, 2], although challenges such as multiplicity and biased reporting complicate this approach. Repeated significance testing increases the likelihood of rejecting the null hypothesis [3], a concern that can be mitigated by using trial sequential methods. Group sequential analysis and cumulative z-curve modeling offer methods for monitoring trial progress and determining when to terminate a trial based on accumulating evidence [4,5,6]. These methods have been extended to meta-analysis for controlling falsely significant results (Type I errors, α) due to biases or random errors from repeated testing [1,2,3, 7,8,9]. Because of the sequential method's capacity for error correction in aggregating evidence, several articles in diverse medical specialties have introduced, endorsed, and apply trial sequential analysis [10,11,12,13,14,15]. These references frequently underscore the importance of identifying key quantities in computing for the method, which include the required information size (RIS), diversity (D²), and α-spending boundaries. Evidence is considered confirmed when the cumulative z-score crosses the α-spending monitoring boundaries [3, 10, 16]. The α-spending boundaries provide a threshold of significance by allocating the overall α among the cumulative analyses. This adjusted threshold guides researchers in determining whether the cumulative evidence reaches statistical significance. Hence, it is imperative to understand better not only the above quantities but also the various factors that are used in computing these quantities and thus may affect the results of a trial sequential analysis.

Between-study variance (τ²) is an important quantity for both the meta-analysis and the trial sequential analysis in a random-effects model. It is well-known that the estimate τ², $\widehat{\tau }$ ², is combined with within-study variance (denoted ν_i) to produce a weight for each study ${W}_{random.i}=\frac{1}{ ({\nu }_{i}+{\widehat{\tau }}^{2})}$, through which $\widehat{\tau }$ ² influences the estimated pooled effect (denoted $\widehat{\theta }$ _random) and it variance (denoted $\widehat{\nu }$ _random) [17]. The standard error and z-score for the pooled result are further derived based on $\widehat{\tau }$ ². Between-study variance is also a critical quantity in trial sequential analysis, as it affects the RIS. The RIS for a meta-analysis using the random-effects model ($\widehat{RIS}$ _random) can be expressed as follows:

$$\widehat{RIS}_{random}=4*\frac{{{(Z}_{\alpha /2}+{Z}_{\beta })}^{2}*{\widehat{\nu }}_{random}}{{\mu }_{random}^{2}}$$

(1)

where

α is the predefined overall probability of a false positive.

β is the predefined overall probability of a false negative.

μ² is the expected effect.

$\widehat{\nu }$ _random is the variance of pooled effects of the meta-analysis in random-effects model.

Note:

1. $\widehat{\nu }$ _random can be obtained from $\frac{1}{\sum_{i=1}^{j}{W}_{random.i}}$.

2. $\widehat{RIS}_{random}$ here is also unadjusted required information size ($\widehat{RIS}$ _{unadjusted.random}).

Considering heterogeneity (also known as between-study variance) in meta-analysis, the inclusion of an adjustment factor (AF) becomes imperative for estimating the RIS ($\widehat{RIS}$). Diversity (D²) has been proposed as a measure and as a foundational quantity for the AF and the adjusted RIS, which is the unadjusted RIS multiplied by the AF [18]. D² is the total relative variance when changing from a pooled analysis using the common-effect model to a meta-analysis using the random-effects model. The $\widehat{D}$ ² can be expressed as follows:

$$\widehat{D}^{2} =\frac{{\widehat{\nu }}_{random}- {\nu }_{fixed}}{{\widehat{\nu }}_{random}}$$

(2)

where

ν_fixed is the variance of pooled effects of the meta-analysis in fixed-effect model (also known as common-effect model).

$\widehat{\nu }$ _random is the variance of pooled effects of the meta-analysis in random-effects model.

An alternative expression of $\widehat{D}$ ² can highlight the role of the estimated between-study variance in the calculation, as follows:

$$\widehat{D}^{2}= \frac{1}{{\widehat{\tau }}^{2}}*({\widehat{\tau }}^{2}+ \frac{{\widehat{\tau }}^{2}* {\nu }_{fixed}}{ {\widehat{\nu }}_{random} - {\nu }_{fixed}})$$

(3)

where

ν_fixed is the variance of pooled effects of the meta-analysis in fixed-effect model (also known as common-effect model).

$\widehat{\nu }$ _random is the variance of pooled effects of the meta-analysis in random-effects model.

${\widehat{\tau }}^{2}$ is estimated between-study variance of the meta-analysis in random-effects model.

Then, the adjustment factor of the $\widehat{RIS}$ (denoted as $\widehat{AF}$) can be derived using D², as expressed $\widehat{AF}$ = $\frac{1}{{(1-\widehat{D}}^{2})}$. Since $\widehat{\tau }$ ² is a crucial quantity in the random-effects model, it impacts the estimates of diversity ($\widehat{D}$ ²), adjustment factor ($\widehat{AF}$), and the adjusted RIS ($\widehat{RIS}$ _adjusted). The estimate of τ², $\widehat{\tau }$ ², is crucial for both meta-analysis and trial sequential analysis using a random-effects model. Previous studies have pointed out the importance of between-study variance within trial sequential analysis using a random-effects model, but the influence of different estimators of between-study variance on various quantities in such an analysis warrants further discussion [3, 18, 19].

The RIS plays a crucial role in trial sequential analysis, serving as both a threshold for the methodological and clinical establishment of evidence, and as the foundation for calculating α-spending boundaries [3, 7, 10]. The concept of RIS originated from the optimal information size, drawing upon quantities such as expected effect size, variance, and predetermined α and Type II errors (β) [3, 7]. The RIS can be obtained using the expected effect size, the variance of the pooled model, and predetermined α and β.

In sequential meta-analysis using the random-effects model, $\widehat{\tau }$ ² plays a critical role due to its impact on both the observed cumulative z-score and $\widehat{D}$ ². An estimator for the between-study variance can be derived by various approaches, e.g., the method of moments, maximum likelihood, and the model error variance estimator [20]. Owing to differences between these different estimators of between-study variance [21,22,23], both $\widehat{D}$ ² and $\widehat{RIS}$ can also differ depending on the estimator chosen. This case study aimed to enhance understanding of how the choice of $\widehat{\tau }$ ² influences quantities used in trial sequential analysis in random-effects model.

Methods

This case study aimed to elucidate the influence of $\widehat{\tau }$ ² in trial sequential analysis by illustrating its relationship with the observed cumulative z-score, estimated diversity ($\widehat{D}$ ²), adjustment factor ($\widehat{AF}$), required information size ($\widehat{RIS}$), and the z-score of the α-spending boundaries. The registration for this study can be accessed through the Open Science Framework (https://osf.io/czstm). In addition to the registration, we add an example to demonstrate the relationship between $\widehat{\tau }$ ² and the conclusiveness of evidence. This study was proposed to depict the aforementioned relationship across various study sizes, considering the presence or absence of significant heterogeneity. Consequently, we intended to identify a Cochrane review that includes both smaller and larger meta-analyses, including cases of both significant and non-significant heterogeneity based on the specified criteria as follows: (1) systematic reviews with meta-analyses having < 10 randomized clinical trials (RCTs) and > 20 RCTs, (2) outcomes with varying levels of between-study variance, and (3) data availability.

The case study used here

A systematic review conducted by Hansel et al. (2022) fulfilling the criteria outlined above was chosen for use in this study [24]. Specifically, this study used data from the Cochrane review that investigated the comparative effects of video-laryngoscopy and direct laryngoscopy in adults undergoing tracheal intubation. The Cochrane review includes outcomes with or without significant heterogeneity according to the P-value from the χ²-test for heterogeneity (P < 0.1) based on the DerSimonian-Laird method (DL). The low power of the χ²-test for heterogeneity in meta-analyses with small or few studies means that a non-significant result cannot be taken as evidence of no heterogeneity, which justifies sometimes using a P-value of 0.10 to assess statistical significance instead of the conventional 0.05 [25]. We extracted four outcomes from the review, including a small meta-analysis comparing hypoxemia using seven RCTs without significant heterogeneity (P-value = 0.68), failed intubation in difficult cases using nine RCTs with significant heterogeneity (P-value = 0.0993). We also analyzed larger meta-analyses focusing on failed intubation using 48 and 62 RCTs, categorized by practitioner expertise without significant heterogeneity (P-value = 0.65) and by obesity status with significant heterogeneity (P-value = 0.0003), respectively (Supplementary file 1). We also selected data on failed intubations comparing hyper-angulated video laryngoscopy and direct laryngoscopy in patients with difficult cases to provide a clear example of evidence conclusiveness using $\widehat{\tau }$ ².

Methods for estimating between-study variance

For each of the above outcomes, we estimated the odds ratio using inverse variance. The present study does not include the results from Mantel–Haenszel and Peto’s pooling methods due to discouragement about their applicability, despite the intention to incorporate them in our initial registration and early analyses. To understand the role of $\widehat{\tau }$ ² in trial sequential analysis, we considered six estimators for τ²: DL [26], restricted maximum-likelihood (REML) [27], Paule-Mandel [28], maximum-likelihood [27], Sidik-Jonkman [29], and Hunter-Schmidt [30]. Equations for the estimators are given in Supplementary file 2.

Methods for trial sequential analysis

Supplementary file 3 presents detailed formulae for the trial sequential analysis methods used in this study [3, 7, 18]. The main formulae are for $\widehat{D}$ ², $\widehat{AF}$, $\widehat{RIS}$, and the z-score of the α-spending boundaries. Briefly, $\widehat{\tau }$ ² is calculated as part of using the random-effects model and influences the variance of the pooled estimate, which directly impacts both $\widehat{D}$ ² and $\widehat{RIS}$. Subsequently, through $\widehat{D}$ ² and $\widehat{RIS}$, $\widehat{\tau }$ ² and the variance of the pooled estimate also affect $\widehat{AF}$ and the z-score of the α-spending boundaries. We describe variability of those quantities using range and quartile coefficient of variation (QCV) [31, 32].

Software

All analyses were done using R version 4.2.2, specifically the function `metabin()` of the package meta (version 7.0–0) to perform meta-analysis and the function `DoTSA()` of the package smiles (version 0.1–0) to perform trial sequential analysis [33,34,35,36]. The R package RTSA has been endorsed by Copenhagen Trial Unit for performing trial sequential analysis [37]; however, it has limited options for estimating the between-study variance. Therefore, we opted the R package smiles, which supports all the estimators of between-study variance mentioned in the R package meta. When given the same inputs, the TSA software (version 0.9.5.10 Beta, Copenhagen Trial Unit, Copenhagen University Hospital – Rigshospitalet, Denmark) and the function `DoTSA()` of the R package smiles returned the same values for the RIS. Supplementary file 4 shows one example using hypoxaemia data with the DL estimator, α = 0.05, β = 0.2, relative risk reduction 20%. Supplementary file 5 gives the R code used for this study. Supplementary file 6 summarizes the quantities computed in the trial sequential analyses.

Results

Supplementary file 7 shows the sequence of calculations through which the estimated between-study variance ($\widehat{\tau }$ ²) impacts the results of both a meta-analysis and a trial sequential analysis. As the observed cumulative z-score of a meta-analysis is commonly compared to the z-score of an α-spending monitoring boundaries in a trial sequential analysis, in the present study we focus on exploring the relationship between $\widehat{\tau }$ ² and the observed cumulative z-score of the meta-analysis, and on the relationship between $\widehat{\tau }$ ² and the calculations done in parallel with $\widehat{D}$ ², $\widehat{AF}$, $\widehat{RIS}$, and the z-score of the α-spending boundaries.

Relationship between $\widehat{{\varvec{\tau}}}$ ² and $\widehat{{\varvec{D}}}$ ²

Figure 1A shows that $\widehat{D}$ ² increases as $\widehat{\tau }$ ² increases; the relationship appears to follow a logarithmic curve because $\widehat{D}$ ² is between 0 and 1. This increasing trend is consistent across the four outcomes, while $\widehat{D}$ ² seems to be sensitive to the selection of $\widehat{\tau }$ ² due to its range between 0 to 0.43, 0.25 to 0.71, 0 to 0.68, and 0.40 to 0.71 in the four outcomes. In meta-analysis with non-significant heterogeneity, the QCV of $\widehat{D}$ ² achieved 100% among the six between-study variance estimators (Supplementary file 8). Because the Sidik-Jonkman approach produces the largest $\widehat{\tau }$ ² across the four outcomes, the red points for Sidik-Jonkman usually have the largest $\widehat{D}$ ².

Fig. 1

Scatter plots of (A) diversity ($\widehat{D}$ ²) versus estimated between-study variance ($\widehat{\tau }$ ²) and (B) required information size ($\widehat{RIS}$) versus estimated between-study variance ($\widehat{\tau }$ ²), for smaller meta-analysis without significant heterogeneity, smaller meta-analysis with significant heterogeneity, larger meta-analysis without significant heterogeneity, and larger meta-analysis with significant heterogeneity. DL, DerSimonian-Laird estimator; HS, Hunter-Schmidt estimator; ML, Maximum-likelihood estimator; PM, Paule-Mandel estimator; REML, Restricted maximum-likelihood estimator; SJ, Sidik-Jonkman estimator

Relationship between $\widehat{{\varvec{\tau}}}$ ², the adjustment factor and the required information size

The diversity-based adjustment factor ($\widehat{AF}$ _D²) increases linearly as $\widehat{\tau }$ ² increases (Supplementary file 8), and the relationship between $\widehat{\tau }$ ² and the diversity-adjusted required information size ($\widehat{RIS}$ _D²) was also nearly linear (Fig. 1B). These two figures have similar patterns because $\widehat{RIS}$ _D² is computed as $\widehat{RIS}$ _unadjusted $\widehat{AF}$ _D². However, both quantities are sensitive to the selection of $\widehat{\tau }$ ², even in meta-analyses without significant heterogeneity. $\widehat{AF}$ _D² varies from 1 to 1.75, 1.34 to 3.48, 1 to 2.81, and 1.68 to 3.41 across the four outcomes. Similarly, $\widehat{RIS}$ _D² varies between 3789 and 6655, 5940 and 15,477, 8254 and 23,190, and 14,588 and 29,629 for the respective outcomes. The QCV ranges between 0 and 16% across the four outcomes.

Relationship between $\widehat{{\varvec{\tau}}}$ ² and α-spending boundaries

Figure 2A shows a trend in which, in most panels of the Figure, the α-spending boundaries increase linearly as $\widehat{\tau }$ ² increases. The minimum α-spending boundary ranges from − 6.88 to − 5.20 (QCV = 0%), from − 7.13 to − 4.53 (QCV = 8%), from − 3.81 to − 2.27 (QCV = 3%), and from − 3.71 to − 2.60 (QCV = 5%) for the four outcomes respectively.

Fig. 2

Scatter plots of (A) α-spending monitoring boundary versus estimated between-study variance ($\widehat{\tau }$ ²) and (B) observed cumulative z-score versus estimated between-study variance ($\widehat{\tau }$ ²), for smaller meta-analysis without significant heterogeneity, smaller meta-analysis with significant heterogeneity, larger meta-analysis without significant heterogeneity, and larger meta-analysis with significant heterogeneity. DL, DerSimonian-Laird estimator; HS, Hunter-Schmidt estimator; ML, Maximum-likelihood estimator; PM, Paule-Mandel estimator; REML, Restricted maximum-likelihood estimator; SJ, Sidik-Jonkman estimator

Relationship between $\widehat{{\varvec{\tau}}}$ ² and observed cumulative z-score

In Fig. 2B, the primary phenomenon in all four outcomes is a decreasing trend in cumulative z-score as $\widehat{\tau }$ ² increases. The minimum cumulative z-score varies, with a range of − 4.13 to − 3.14 for QCV at 0%, − 3.32 to − 2.99 for QCV at 2%, − 8.23 to − 4.90 for QCV at 4%, and − 6.03 to − 4.29 for QCV at 5%, reflecting the four distinct outcomes. Notably, the Sidik-Jonkman approach seems to yield the largest $\widehat{\tau }$ ² among the estimators irrespective of outcome.

Evidence conclusiveness by $\widehat{{\varvec{\tau}}}$ ²

Based on data comparing failed intubations between hyper-angulated video laryngoscopy and direct laryngoscopy in patients with difficult cases, the conclusiveness of the evidence depends on the choice of the estimator $\widehat{\tau }$ ². The evidence is conclusive and statistically significant when applying REML, Hunter-Schmidt, and maximum-likelihood estimators (Fig. 3).

Fig. 3

Trial sequential analysis plots using different between-study variance estimators ($\widehat{\tau }$ ²) for failed intubations comparing hyper-angulated video laryngoscopy and direct laryngoscopy. DL, DerSimonian-Laird estimator; HS, Hunter-Schmidt estimator; ML, Maximum-likelihood estimator; PM, Paule-Mandel estimator; REML, Restricted maximum-likelihood estimator; SJ, Sidik-Jonkman estimator

Discussion

This study found distinct patterns showing how the estimator of between-study variance, $\widehat{\tau }$ ², affects both meta-analysis and trial sequential analysis. The profound implications of choosing an estimator for τ² are well known [20,21,22,23, 38]. These investigations found wide disparities in estimates of τ² across diverse scenarios characterized by differing data types (dichotomous and continuous), effect sizes, and levels of true τ². For instance, the Sidik-Jonkman approach consistently tends to overestimate τ², irrespective of whether the meta-analysis is small or large, and regardless of the type of outcome (dichotomous or continuous) [20, 23]. This phenomenon can also be observed in our study (red points in Figs. 1 and 2).

Importantly, $\widehat{\tau }$ ² is the bedrock of trial sequential analysis using the random-effects model [15]. It is a critical quantity for estimating the RIS, which is essential for establishing a threshold of information sufficiency and for defining α-spending boundaries [3, 7, 18]. Typically, adjusting the RIS is necessary when there is between-study variance (τ²) in a meta-analysis using the random-effects model. The adjustment is fundamentally based on $\widehat{\tau }$ ² and is especially relevant when incorporating diversity ($\widehat{D}$ ²) into the adjustment process [18]. The patterns in the figures offer an overview on the crucial role of selecting an estimator of between-study variance in trial sequential analysis due to the impact of $\widehat{\tau }$ ² on $\widehat{D}$ ², $\widehat{AF}$, $\widehat{RIS}$, and the z-score of the α-spending boundaries.

Implications

Given the need to choose an estimator of between-study variance in trial sequential analysis, two practical recommendations emerge. First, consistency between meta-analysis and trial sequential analysis in selecting between-study variance estimators is paramount for maintaining the integrity and validity of the research findings. Inconsistency in these choices can introduce biases and undermine the reliability of conclusions drawn from sequential analyses. For instance, using different between-study variance estimators in the two analytical phases may lead to discrepancies in effect size estimates and uncertainty assessments, impacting the interpretation of results. Therefore, researchers must carefully align these choices of methods across meta-analysis and trial sequential analysis. Secondly, protocol design for synthesizing evidence using sequential methods requires careful consideration of the estimator that is chosen. Therefore, including these choices of methods in the protocol ensures the consistency and rigor necessary for producing trustworthy evidence through sequential synthesis methods.

Limitations

While this study delineates step-by-step the role of between-study variance at various stages of trial sequential analysis, it is important to acknowledge several limitations. Primarily, the study's scope was constrained by the availability of published data from a Cochrane review, limiting the depth of analysis for certain phenomena. For instance, while this study provides valuable insights into estimated between-study variance ($\widehat{\tau }$ ²), further investigation is warranted across diverse scenarios such as different effect sizes, sparse data, and unequal sample sizes. A comprehensive examination of the varying impacts of $\widehat{\tau }$ ² will require exploration through simulation studies. Second, this study's findings rely on between-study variance estimators that are commonly used in meta-analyses and cannot show which estimators are better or worse for trial sequential analysis across different scenarios, underscoring the need for further research in this area. Third, this work performs the role of $\widehat{\tau }$ ² in trial sequential analysis based on dichotomous data, while we know that $\widehat{\tau }$ ² varies by differing data types. Further studies are encouraged to use continuous data to illustrate the relationships and quantities in trial sequential meta-analysis within a random-effects model. Lastly, the study does not explore the relationship between $\widehat{\tau }$ ² and the β-spending function, which is a reasonable decision considering its primary focus. However, we contend that $\widehat{\tau }$ ² may have an impact on the β-spending function because of its influence on the RIS. Further investigation into how $\widehat{\tau }$ ² affects the β-spending function would be valuable.

Conclusions

This study sheds light on the influence of the estimator of between-study variance in trial sequential analysis, emphasizing the crucial need for predetermined and consistent use of estimation methods for this variance. Besides, trial sequential analysis seems to be sensitive to the $\widehat{\tau }$ ² used to establish its boundaries. This presents a challenge, as the estimation of between-study variance frequently lacks precision, especially in cases with a limited number of studies. By enriching our understanding of these complexities, these efforts will enhance the integrity and utility of trial sequential analysis in informing evidence-based practice and decision-making in healthcare.

Availability and requirements

Project name: Sequential Method In Leading Evidence Synthesis (SMILES).

Project home page: https://osf.io/td689/.

Operating system: Microsoft Windows.

Programming language: R

Other requirements: R version 4.2.2

License: General Public License.