Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data

Wan, Wen; Murugesan, Manoradhan; Nocon, Robert S.; Bolton, Joshua; Konetzka, R. Tamara; Chin, Marshall H.; Huang, Elbert S.

doi:10.1186/s12874-024-02228-z

Table 3 Evaluation of OW and FS methods by simulation with the constant true effect and by exposure prevalence with observed outcome risk or with simulated outcome risk of 1%

From: Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data

Exposure Prevalence	Methods	MB	rbias	SE	SD(rbias)	rMSE	Coverage	CoverageT	Significance	N used
Outcome risk = 27.75% (observed)
2.5%	Crude	0.6434	-72.82	0.139	13.020	0.740	0.0	0.0	53.0	4000
	FS_F−equ	0.1965	-60.23	0.163	14.365	0.619	0.4	0.4	69.6	3859
	FS_X−equ	0.1884	-60.44	0.187	13.245	0.619	0.6	0.6	55.8	2801
	FS_F−unequ	0.2906	-60.23	0.163	14.365	0.619	0.4	0.4	69.6	3859
	FS_X−unequ	0.2932	-59.23	0.165	14.428	0.610	0.8	0.8	71.0	2801
	OW_F	0.0312	-54.61	0.140	12.077	0.559	1.0	1.0	89.0	4000
	OW_X	0.1551	-58.73	0.170	14.517	0.605	1.2	1.0	70.2	2883
10%	Crude	0.5556	-71.85	0.073	7.672	0.723	0.0	0.0	94.8	4000
	FS_F−equ	0.0851	-60.21	0.077	6.703	0.606	0.0	0.0	99.8	3969
	FS_X−equ	0.0783	-60.33	0.078	6.840	0.607	0.0	0.0	99.8	3804
	FS_F−unequ	0.1195	-60.21	0.077	6.703	0.606	0.0	0.0	99.8	3969
	FS_X−unequ	0.1131	-60.33	0.078	6.840	0.607	0.0	0.0	99.8	3804
	OW_F	0.0008	-54.77	0.075	6.666	0.552	0.0	0.0	100.0	4000
	OW_X	0.0001	-54.82	0.075	6.769	0.552	0.0	0.0	100.0	3828
30%	Crude	0.5173	-72.00	0.050	4.860	0.722	0.0	0.0	100.0	4000
	FS_F−equ	0.0505	-59.43	0.051	4.167	0.596	0.0	0.0	100.0	3985
	FS_X−equ	0.0480	-59.45	0.052	4.238	0.596	0.0	0.0	100.0	3925
	FS_F−unequ	0.0710	-59.43	0.051	4.167	0.596	0.0	0.0	100.0	3985
	FS_X−unequ	0.0677	-59.45	0.052	4.238	0.596	0.0	0.0	100.0	3925
	OW_F	0.0003	-55.83	0.052	4.303	0.560	0.0	0.0	100.0	4000
	OW_X	0.0000	-55.85	0.053	4.335	0.560	0.0	0.0	100.0	3938
Outcome risk = 1% *
10%	Crude	0.5555	-76.04	0.492	120.55	1.426	79.2	17.0	17.2	4000
	FS_F−equ	0.0831	-34.09	0.522	131.28	1.357	94.8	39.0	40.4	3967
	FS_X−equ	0.0790	-36.08	0.525	131.62	1.365	95.8	36.6	37.8	3803
	FS_F−unequ	0.1170	-33.69	0.522	123.27	1.278	94.8	39.0	40.4	3967
	FS_X−unequ	0.1142	-35.77	0.525	123.63	1.287	95.8	36.6	37.8	3803
	OW_F	0.0007	-16.84	0.505	129.39	1.305	96.4	48.8	51.8	4000
	OW_X	0.0001	-17.35	0.508	129.60	1.308	96.6	48.0	51.0	3829
30%	Crude	0.5155	-63.22	0.316	33.09	0.714	47.4	22.4	22.4	4000
	FS_F−equ	0.0487	-20.00	0.326	32.86	0.385	92.2	69.8	70.4	3985
	FS_X−equ	0.0467	-21.63	0.328	32.75	0.392	91.8	67.2	67.4	3925
	FS_F−unequ	0.0686	-20.00	0.326	32.86	0.385	92.2	69.8	70.4	3985
	FS_X−unequ	0.0658	-21.63	0.328	32.75	0.392	91.8	67.2	67.4	3925
	OW_F	0.0003	-4.04	0.330	32.95	0.332	96.0	80.2	82.0	4000
	OW_X	0.0000	-4.75	0.332	33.22	0.336	96.0	80.4	82.2	3939

Footnotes:
*The simulation scenario with 1% outcome risk and 2.5% exposure prevalence is not shown here because it has been shown in Table 1
1. Crude = summarized by raw data without any balancing method; OW = overlap weighting method; FS = propensity score based fine stratification method
2. ‘F’ = a full set of data; ‘X’ = a subset of data after removing those unmatched
3. ‘equ’ = ATE with the equal weighting between groups; ‘unequ’ = ATE with the unequal weighting, where total weight in one group equivalent to the sample size in that group
4. The best values are bolded and can be used to guide which method performs the best per evaluation criterion
5. MB = Mahalanobis balance; rBias = relative bias = 100*(estimated effect – true effect) /true effect; SE = average estimated standard error; SD(rBias) = empirical standard deviation of relative bias x 100; rMSE = square root of mean squared error that combines squared bias (not relative bias) and its variance; Coverage = proportion of samples whose 95% CI cover the true effect; CoverageT = proportion of samples whose 95% CI cover the true effect but not zero; Significance = proportion of samples obtaining a significant effect (by a weighted GLM with a two-sided p-value < 0.05); N used = average total sample size that was used further for GLM.

Back to article page

ISSN: 1471-2288

Contact us

General enquiries: journalsubmissions@springernature.com

Your privacy, your choice

Table 3 Evaluation of OW and FS methods by simulation with the constant true effect and by exposure prevalence with observed outcome risk or with simulated outcome risk of 1%

BMC Medical Research Methodology

Contact us