Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data

Wan, Wen; Murugesan, Manoradhan; Nocon, Robert S.; Bolton, Joshua; Konetzka, R. Tamara; Chin, Marshall H.; Huang, Elbert S.

doi:10.1186/s12874-024-02228-z

Table 2 Evaluation of OW and FS methods by simulation with the constant true effect by outcome risk along with observed/simulated exposure

From: Comparison of two propensity score-based methods for balancing covariates: the overlap weighting and fine stratification methods in real-world claims data

Outcome risk	Methods	MB	rbias	SE	SD(rBias)	rMSE	Coverage	CoverageT	Significance	N used
Exposure prevalence = 10.55% (observed)
1%	Crude	0.5542	-66.85	0.470	49.37	0.831	78.0	17.2	17.2	4000
	FS_F−equ	0.0811	-25.04	0.498	51.46	0.572	96.0	41.0	41.8	3971
	FS_X−equ	0.0760	-26.00	0.500	52.11	0.582	96.2	38.8	39.6	3812
	FS_F−unequ	0.1146	-25.04	0.498	51.46	0.572	96.0	41.0	41.8	3971
	FS_X−unequ	0.1098	-26.00	0.500	52.11	0.582	96.2	38.8	39.6	3812
	OW_F	0.0006	-7.69	0.483	47.79	0.484	97.0	52.4	54.8	4000
	OW_X	0.0001	-8.08	0.486	48.45	0.491	97.2	51.6	54.0	3835
10%	Crude	0.5524	-70.06	0.136	14.49	0.715	0.0	0.0	59.6	4000
	FS_F−equ	0.0810	-47.74	0.146	12.95	0.495	5.2	5.2	94.0	3970
	FS_X−equ	0.0761	-47.92	0.147	13.08	0.497	5.0	5.0	93.8	3815
	FS_F−unequ	0.1145	-47.74	0.146	12.95	0.495	5.2	5.2	94.0	3970
	FS_X−unequ	0.1100	-47.92	0.147	13.08	0.497	5.0	5.0	93.8	3815
	OW_F	0.0006	-39.30	0.140	12.30	0.412	14.4	14.4	99.6	4000
	OW_X	0.0001	-39.52	0.141	12.28	0.414	14.6	14.6	99.6	3839
30%	Crude	0.5526	-72.39	0.067	6.91	0.727	0.0	0.0	96.8	4000
	FS_F−equ	0.0803	-61.44	0.071	6.18	0.617	0.0	0.0	100.0	3968
	FS_X−equ	0.0759	-61.47	0.071	6.19	0.618	0.0	0.0	100.0	3814
	FS_F−unequ	0.1135	-61.44	0.071	6.18	0.617	0.0	0.0	100.0	3968
	FS_X−unequ	0.1096	-61.47	0.071	6.19	0.618	0.0	0.0	100.0	3814
	OW_F	0.0006	-56.17	0.069	6.05	0.565	0.0	0.0	100.0	4000
	OW_X	0.0001	-56.20	0.070	6.10	0.565	0.0	0.0	100.0	3840
Exposure prevalence = 2.5%
1%	Crude	0.643	-564.17	0.815	959.38	11.17	76.4	10.0	33.6	4000
	FS_F−equ	0.196	-689.45	0.849	1173.21	13.64	72.2	17.2	44.4	3859
	FS_X−equ	0.188	-663.38	0.863	1157.41	13.36	72.4	15.4	41.8	3360
	FS_F−unequ	0.291	-596.65	0.849	1005.94	11.69	72.2	17.2	44.4	3859
	FS_X−unequ	0.293	-575.08	0.863	996.42	11.50	72.4	15.4	41.8	3360
	OW_F	0.031	-606.14	0.825	1129.22	12.82	74.8	24.0	49.2	4000
	OW_X	0.116	-614.94	0.836	1132.19	12.89	74.6	23.4	48.8	3459
10%	Crude	0.645	-73.23	0.270	27.75	0.78	13.2	13.2	24.8	4000
	FS_F−equ	0.196	-48.49	0.319	30.44	0.57	76.0	42.4	42.6	3857
	FS_X−equ	0.188	-48.96	0.323	31.65	0.58	76.0	42.0	42.2	3339
	FS_F−unequ	0.291	-48.49	0.319	30.44	0.57	76.0	42.4	42.6	3857
	FS_X−unequ	0.294	-48.96	0.323	31.65	0.58	76.0	42.0	42.2	3339
	OW_F	0.006	-40.89	0.273	22.97	0.47	78.4	61.0	63.4	4000
	OW_X	0.123	-42.62	0.277	35.88	0.56	77.6	59.6	61.6	3443
30%	Crude	0.645	-73.25	0.131	12.57	0.74	0.0	0.0	56.2	4000
	FS_F−equ	0.191	-61.63	0.154	14.17	0.63	0.0	0.0	70.8	3860
	FS_X−equ	0.186	-60.66	0.155	13.79	0.62	0.2	0.2	72.4	3355
	FS_F−unequ	0.281	-61.63	0.154	14.17	0.63	0.0	0.0	70.8	3860
	FS_X−unequ	0.287	-60.66	0.155	13.79	0.62	0.2	0.2	72.4	3355
	OW_F	0.030	-56.15	0.132	11.02	0.57	0.0	0.0	90.8	4000
	OW_X	0.031	-56.21	0.133	11.06	0.57	0.2	0.2	90.6	3454

Footnotes:
1. Crude = summarized by raw data without any balancing method; OW = overlap weighting method; FS = propensity score based fine stratification method
2. ‘F’ = a full set of data; ‘X’ = a subset of data after removing those unmatched
3. ‘equ’ = ATE with the equal weighting between groups; ‘unequ’ = ATE with the unequal weighting, where total weight in one group equivalent to the sample size in that group
4. The best values are bolded and can be used to guide which method performs the best per evaluation criterion
5. MB = Mahalanobis balance; rBias = relative bias = 100*(estimated effect – true effect) /true effect; SE = average estimated standard error; SD(rBias) = empirical standard deviation of relative bias x 100; rMSE = square root of mean squared error that combines squared bias (not relative bias) and its variance; Coverage = proportion of samples whose 95% CI cover the true effect; CoverageT = proportion of samples whose 95% CI cover the true effect but not zero; Significance = proportion of samples obtaining a significant effect (by a weighted GLM with a two-sided p-value < 0.05); N used = average total sample size that was used further for GLM.

Back to article page

ISSN: 1471-2288

Contact us

General enquiries: journalsubmissions@springernature.com

BMC Medical Research Methodology

Contact us