{"title":"The wrong comparison.","description":"When customers choose their own treatment, correlation can masquerade as impact. Change the assignment mechanism and inspect the bias.","theme":"violet","kind":"Causal inference","cases":[{"id":"randomized","name":"Random assignment","title":"More data cannot fix the wrong comparison.","unit":"Estimated average treatment effect","parameter":"Sample size","x":[200,500,1000,2000,5000],"series":[{"label":"Unadjusted difference","values":[1.9895563,1.9454889,2.0195726,2.0149538,1.9966412],"low":[1.9123195,1.8972333,1.9859384,1.9905511,1.9812939],"high":[2.0667931,1.9937445,2.0532067,2.0393565,2.0119884]},{"label":"OLS · adjust Z","values":[2.0100118,1.9548224,2.0098566,2.0064807,2.0011581],"low":[1.9558641,1.920299,1.9842278,1.9896763,1.9887777],"high":[2.0641595,1.9893459,2.0354854,2.0232851,2.0135384]},{"label":"IPW · observed propensity","values":[1.9525537,1.9174116,2.026318,1.9777292,2.0034781],"low":[1.811535,1.8289066,1.9590513,1.931284,1.9737862],"high":[2.0935725,2.0059166,2.0935847,2.0241745,2.03317]},{"label":"Ground truth","values":[2,2,2,2,2]}],"snapshots":[{"rows":[["Unadjusted difference",1.9895563118792652,-0.010443688120734818,0.3504084632608925],["OLS · adjust Z",2.010011776049913,0.010011776049913212,0.24575260508062394],["IPW · observed propensity",1.9525537484683824,-0.047446251531617634,0.6412483768272909]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",1.9454888897642035,-0.05451111023579647,0.22551658100255306],["OLS · adjust Z",1.954822442045272,-0.045177557954728,0.16294491584984805],["IPW · observed propensity",1.9174115792011386,-0.08258842079886142,0.40976117941592427]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",2.0195725655188186,0.019572565518818585,0.15377460036341092],["OLS · adjust Z",2.009856575513873,0.00985657551387309,0.11663844280559221],["IPW · observed propensity",2.0263179914319145,0.02631799143191449,0.30617363376989304]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",2.014953798508226,0.01495379850822598,0.1116670384814446],["OLS · adjust Z",2.006480667360619,0.00648066736061903,0.07647951714823942],["IPW · observed propensity",1.9777292031607216,-0.02227079683927835,0.21179377801326169]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",1.996641186206117,-0.00335881379388292,0.06967764411131037],["OLS · adjust Z",2.0011580607674255,0.0011580607674255283,0.05615411370743293],["IPW · observed propensity",2.0034780938435213,0.0034780938435212683,0.13469147007137552]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."}],"columns":["Estimator","Mean estimate","Bias","RMSE"],"context":"A is independent of Z and U. Outcome Y = 2A + 3Z + 3U + noise.","readout":"80 repeated datasets per size. Bands describe uncertainty of the mean estimate across simulations (±1.96 SE), not the sampling interval for one fitted effect.","note":""},{"id":"observed","name":"Observed confounding","title":"More data cannot fix the wrong comparison.","unit":"Estimated average treatment effect","parameter":"Sample size","x":[200,500,1000,2000,5000],"series":[{"label":"Unadjusted difference","values":[3.3723405,3.4025538,3.3680843,3.3927242,3.3808259],"low":[3.3058216,3.3594949,3.3323892,3.3663199,3.3660747],"high":[3.4388594,3.4456126,3.4037793,3.4191284,3.3955771]},{"label":"OLS · adjust Z","values":[1.9594387,2.0336303,1.9847009,2.0042667,2.0037383],"low":[1.9055153,1.9932129,1.9564217,1.9829808,1.9894923],"high":[2.0133622,2.0740478,2.0129801,2.0255526,2.0179843]},{"label":"IPW · observed propensity","values":[2.0048303,2.0103294,2.0152793,1.9836505,2.0015887],"low":[1.8600201,1.9104052,1.9503468,1.9323821,1.9711663],"high":[2.1496406,2.1102535,2.0802117,2.034919,2.0320111]},{"label":"Ground truth","values":[2,2,2,2,2]}],"snapshots":[{"rows":[["Unadjusted difference",3.3723405157959916,1.3723405157959916,1.4051017330585969],["OLS · adjust Z",1.9594387445166894,-0.040561255483310577,0.2478729521423502],["IPW · observed propensity",2.0048303483856147,0.004830348385614691,0.6567023408135676]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",3.402553767109481,1.402553767109481,1.416080747358747],["OLS · adjust Z",2.0336303390336523,0.03363033903365231,0.18634454293041602],["IPW · observed propensity",2.0103293635958837,0.010329363595883656,0.45325295074172783]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",3.368084253685862,1.3680842536858622,1.3776270454539759],["OLS · adjust Z",1.9847009219459077,-0.015299078054092252,0.12914975663046394],["IPW · observed propensity",2.015279271799421,0.01527927179942079,0.2948514887766514]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",3.392724151062828,1.3927241510628279,1.3978618324367664],["OLS · adjust Z",2.0042666845407453,0.0042666845407453025,0.09662141920457389],["IPW · observed propensity",1.9836505442338523,-0.016349455766147658,0.23306593997922456]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",3.3808258950182277,1.3808258950182277,1.382445264241723],["OLS · adjust Z",2.003738305575805,0.003738305575804901,0.06471060052722315],["IPW · observed propensity",2.001588711809229,0.0015887118092288155,0.13796859594141556]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."}],"columns":["Estimator","Mean estimate","Bias","RMSE"],"context":"Z causes both treatment A and outcome Y. U affects Y but not A. Adjusting Z blocks the confounding path.","readout":"80 repeated datasets per size. Bands describe uncertainty of the mean estimate across simulations (±1.96 SE), not the sampling interval for one fitted effect.","note":""},{"id":"hidden","name":"Hidden confounding remains","title":"More data cannot fix the wrong comparison.","unit":"Estimated average treatment effect","parameter":"Sample size","x":[200,500,1000,2000,5000],"series":[{"label":"Unadjusted difference","values":[4.2684511,4.2962473,4.2745925,4.2855566,4.2938829],"low":[4.210707,4.2561643,4.2445081,4.2653639,4.2812249],"high":[4.3261953,4.3363303,4.3046769,4.3057493,4.3065409]},{"label":"OLS · adjust Z","values":[3.3568445,3.3428034,3.3342782,3.3341806,3.3496323],"low":[3.3074345,3.310367,3.3091208,3.3170671,3.3374563],"high":[3.4062544,3.3752398,3.3594356,3.3512941,3.3618083]},{"label":"IPW · observed propensity","values":[3.2562705,3.3294682,3.3431,3.30924,3.3508414],"low":[3.1039798,3.2352565,3.2754048,3.2654803,3.3206387],"high":[3.4085611,3.4236798,3.4107953,3.3529997,3.3810441]},{"label":"Ground truth","values":[2,2,2,2,2]}],"snapshots":[{"rows":[["Unadjusted difference",4.268451132619028,2.2684511326190284,2.2835148580351396],["OLS · adjust Z",3.356844460099358,1.3568444600993579,1.3752205230214287],["IPW · observed propensity",3.2562704663076674,1.2562704663076674,1.4335804617242598]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",4.296247274083103,2.296247274083103,2.303430306612371],["OLS · adjust Z",3.342803396956974,1.342803396956974,1.3508356962822183],["IPW · observed propensity",3.3294681898114105,1.3294681898114105,1.3964280867168635]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",4.274592496356013,2.2745924963560133,2.2786801611372813],["OLS · adjust Z",3.3342782432915463,1.3342782432915463,1.3391465567149103],["IPW · observed propensity",3.3431000488505775,1.3431000488505775,1.3777361015642107]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",4.285556606329669,2.285556606329669,2.287390216115742],["OLS · adjust Z",3.3341805558152116,1.3341805558152116,1.336435734459825],["IPW · observed propensity",3.309240016100806,1.309240016100806,1.3241934658362227]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."},{"rows":[["Unadjusted difference",4.293882939666228,2.2938829396662284,2.2946010254150306],["OLS · adjust Z",3.3496322960550957,1.3496322960550957,1.3507613058562782],["IPW · observed propensity",3.3508413904069,1.3508413904069,1.3577670342116959]],"note":"The true effect is 2 for every unit. Adjustment only includes Z. Hidden U influences both treatment and outcome in the third scenario; more data cannot repair that missing identification assumption."}],"columns":["Estimator","Mean estimate","Bias","RMSE"],"context":"Both Z and hidden U cause A and Y. The estimator only sees Z; neither adjustment method has enough information.","readout":"80 repeated datasets per size. Bands describe uncertainty of the mean estimate across simulations (±1.96 SE), not the sampling interval for one fitted effect.","note":""}],"config":{"seed":20260928,"runs":80,"sizes":[200,500,1000,2000,5000],"trueATE":2},"method":["Binary Z and U are independent Bernoulli(0.5). Treatment uses either a coin flip, sigmoid(−1+2Z), or sigmoid(−2+2Z+2U). Noise is independent N(0,1). The structural outcome equation gives a constant effect of 2.","Unadjusted difference compares group means. OLS includes intercept, A and Z. Horvitz–Thompson IPW uses the known marginal P(A|Z) of the generator, not an estimated propensity; in the hidden case it averages over U.","Within a scenario, estimators share each generated dataset. Identification assumptions differ by scenario; numerical estimation cannot establish those assumptions from observations alone."],"limits":["This is a NumPy simulation, not a DoWhy/EconML package comparison, a discovery of a causal graph, or a real business effect estimate.","No treatment-effect heterogeneity, interference, missing data or positivity failure. Known propensities give IPW information usually unavailable in practice."],"references":[["DoWhy · Separate identification and estimation","https://www.pywhy.org/dowhy/v0.9.1/user_guide/effect_inference/identify.html"]],"provenance":{"python":"3.13.0","numpy":"2.4.6","sourceSha256":{"showcase_benchmark.py":"2a8f490e4f2baab03e71a7851d588ca08f2a886e4eafe850a4b566d821378ca9"}}}