{"title":"After the model ships.","description":"Train, validate, serve—and discover why a drift alert is not the same thing as a failing model. A complete small local prediction pipeline.","theme":"blue","kind":"MLOps","cases":[{"id":"covariate","name":"Input distribution shifts","title":"Healthy inputs do not guarantee healthy predictions.","unit":"Root mean squared error ↓","parameter":"Shift magnitude","x":[0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9,2.0],"series":[{"label":"Deployed ridge model","values":[0.4994851,0.4994962,0.4995091,0.4995239,0.4995406,0.4995591,0.4995795,0.4996018,0.499626,0.499652,0.49968,0.4997098,0.4997414,0.4997749,0.4998103,0.4998476,0.4998868,0.4999278,0.4999707,0.5000154,0.500062],"low":[0.4928347,0.4928422,0.4928515,0.4928628,0.4928759,0.4928909,0.4929077,0.4929264,0.492947,0.4929694,0.4929937,0.4930199,0.4930479,0.4930778,0.4931096,0.4931432,0.4931787,0.4932161,0.4932553,0.4932964,0.4933394],"high":[0.5061356,0.5061502,0.5061666,0.506185,0.5062053,0.5062274,0.5062514,0.5062773,0.506305,0.5063347,0.5063662,0.5063996,0.5064349,0.506472,0.5065111,0.506552,0.5065948,0.5066395,0.506686,0.5067344,0.5067847]},{"label":"Frozen mean baseline","values":[3.6663336,3.6925193,3.7426225,3.8157043,3.9104806,4.0254236,4.1588653,4.3090909,4.4744128,4.653224,4.8440322,5.0454772,5.2563366,5.475523,5.7020761,5.9351523,6.1740127,6.4180112,6.6665836,6.9192366,7.1755391],"low":[3.6271279,3.6531225,3.7027494,3.775118,3.8690059,3.9829492,4.1153374,4.264502,4.4287894,4.6066156,4.7965018,4.9970943,5.2071723,5.4256469,5.6515541,5.8840457,6.1223777,6.3658987,6.6140395,6.8663022,7.1222512],"high":[3.7055394,3.7319161,3.7824956,3.8562907,3.9519554,4.067898,4.2023932,4.3536798,4.5200361,4.6998323,4.8915625,5.0938601,5.3055008,5.5253991,5.7525981,5.9862589,6.2256477,6.4701238,6.7191277,6.9721711,7.2288271]}],"snapshots":[{"rows":[["Ridge RMSE",0.49948511577849686],["Constant baseline RMSE",3.6663336440921612],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49949616745375935],["Constant baseline RMSE",3.692519279715349],["Mean max standardized feature shift",0.1762959710030173],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.4995090936213131],["Constant baseline RMSE",3.7426225047721826],["Mean max standardized feature shift",0.27738485783617045],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.4995238941441856],["Constant baseline RMSE",3.8157043410696576],["Mean max standardized feature shift",0.3795398188894354],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49954056886436027],["Constant baseline RMSE",3.9104806474096674],["Mean max standardized feature shift",0.4816947799427004],["Share of batches triggering drift > 0.5",0.3333333333333333]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49955911760278354],["Constant baseline RMSE",4.025423577303614],["Mean max standardized feature shift",0.5838497409959654],["Share of batches triggering drift > 0.5",0.9666666666666667]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.499579540159376],["Constant baseline RMSE",4.158865301607911],["Mean max standardized feature shift",0.6860047020492305],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49960183631304134],["Constant baseline RMSE",4.309090904236219],["Mean max standardized feature shift",0.7881596631024954],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49962600582167976],["Constant baseline RMSE",4.474412764220707],["Mean max standardized feature shift",0.8903146241557605],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49965204842219973],["Constant baseline RMSE",4.653223977460666],["Mean max standardized feature shift",0.9924695852090254],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49967996383053326],["Constant baseline RMSE",4.8440321782900595],["Mean max standardized feature shift",1.0946245462622906],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.499709751741651],["Constant baseline RMSE",5.045477190770447],["Mean max standardized feature shift",1.1967795073155556],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.4997414118295786],["Constant baseline RMSE",5.2563365852025274],["Mean max standardized feature shift",1.2989344683688206],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49977494374741466],["Constant baseline RMSE",5.475522960790161],["Mean max standardized feature shift",1.401089429422086],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49981034712734956],["Constant baseline RMSE",5.70207609382463],["Mean max standardized feature shift",1.5032443904753507],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.4998476215806858],["Constant baseline RMSE",5.9351523049733235],["Mean max standardized feature shift",1.6053993515286153],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49988676669785864],["Constant baseline RMSE",6.174012684375439],["Mean max standardized feature shift",1.7075543125818806],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.4999277820484587],["Constant baseline RMSE",6.418011238891693],["Mean max standardized feature shift",1.809709273635146],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.49997066718125666],["Constant baseline RMSE",6.6665836006294334],["Mean max standardized feature shift",1.9118642346884107],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.5000154216242255],["Constant baseline RMSE",6.919236640027781],["Mean max standardized feature shift",2.014019195741676],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.5000620448845693],["Constant baseline RMSE",7.175539131815735],["Mean max standardized feature shift",2.1161741567949406],["Share of batches triggering drift > 0.5",1.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."}],"columns":["Measurement","Value"],"context":"A ridge model is trained once, selected on validation data and evaluated on a separate test split. These charts use fresh synthetic monitoring batches.","readout":"30 monitoring seeds per shift. Bands are pointwise ±1.96 SE of average batch RMSE. Drift uses only feature means; RMSE needs outcome labels.","note":""},{"id":"concept","name":"Outcome relationship shifts","title":"Healthy inputs do not guarantee healthy predictions.","unit":"Root mean squared error ↓","parameter":"Shift magnitude","x":[0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1,1.2,1.3,1.4,1.5,1.6,1.7,1.8,1.9,2.0],"series":[{"label":"Deployed ridge model","values":[0.4994851,0.5112342,0.5416481,0.5878286,0.6463984,0.714313,0.7891623,0.8691565,0.9530013,1.0397659,1.1287776,1.2195446,1.3117028,1.4049786,1.4991634,1.5940963,1.6896511,1.785728,1.8822473,1.979144,2.0763655],"low":[0.4928347,0.5043842,0.5346072,0.5805181,0.6386708,0.7060029,0.7801233,0.8592744,0.9421926,1.0279709,1.1159541,1.2056624,1.2967401,1.3889193,1.4819958,1.5758112,1.6702417,1.765189,1.8605743,1.9563338,2.0524152],"high":[0.5061356,0.5180843,0.5486889,0.5951391,0.6541261,0.7226232,0.7982014,0.8790386,0.9638101,1.0515609,1.1416011,1.2334269,1.3266656,1.4210378,1.5163311,1.6123813,1.7090604,1.8062671,1.9039202,2.0019543,2.1003159]},{"label":"Frozen mean baseline","values":[3.6663336,3.7486425,3.8318093,3.9157793,4.000502,4.0859307,4.1720219,4.2587356,4.3460344,4.4338839,4.522252,4.6111088,4.7004267,4.79018,4.8803445,4.9708979,5.0618194,5.1530894,5.2446899,5.3366037,5.428815],"low":[3.6271279,3.7084218,3.7905606,3.8734911,3.957164,4.0415337,4.1265578,4.212197,4.2984149,4.3851777,4.4724537,4.5602139,4.6484308,4.7370793,4.8261356,4.9155777,5.005385,5.0955383,5.1860197,5.2768124,5.3679006],"high":[3.7055394,3.7888633,3.873058,3.9580676,4.0438401,4.1303277,4.217486,4.3052741,4.3936539,4.4825901,4.5720502,4.6620038,4.7524227,4.8432806,4.9345534,5.0262181,5.1182538,5.2106405,5.30336,5.3963951,5.4897295]}],"snapshots":[{"rows":[["Ridge RMSE",0.49948511577849686],["Constant baseline RMSE",3.6663336440921612],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.5112342085836971],["Constant baseline RMSE",3.7486425321507553],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.5416480528853904],["Constant baseline RMSE",3.831809294413348],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.5878285707412179],["Constant baseline RMSE",3.915779310279566],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.6463984337454818],["Constant baseline RMSE",4.000502035333213],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.7143130225255033],["Constant baseline RMSE",4.08593067916517],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.7891623065132976],["Constant baseline RMSE",4.1720219063955595],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.869156527743512],["Constant baseline RMSE",4.258735560129663],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",0.9530013127026706],["Constant baseline RMSE",4.34603440683696],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.0397659397033137],["Constant baseline RMSE",4.433883901489199],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.1287776069248323],["Constant baseline RMSE",4.522251971709597],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.2195446281324316],["Constant baseline RMSE",4.611108819652559],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.3117028237462884],["Constant baseline RMSE",4.700426740338031],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.4049785847123508],["Constant baseline RMSE",4.790179955195624],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.4991634137951402],["Constant baseline RMSE",4.880344459622527],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.594096250021609],["Constant baseline RMSE",4.970897883419865],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.6896510645251073],["Constant baseline RMSE",5.061819363039434],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.78572804765732],["Constant baseline RMSE",5.15308942464344],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.8822472662812646],["Constant baseline RMSE",5.244689877051152],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",1.9791440396319535],["Constant baseline RMSE",5.336603713716733],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."},{"rows":[["Ridge RMSE",2.076365525498136],["Constant baseline RMSE",5.428815022950261],["Mean max standardized feature shift",0.0955111626831038],["Share of batches triggering drift > 0.5",0.0]],"note":"An input-distribution alert is not a model-quality verdict. Covariate shift can be harmless to this correctly specified linear model; concept shift can worsen error without changing the observed feature distribution."}],"columns":["Measurement","Value"],"context":"A ridge model is trained once, selected on validation data and evaluated on a separate test split. These charts use fresh synthetic monitoring batches.","readout":"30 monitoring seeds per shift. Bands are pointwise ±1.96 SE of average batch RMSE. Drift uses only feature means; RMSE needs outcome labels.","note":""}],"config":{"training":{"seeds":{"train":10,"validation":11,"test":12},"sizes":[1000,300,300],"validationRMSE":[0.5069998596256045,0.5070043615834015,0.5070585202374001,0.5089200853618837,0.6231278863031755],"testRMSE":0.5266189850306064,"baselineTestRMSE":3.4534222794254976},"penalty":0.0,"runs":30,"monitoringSeed":100},"method":["Synthetic y = 1 + 3x₁ − 2x₂ + 0.5x₃ + N(0,0.5²). Fit preprocessing on training data only. Pick ridge strength using validation RMSE; report test RMSE once. Seeds and split sizes are recorded.","Covariate shift adds to the mean of x₁ while preserving the outcome equation. Concept shift changes the coefficient of x₁ while preserving the input distribution. All other generator settings are unchanged.","local_demo/pipeline.py exports a portable model.json. local_demo/serve.py exposes loopback /health and /predict with a bounded three-signal input contract. The static page shows measured benchmark results; it does not call the server."],"limits":["This is a NumPy/stdlib local vertical slice, not an MLflow/DVC/FastAPI deployment or a production monitoring service.","The HTTP server has no authentication/TLS and must stay on loopback. Drift thresholds are illustrative, not statistically calibrated; mean-only monitoring misses many distribution changes."],"references":[["scikit-learn · Avoid data leakage","https://scikit-learn.org/1.8/common_pitfalls.html"]],"provenance":{"python":"3.13.0","numpy":"2.4.6","sourceSha256":{"showcase_benchmark.py":"e52a6661429db545ac1f0541ab74ab092faa814aef157077c1195ce0a48f1380","local_demo/pipeline.py":"dc325850697df23392370620d0c2077f8f06b4a943f8ad767781baae62c43e9d","local_demo/serve.py":"a25f76d62092e5175b7f910caee4dcf73c7faa8e8710515f93118ceeb2a1629d"}}}