{"title":"The long way home.","description":"An agent can know the shortest route and still choose a safer one. Explore planning, learning and the cost of a wrong step.","theme":"violet","kind":"Reinforcement learning","cases":[{"id":"slip-0","name":"0% random-action slip","title":"Plan a route. Account for risk.","unit":"Max value error vs converged solution ↓","parameter":"Bellman iteration","x":[0,1,2,3,4,5,6,7,8],"series":[{"label":"Value error","values":[6.0332541,5.0332541,4.0832541,3.1807541,2.3233791,1.5088728,0.7350919,0.0,0.0]}],"snapshots":[{"board":[{"label":"S","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-2.9,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-3.7,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-3.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-3.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-4.5,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-5.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-6.0,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-6.0,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-7.0,1.0,0.0,0.0],["QLearning",-7.0,1.0,0.0,0.0],["SARSA",-7.0,1.0,0.0,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."}],"columns":["Policy","Mean return ↑","Goal rate","Cliff rate","Timeout rate"],"caption":"Final policy evaluation · 200 held-out episodes · not tied to iteration slider","context":"A 5×5 world. An action is replaced by a uniformly random action with probability 0%. Cliffs end an episode at −100.","readout":"Each ordinary move costs −1; reaching the goal gives 0 and ends the episode. γ=0.95. Move the slider to see values propagate across the grid.","note":""},{"id":"slip-15","name":"15% random-action slip","title":"Plan a route. Account for risk.","unit":"Max value error vs converged solution ↓","parameter":"Bellman iteration","x":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29],"series":[{"label":"Value error","values":[16.391501,12.4964662,11.2819506,10.1354107,9.0490675,8.0267387,7.0640545,6.1582594,5.3057009,4.5031707,3.7475297,3.0358737,2.3687362,1.7517002,1.1834189,0.6620973,0.184518,0.0461088,0.0149854,0.0052101,0.0019226,0.0007451,0.0002949,0.0001206,4.95e-05,2.05e-05,8.3e-06,3.2e-06,9e-07,0.0]}],"snapshots":[{"board":[{"label":"S","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-1.0,"kind":""},{"label":"→","value":-4.7,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-4.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-4.7,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-4.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-4.7,"kind":""},{"label":"→","value":-4.7,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-4.7,"kind":""},{"label":"↓","value":-4.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-3.8,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-3.8,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-2.2,"kind":""},{"label":"→","value":-5.8,"kind":""},{"label":"→","value":-2.2,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-5.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-5.8,"kind":""},{"label":"→","value":-2.2,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-5.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-9.0,"kind":""},{"label":"→","value":-5.8,"kind":""},{"label":"↓","value":-2.2,"kind":""},{"label":"↓","value":-2.2,"kind":""},{"label":"↓","value":-5.8,"kind":""},{"label":"↓","value":-5.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.0,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"→","value":-2.1,"kind":""},{"label":"→","value":-2.2,"kind":""},{"label":"→","value":-4.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-3.4,"kind":""},{"label":"→","value":-6.9,"kind":""},{"label":"→","value":-3.1,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-7.0,"kind":""},{"label":"→","value":-3.1,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-10.0,"kind":""},{"label":"→","value":-7.0,"kind":""},{"label":"↓","value":-3.1,"kind":""},{"label":"↓","value":-3.1,"kind":""},{"label":"↓","value":-6.8,"kind":""},{"label":"↓","value":-7.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.0,"kind":""},{"label":"→","value":-2.9,"kind":""},{"label":"→","value":-3.0,"kind":""},{"label":"→","value":-3.2,"kind":""},{"label":"→","value":-4.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-4.4,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"→","value":-4.0,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"↓","value":-3.7,"kind":""},{"label":"↓","value":-8.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-7.8,"kind":""},{"label":"→","value":-4.0,"kind":""},{"label":"↓","value":-3.7,"kind":""},{"label":"↓","value":-7.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.1,"kind":""},{"label":"→","value":-7.8,"kind":""},{"label":"↓","value":-3.9,"kind":""},{"label":"↓","value":-4.0,"kind":""},{"label":"↓","value":-7.6,"kind":""},{"label":"↓","value":-8.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.1,"kind":""},{"label":"→","value":-3.7,"kind":""},{"label":"→","value":-3.9,"kind":""},{"label":"→","value":-4.0,"kind":""},{"label":"→","value":-4.1,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-5.5,"kind":""},{"label":"→","value":-8.5,"kind":""},{"label":"→","value":-4.8,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"↓","value":-9.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-8.6,"kind":""},{"label":"→","value":-4.8,"kind":""},{"label":"↓","value":-4.5,"kind":""},{"label":"↓","value":-8.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.8,"kind":""},{"label":"→","value":-8.6,"kind":""},{"label":"↓","value":-4.7,"kind":""},{"label":"↓","value":-4.8,"kind":""},{"label":"↓","value":-8.4,"kind":""},{"label":"↓","value":-8.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.1,"kind":""},{"label":"→","value":-4.6,"kind":""},{"label":"→","value":-4.7,"kind":""},{"label":"→","value":-4.8,"kind":""},{"label":"→","value":-4.1,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-6.4,"kind":""},{"label":"→","value":-9.3,"kind":""},{"label":"→","value":-5.6,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-9.4,"kind":""},{"label":"→","value":-5.6,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-9.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-12.5,"kind":""},{"label":"→","value":-9.3,"kind":""},{"label":"↓","value":-5.1,"kind":""},{"label":"↓","value":-5.6,"kind":""},{"label":"↓","value":-9.2,"kind":""},{"label":"↓","value":-9.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.1,"kind":""},{"label":"→","value":-5.4,"kind":""},{"label":"→","value":-5.5,"kind":""},{"label":"→","value":-5.1,"kind":""},{"label":"→","value":-4.1,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-7.3,"kind":""},{"label":"→","value":-10.0,"kind":""},{"label":"→","value":-6.4,"kind":""},{"label":"→","value":-6.1,"kind":""},{"label":"↓","value":-6.0,"kind":""},{"label":"↓","value":-10.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-10.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"↓","value":-5.9,"kind":""},{"label":"↓","value":-10.2,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.2,"kind":""},{"label":"→","value":-9.7,"kind":""},{"label":"↓","value":-5.2,"kind":""},{"label":"↓","value":-6.4,"kind":""},{"label":"↓","value":-9.9,"kind":""},{"label":"↓","value":-9.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-6.1,"kind":""},{"label":"→","value":-6.0,"kind":""},{"label":"→","value":-5.2,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-8.2,"kind":""},{"label":"→","value":-10.7,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.8,"kind":""},{"label":"↓","value":-6.6,"kind":""},{"label":"↓","value":-11.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-10.8,"kind":""},{"label":"→","value":-6.9,"kind":""},{"label":"↓","value":-6.0,"kind":""},{"label":"↓","value":-10.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.6,"kind":""},{"label":"→","value":-9.8,"kind":""},{"label":"↓","value":-5.2,"kind":""},{"label":"↓","value":-7.1,"kind":""},{"label":"↓","value":-10.4,"kind":""},{"label":"↓","value":-9.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-6.7,"kind":""},{"label":"→","value":-6.2,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-9.0,"kind":""},{"label":"→","value":-11.3,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"→","value":-7.3,"kind":""},{"label":"↓","value":-6.8,"kind":""},{"label":"↓","value":-12.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"↓","value":-6.1,"kind":""},{"label":"↓","value":-11.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.7,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-7.7,"kind":""},{"label":"↓","value":-10.5,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-6.9,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-9.7,"kind":""},{"label":"→","value":-11.9,"kind":""},{"label":"→","value":-8.2,"kind":""},{"label":"→","value":-7.5,"kind":""},{"label":"↓","value":-6.9,"kind":""},{"label":"↓","value":-13.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.4,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.1,"kind":""},{"label":"↓","value":-12.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-7.9,"kind":""},{"label":"↓","value":-10.6,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.0,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-10.5,"kind":""},{"label":"→","value":-12.4,"kind":""},{"label":"→","value":-8.5,"kind":""},{"label":"→","value":-7.6,"kind":""},{"label":"↓","value":-6.9,"kind":""},{"label":"↓","value":-13.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.5,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.0,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-11.1,"kind":""},{"label":"→","value":-12.7,"kind":""},{"label":"→","value":-8.6,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-14.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-11.7,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-8.6,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-15.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-12.3,"kind":""},{"label":"→","value":-12.9,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-15.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-12.8,"kind":""},{"label":"→","value":-12.9,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.3,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-13.5,"kind":""},{"label":"→","value":-13.0,"kind":""},{"label":"→","value":-8.7,"kind":""},{"label":"→","value":-7.7,"kind":""},{"label":"↓","value":-7.0,"kind":""},{"label":"↓","value":-16.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-7.2,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-5.3,"kind":""},{"label":"↓","value":-8.1,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↓","value":-10.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-4.2,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-6.3,"kind":""},{"label":"→","value":-5.3,"kind":""},{"label":"→","value":-4.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-16.925,0.91,0.09,0.0],["QLearning",-19.825,0.885,0.115,0.0],["SARSA",-17.615,0.905,0.095,0.0]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."}],"columns":["Policy","Mean return ↑","Goal rate","Cliff rate","Timeout rate"],"caption":"Final policy evaluation · 200 held-out episodes · not tied to iteration slider","context":"A 5×5 world. An action is replaced by a uniformly random action with probability 15%. Cliffs end an episode at −100.","readout":"Each ordinary move costs −1; reaching the goal gives 0 and ends the episode. γ=0.95. Move the slider to see values propagate across the grid.","note":""},{"id":"slip-35","name":"35% random-action slip","title":"Plan a route. Account for risk.","unit":"Max value error vs converged solution ↓","parameter":"Bellman iteration","x":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56],"series":[{"label":"Value error","values":[32.9190865,25.1451533,22.7550126,20.5580544,18.4984248,16.6330621,14.933707,13.4001606,12.010911,10.7546667,9.6158421,8.5826685,7.6433823,6.7881029,6.0077906,5.2945965,4.6415199,4.0424118,3.4918319,2.9849966,2.5176525,2.0861771,1.6876274,1.3578607,1.0631584,0.7940713,0.550803,0.3457684,0.2271105,0.1475009,0.0965954,0.0630472,0.0412434,0.0269379,0.0176096,0.0115014,0.0075152,0.0049078,0.0032057,0.002093,0.0013666,0.000892,0.000582,0.0003796,0.0002474,0.000161,0.0001046,6.78e-05,4.38e-05,2.81e-05,1.78e-05,1.11e-05,6.7e-06,3.9e-06,2e-06,8e-07,0.0]}],"snapshots":[{"board":[{"label":"S","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↑","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"↓","value":0.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"→","value":0.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-1.0,"kind":""},{"label":"→","value":-9.7,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↑","value":-9.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-9.7,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-9.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-9.7,"kind":""},{"label":"→","value":-9.7,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-1.0,"kind":""},{"label":"↓","value":-9.7,"kind":""},{"label":"↓","value":-9.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-8.9,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-1.0,"kind":""},{"label":"→","value":-8.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-3.4,"kind":""},{"label":"→","value":-11.2,"kind":""},{"label":"→","value":-3.4,"kind":""},{"label":"→","value":-1.9,"kind":""},{"label":"↓","value":-2.0,"kind":""},{"label":"↑","value":-12.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-11.2,"kind":""},{"label":"→","value":-3.4,"kind":""},{"label":"↓","value":-1.9,"kind":""},{"label":"↓","value":-12.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-18.0,"kind":""},{"label":"→","value":-11.2,"kind":""},{"label":"↓","value":-3.3,"kind":""},{"label":"↓","value":-3.4,"kind":""},{"label":"↓","value":-11.2,"kind":""},{"label":"↓","value":-12.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-9.8,"kind":""},{"label":"→","value":-2.0,"kind":""},{"label":"→","value":-2.7,"kind":""},{"label":"→","value":-3.3,"kind":""},{"label":"→","value":-9.8,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-5.6,"kind":""},{"label":"→","value":-13.3,"kind":""},{"label":"→","value":-4.5,"kind":""},{"label":"→","value":-3.1,"kind":""},{"label":"↓","value":-2.9,"kind":""},{"label":"↑","value":-14.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-4.4,"kind":""},{"label":"↓","value":-3.1,"kind":""},{"label":"↓","value":-14.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-19.5,"kind":""},{"label":"→","value":-13.8,"kind":""},{"label":"↓","value":-4.4,"kind":""},{"label":"↓","value":-4.6,"kind":""},{"label":"↓","value":-12.8,"kind":""},{"label":"↓","value":-14.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.0,"kind":""},{"label":"→","value":-3.0,"kind":""},{"label":"→","value":-3.8,"kind":""},{"label":"→","value":-5.0,"kind":""},{"label":"→","value":-10.0,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-7.6,"kind":""},{"label":"→","value":-14.4,"kind":""},{"label":"→","value":-5.8,"kind":""},{"label":"→","value":-4.0,"kind":""},{"label":"↓","value":-3.7,"kind":""},{"label":"↑","value":-15.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-14.7,"kind":""},{"label":"→","value":-5.7,"kind":""},{"label":"↓","value":-4.0,"kind":""},{"label":"↓","value":-15.2,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-21.7,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-5.5,"kind":""},{"label":"↓","value":-5.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↓","value":-15.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.1,"kind":""},{"label":"→","value":-4.1,"kind":""},{"label":"→","value":-4.9,"kind":""},{"label":"→","value":-6.1,"kind":""},{"label":"→","value":-10.2,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-9.5,"kind":""},{"label":"→","value":-15.6,"kind":""},{"label":"→","value":-6.7,"kind":""},{"label":"→","value":-4.9,"kind":""},{"label":"↓","value":-4.6,"kind":""},{"label":"↑","value":-17.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-15.9,"kind":""},{"label":"→","value":-6.6,"kind":""},{"label":"↓","value":-4.9,"kind":""},{"label":"↓","value":-16.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-22.5,"kind":""},{"label":"→","value":-15.8,"kind":""},{"label":"↓","value":-6.3,"kind":""},{"label":"↓","value":-6.7,"kind":""},{"label":"↓","value":-14.9,"kind":""},{"label":"↓","value":-16.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.2,"kind":""},{"label":"→","value":-5.1,"kind":""},{"label":"→","value":-5.9,"kind":""},{"label":"→","value":-7.1,"kind":""},{"label":"→","value":-10.3,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-11.2,"kind":""},{"label":"→","value":-16.4,"kind":""},{"label":"→","value":-7.6,"kind":""},{"label":"→","value":-5.7,"kind":""},{"label":"↓","value":-5.4,"kind":""},{"label":"↑","value":-19.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-16.7,"kind":""},{"label":"→","value":-7.5,"kind":""},{"label":"↓","value":-5.7,"kind":""},{"label":"↓","value":-17.2,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-23.5,"kind":""},{"label":"→","value":-16.5,"kind":""},{"label":"↓","value":-7.1,"kind":""},{"label":"↓","value":-7.7,"kind":""},{"label":"↓","value":-15.8,"kind":""},{"label":"↓","value":-17.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.3,"kind":""},{"label":"→","value":-6.0,"kind":""},{"label":"→","value":-6.9,"kind":""},{"label":"→","value":-8.0,"kind":""},{"label":"→","value":-10.4,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-12.7,"kind":""},{"label":"→","value":-17.3,"kind":""},{"label":"→","value":-8.4,"kind":""},{"label":"→","value":-6.5,"kind":""},{"label":"↓","value":-6.2,"kind":""},{"label":"↑","value":-20.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-17.5,"kind":""},{"label":"→","value":-8.2,"kind":""},{"label":"↓","value":-6.5,"kind":""},{"label":"↓","value":-18.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.1,"kind":""},{"label":"→","value":-17.2,"kind":""},{"label":"↓","value":-7.8,"kind":""},{"label":"↓","value":-8.6,"kind":""},{"label":"↓","value":-16.6,"kind":""},{"label":"↓","value":-18.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.4,"kind":""},{"label":"→","value":-6.9,"kind":""},{"label":"→","value":-7.8,"kind":""},{"label":"→","value":-8.8,"kind":""},{"label":"→","value":-10.5,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-14.1,"kind":""},{"label":"→","value":-18.1,"kind":""},{"label":"→","value":-9.2,"kind":""},{"label":"→","value":-7.3,"kind":""},{"label":"↓","value":-6.9,"kind":""},{"label":"↑","value":-21.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-18.1,"kind":""},{"label":"→","value":-9.0,"kind":""},{"label":"↓","value":-7.2,"kind":""},{"label":"↓","value":-18.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.7,"kind":""},{"label":"→","value":-17.8,"kind":""},{"label":"↓","value":-8.5,"kind":""},{"label":"↓","value":-9.5,"kind":""},{"label":"↓","value":-17.4,"kind":""},{"label":"↓","value":-19.2,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.4,"kind":""},{"label":"→","value":-7.8,"kind":""},{"label":"→","value":-8.6,"kind":""},{"label":"→","value":-9.6,"kind":""},{"label":"→","value":-10.5,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-15.4,"kind":""},{"label":"→","value":-18.8,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"→","value":-8.0,"kind":""},{"label":"↓","value":-7.6,"kind":""},{"label":"↑","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-18.8,"kind":""},{"label":"→","value":-9.6,"kind":""},{"label":"↓","value":-7.9,"kind":""},{"label":"↓","value":-19.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-25.3,"kind":""},{"label":"→","value":-18.4,"kind":""},{"label":"↓","value":-9.1,"kind":""},{"label":"↓","value":-10.3,"kind":""},{"label":"↓","value":-18.1,"kind":""},{"label":"↓","value":-19.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.5,"kind":""},{"label":"→","value":-8.6,"kind":""},{"label":"→","value":-9.4,"kind":""},{"label":"→","value":-10.3,"kind":""},{"label":"→","value":-10.6,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-16.5,"kind":""},{"label":"→","value":-19.4,"kind":""},{"label":"→","value":-10.5,"kind":""},{"label":"→","value":-8.6,"kind":""},{"label":"↓","value":-8.3,"kind":""},{"label":"↑","value":-24.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-19.3,"kind":""},{"label":"→","value":-10.3,"kind":""},{"label":"↓","value":-8.6,"kind":""},{"label":"↓","value":-20.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-18.9,"kind":""},{"label":"↓","value":-9.7,"kind":""},{"label":"↓","value":-11.0,"kind":""},{"label":"↓","value":-18.8,"kind":""},{"label":"↓","value":-20.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.6,"kind":""},{"label":"→","value":-9.4,"kind":""},{"label":"→","value":-10.2,"kind":""},{"label":"→","value":-11.0,"kind":""},{"label":"→","value":-10.7,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-17.6,"kind":""},{"label":"→","value":-20.0,"kind":""},{"label":"→","value":-11.2,"kind":""},{"label":"→","value":-9.3,"kind":""},{"label":"↓","value":-8.9,"kind":""},{"label":"↑","value":-24.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-19.9,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"↓","value":-9.2,"kind":""},{"label":"↓","value":-21.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-26.2,"kind":""},{"label":"→","value":-19.5,"kind":""},{"label":"↓","value":-10.3,"kind":""},{"label":"↓","value":-11.8,"kind":""},{"label":"↓","value":-19.4,"kind":""},{"label":"↓","value":-21.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.6,"kind":""},{"label":"→","value":-10.1,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-10.7,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-18.5,"kind":""},{"label":"→","value":-20.6,"kind":""},{"label":"→","value":-11.7,"kind":""},{"label":"→","value":-9.9,"kind":""},{"label":"↓","value":-9.5,"kind":""},{"label":"↑","value":-25.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-20.4,"kind":""},{"label":"→","value":-11.5,"kind":""},{"label":"↓","value":-9.8,"kind":""},{"label":"↓","value":-21.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-26.7,"kind":""},{"label":"→","value":-19.9,"kind":""},{"label":"↓","value":-10.8,"kind":""},{"label":"↓","value":-12.4,"kind":""},{"label":"↓","value":-20.0,"kind":""},{"label":"↓","value":-21.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.7,"kind":""},{"label":"→","value":-10.8,"kind":""},{"label":"→","value":-11.6,"kind":""},{"label":"→","value":-12.1,"kind":""},{"label":"→","value":-10.8,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-19.4,"kind":""},{"label":"→","value":-21.1,"kind":""},{"label":"→","value":-12.3,"kind":""},{"label":"→","value":-10.4,"kind":""},{"label":"↓","value":-10.1,"kind":""},{"label":"↑","value":-26.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-20.9,"kind":""},{"label":"→","value":-12.0,"kind":""},{"label":"↓","value":-10.4,"kind":""},{"label":"↓","value":-22.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-27.1,"kind":""},{"label":"→","value":-20.4,"kind":""},{"label":"↓","value":-11.3,"kind":""},{"label":"↓","value":-13.1,"kind":""},{"label":"↓","value":-20.6,"kind":""},{"label":"↓","value":-22.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.7,"kind":""},{"label":"→","value":-11.5,"kind":""},{"label":"→","value":-12.2,"kind":""},{"label":"→","value":-12.3,"kind":""},{"label":"→","value":-10.8,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-20.1,"kind":""},{"label":"→","value":-21.6,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-11.0,"kind":""},{"label":"↓","value":-10.7,"kind":""},{"label":"↑","value":-27.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-21.4,"kind":""},{"label":"→","value":-12.6,"kind":""},{"label":"↓","value":-10.9,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-27.5,"kind":""},{"label":"→","value":-20.8,"kind":""},{"label":"↓","value":-11.8,"kind":""},{"label":"↓","value":-13.7,"kind":""},{"label":"↓","value":-21.1,"kind":""},{"label":"↓","value":-22.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.8,"kind":""},{"label":"→","value":-12.1,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-12.5,"kind":""},{"label":"→","value":-10.8,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-20.9,"kind":""},{"label":"→","value":-22.1,"kind":""},{"label":"→","value":-13.3,"kind":""},{"label":"→","value":-11.5,"kind":""},{"label":"↓","value":-11.2,"kind":""},{"label":"↑","value":-27.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-21.8,"kind":""},{"label":"→","value":-13.1,"kind":""},{"label":"↓","value":-11.4,"kind":""},{"label":"↓","value":-23.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-27.9,"kind":""},{"label":"→","value":-21.2,"kind":""},{"label":"↓","value":-12.2,"kind":""},{"label":"↓","value":-14.3,"kind":""},{"label":"↓","value":-21.6,"kind":""},{"label":"↓","value":-22.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.8,"kind":""},{"label":"→","value":-12.7,"kind":""},{"label":"→","value":-13.3,"kind":""},{"label":"→","value":-12.6,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-21.5,"kind":""},{"label":"→","value":-22.6,"kind":""},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-12.0,"kind":""},{"label":"↓","value":-11.7,"kind":""},{"label":"↑","value":-28.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-22.2,"kind":""},{"label":"→","value":-13.5,"kind":""},{"label":"↓","value":-11.9,"kind":""},{"label":"↓","value":-23.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-28.2,"kind":""},{"label":"→","value":-21.6,"kind":""},{"label":"↓","value":-12.3,"kind":""},{"label":"↓","value":-14.8,"kind":""},{"label":"↓","value":-22.1,"kind":""},{"label":"↓","value":-22.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.8,"kind":""},{"label":"→","value":-13.2,"kind":""},{"label":"→","value":-13.8,"kind":""},{"label":"→","value":-12.6,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-22.1,"kind":""},{"label":"→","value":-23.0,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.4,"kind":""},{"label":"↓","value":-12.1,"kind":""},{"label":"↑","value":-29.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-22.6,"kind":""},{"label":"→","value":-14.0,"kind":""},{"label":"↓","value":-12.3,"kind":""},{"label":"↓","value":-24.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-28.5,"kind":""},{"label":"→","value":-21.7,"kind":""},{"label":"↓","value":-12.4,"kind":""},{"label":"↓","value":-15.3,"kind":""},{"label":"↓","value":-22.4,"kind":""},{"label":"↓","value":-22.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.8,"kind":""},{"label":"→","value":-13.7,"kind":""},{"label":"→","value":-13.9,"kind":""},{"label":"→","value":-12.7,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-22.7,"kind":""},{"label":"→","value":-23.4,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"→","value":-12.9,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↑","value":-29.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-23.0,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"↓","value":-12.7,"kind":""},{"label":"↓","value":-24.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-28.7,"kind":""},{"label":"→","value":-21.9,"kind":""},{"label":"↓","value":-12.5,"kind":""},{"label":"↓","value":-15.8,"kind":""},{"label":"↓","value":-22.6,"kind":""},{"label":"↓","value":-22.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.2,"kind":""},{"label":"→","value":-14.1,"kind":""},{"label":"→","value":-12.7,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-23.2,"kind":""},{"label":"→","value":-23.8,"kind":""},{"label":"→","value":-15.1,"kind":""},{"label":"→","value":-13.3,"kind":""},{"label":"↓","value":-13.0,"kind":""},{"label":"↑","value":-30.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-23.3,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.0,"kind":""},{"label":"↓","value":-25.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-28.8,"kind":""},{"label":"→","value":-22.0,"kind":""},{"label":"↓","value":-12.5,"kind":""},{"label":"↓","value":-16.2,"kind":""},{"label":"↓","value":-22.7,"kind":""},{"label":"↓","value":-22.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.5,"kind":""},{"label":"→","value":-14.1,"kind":""},{"label":"→","value":-12.7,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-23.6,"kind":""},{"label":"→","value":-24.1,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"→","value":-13.7,"kind":""},{"label":"↓","value":-13.3,"kind":""},{"label":"↑","value":-30.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-23.6,"kind":""},{"label":"→","value":-15.0,"kind":""},{"label":"↓","value":-13.2,"kind":""},{"label":"↓","value":-25.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-28.9,"kind":""},{"label":"→","value":-22.0,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.5,"kind":""},{"label":"↓","value":-22.8,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"→","value":-14.2,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-24.1,"kind":""},{"label":"→","value":-24.5,"kind":""},{"label":"→","value":-15.8,"kind":""},{"label":"→","value":-14.0,"kind":""},{"label":"↓","value":-13.6,"kind":""},{"label":"↑","value":-30.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-23.9,"kind":""},{"label":"→","value":-15.1,"kind":""},{"label":"↓","value":-13.2,"kind":""},{"label":"↓","value":-25.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.0,"kind":""},{"label":"→","value":-22.1,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.7,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.8,"kind":""},{"label":"→","value":-14.2,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-24.5,"kind":""},{"label":"→","value":-24.8,"kind":""},{"label":"→","value":-16.2,"kind":""},{"label":"→","value":-14.2,"kind":""},{"label":"↓","value":-13.7,"kind":""},{"label":"↑","value":-31.2,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.0,"kind":""},{"label":"→","value":-15.3,"kind":""},{"label":"↓","value":-13.3,"kind":""},{"label":"↓","value":-26.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.0,"kind":""},{"label":"→","value":-22.1,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.8,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.8,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-24.8,"kind":""},{"label":"→","value":-25.1,"kind":""},{"label":"→","value":-16.4,"kind":""},{"label":"→","value":-14.4,"kind":""},{"label":"↓","value":-13.8,"kind":""},{"label":"↑","value":-31.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.1,"kind":""},{"label":"→","value":-15.3,"kind":""},{"label":"↓","value":-13.3,"kind":""},{"label":"↓","value":-26.2,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.1,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.8,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.8,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-25.2,"kind":""},{"label":"→","value":-25.3,"kind":""},{"label":"→","value":-16.5,"kind":""},{"label":"→","value":-14.5,"kind":""},{"label":"↓","value":-13.8,"kind":""},{"label":"↑","value":-31.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.2,"kind":""},{"label":"→","value":-15.4,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.1,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.9,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-22.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-25.5,"kind":""},{"label":"→","value":-25.4,"kind":""},{"label":"→","value":-16.6,"kind":""},{"label":"→","value":-14.6,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.1,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.2,"kind":""},{"label":"→","value":-15.4,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.3,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.1,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.9,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-25.7,"kind":""},{"label":"→","value":-25.6,"kind":""},{"label":"→","value":-16.7,"kind":""},{"label":"→","value":-14.6,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.4,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.1,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.9,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-25.9,"kind":""},{"label":"→","value":-25.6,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.6,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.6,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-16.9,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.0,"kind":""},{"label":"→","value":-25.7,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.6,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.7,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.4,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.0,"kind":""},{"label":"→","value":-25.7,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.6,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.7,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.8,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."},{"board":[{"label":"S","value":-26.1,"kind":""},{"label":"→","value":-25.8,"kind":""},{"label":"→","value":-16.8,"kind":""},{"label":"→","value":-14.7,"kind":""},{"label":"↓","value":-13.9,"kind":""},{"label":"↑","value":-32.9,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-24.3,"kind":""},{"label":"→","value":-15.5,"kind":""},{"label":"↓","value":-13.4,"kind":""},{"label":"↓","value":-26.5,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"→","value":-29.2,"kind":""},{"label":"→","value":-22.2,"kind":""},{"label":"↓","value":-12.6,"kind":""},{"label":"↓","value":-17.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"↓","value":-23.0,"kind":""},{"label":"×","value":0.0,"kind":"cliff"},{"label":"↓","value":-10.9,"kind":""},{"label":"→","value":-14.9,"kind":""},{"label":"→","value":-14.3,"kind":""},{"label":"→","value":-12.8,"kind":""},{"label":"→","value":-10.9,"kind":""},{"label":"G","value":0.0,"kind":"goal"}],"rows":[["Value iteration",-33.01,0.765,0.235,0.0],["QLearning",-141.785,0.05,0.63,0.32],["SARSA",-105.255,0.4,0.55,0.05]],"note":"Tile values follow the selected Bellman iteration. Arrows show the final converged policy. The table always reports final greedy policies on 200 fresh episodes; Q-learning/SARSA each use one 500-episode training seed."}],"columns":["Policy","Mean return ↑","Goal rate","Cliff rate","Timeout rate"],"caption":"Final policy evaluation · 200 held-out episodes · not tied to iteration slider","context":"A 5×5 world. An action is replaced by a uniformly random action with probability 35%. Cliffs end an episode at −100.","readout":"Each ordinary move costs −1; reaching the goal gives 0 and ends the episode. γ=0.95. Move the slider to see values propagate across the grid.","note":""}],"config":{"seed":20260927,"trainingEpisodes":500,"evaluationEpisodes":200,"stepCap":150,"gamma":0.95},"method":["Value iteration uses the exact transition model: intended action probability 1−p+p/4, other actions p/4. Terminal cliff and goal states have zero continuation value.","Q-learning and SARSA reuse rl_utils with α=0.1, γ=0.95, ε=0.1. They train for 500 episodes with a 150-step safety cap; this cap is a truncation, not a terminal state in their update.","Evaluation removes exploration, uses fresh environment seeds, and reports undiscounted episode return, goal, cliff and timeout frequencies. The plot is a numerical convergence diagnostic, not a statistical learning curve."],"limits":["One training seed per learned policy is illustrative, not a reliable algorithm ranking. The planner has privileged knowledge of the MDP.","This custom cliff terminates the episode; it differs from continuing reset-to-start cliff-walking. No PPO/SAC claim is made."],"references":[["Sutton & Barto · Reinforcement Learning","http://incompleteideas.net/book/the-book-2nd.html"]],"provenance":{"python":"3.13.0","numpy":"2.4.6","sourceSha256":{"showcase_benchmark.py":"c656287bb149fac541b882ab3ad32915bf1e5e4a55fb6cc6213a990ec997e947","rl_utils/tabular.py":"3645437bea7eeec5371ae26136ba445b51c2234997a821bf05a84d79431f523e"}}}