Value-Based Methods

$L = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}}\left[\left(r + \gamma \max_{a'} Q_{\bar{\theta}}(s', a') - Q_\theta(s, a)\right)^2\right]$$where$\mathcal{D}$is the replay buffer and$Q_{\bar{\theta}}$ is the target network.

Standard DQN overestimates Q-values. Double DQN decouples action selection from evaluation:

$$y = r + \gamma Q_{\bar{\theta}}(s', \arg\max_{a'} Q_\theta(s', a'))$$