A2C

Advantage Actor-Critic (A2C) is a synchronous and deterministic version of Asynchronous Advantage Actor-Critic (A3C). It combines value optimization and policy optimization approaches. This implementation of the A2C algorithm is built on PPO algorithm for simplicity, and it supports the following extensions:

  • Target network: ✔️

  • Gradient clipping: ✔️

  • Reward clipping: ❌

  • Generalized Advantage Estimation (GAE): ✔️

  • Discrete version: ✔️

Warning

The implementation of A2C serves as a pedagogical goal. For practitioners, we recommend using the PPO algorithm for training agents. Without the trust-region and clipped ratio, hyper-parameters in A2C, e.g., repeat_times, need to be fine-tuned to avoid performance collapse.

Code Snippet

import torch
from elegantrl.run import train_and_evaluate
from elegantrl.config import Arguments
from elegantrl.train.config import build_env
from elegantrl.agents.AgentA2C import AgentA2C

# train and save
args = Arguments(env=build_env('Pendulum-v0'), agent=AgentA2C())
args.cwd = 'demo_Pendulum_A2C'
args.env.target_return = -200
args.reward_scale = 2 ** -2
train_and_evaluate(args)

# test
agent = AgentA2C()
agent.init(args.net_dim, args.state_dim, args.action_dim)
agent.save_or_load_agent(cwd=args.cwd, if_save=False)

env = build_env('Pendulum-v0')
state = env.reset()
episode_reward = 0
for i in range(2 ** 10):
    action = agent.select_action(state)
    next_state, reward, done, _ = env.step(action)

    episode_reward += reward
    if done:
        print(f'Step {i:>6}, Episode return {episode_reward:8.3f}')
        break
    else:
        state = next_state
    env.render()

Parameters

class elegantrl.agents.AgentA2C.AgentA2C[source]

Bases: AgentPPO

A2C algorithm. “Asynchronous Methods for Deep Reinforcement Learning”. Mnih V. et al.. 2016.

Parameters
  • net_dim[int] – the dimension of networks (the width of neural networks)

  • state_dim[int] – the dimension of state (the number of state vector)

  • action_dim[int] – the dimension of action (the number of discrete action)

  • learning_rate[float] – learning rate of optimizer

  • if_per_or_gae[bool] – PER (off-policy) or GAE (on-policy) for sparse reward

  • env_num[int] – the env number of VectorEnv. env_num == 1 means don’t use VectorEnv

  • agent_id[int] – if the visible_gpu is ‘1,9,3,4’, agent_id=1 means (1,9,4,3)[agent_id] == 9

update_net(buffer, batch_size, repeat_times, soft_update_tau)[source]

Update the neural networks by sampling batch data from ReplayBuffer.

Note

Using advantage normalization and entropy loss.

Parameters
  • buffer – the ReplayBuffer instance that stores the trajectories.

  • batch_size – the size of batch data for Stochastic Gradient Descent (SGD).

  • repeat_times – the re-using times of each trajectory.

  • soft_update_tau – the soft update parameter.

Returns

a tuple of the log information.

class elegantrl.agents.AgentA2C.AgentDiscreteA2C[source]

Bases: AgentA2C

Parameters
  • net_dim[int] – the dimension of networks (the width of neural networks)

  • state_dim[int] – the dimension of state (the number of state vector)

  • action_dim[int] – the dimension of action (the number of discrete action)

  • learning_rate[float] – learning rate of optimizer

  • if_per_or_gae[bool] – PER (off-policy) or GAE (on-policy) for sparse reward

  • env_num[int] – the env number of VectorEnv. env_num == 1 means don’t use VectorEnv

  • agent_id[int] – if the visible_gpu is ‘1,9,3,4’, agent_id=1 means (1,9,4,3)[agent_id] == 9

explore_one_env(env, target_step)[source]

Collect trajectories through the actor-environment interaction for a single environment instance.

Parameters
  • env – the DRL environment instance.

  • target_step – the total step for the interaction.

  • reward_scale – a reward scalar to clip the reward.

  • gamma – the discount factor.

Returns

a list of trajectories [traj, …] where each trajectory is a list of transitions [(state, other), …].

explore_vec_env(env, target_step)[source]

Collect trajectories through the actor-environment interaction for a vectorized environment instance.

Parameters
  • env – the DRL environment instance.

  • target_step – the total step for the interaction.

  • reward_scale – a reward scalar to clip the reward.

  • gamma – the discount factor.

Returns

a list of trajectories [traj, …] where each trajectory is a list of transitions [(state, other), …].

Networks

class elegantrl.agents.net.ActorPPO(*args: Any, **kwargs: Any)[source]
class elegantrl.agents.net.ActorDiscretePPO(*args: Any, **kwargs: Any)[source]
class elegantrl.agents.net.CriticPPO(*args: Any, **kwargs: Any)[source]