zjowowen's picture
Upload README.md with huggingface_hub
f7b9677 verified
---
language: en
license: apache-2.0
library_name: pytorch
tags:
- deep-reinforcement-learning
- reinforcement-learning
- DI-engine
- MsPacmanNoFrameskip-v4
benchmark_name: OpenAI/Gym/Atari
task_name: MsPacmanNoFrameskip-v4
pipeline_tag: reinforcement-learning
model-index:
- name: SampledEfficientZero
results:
- task:
type: reinforcement-learning
name: reinforcement-learning
dataset:
name: MsPacmanNoFrameskip-v4
type: MsPacmanNoFrameskip-v4
metrics:
- type: mean_reward
value: 1028.0 +/- 186.43
name: mean_reward
---
# Play **MsPacmanNoFrameskip-v4** with **SampledEfficientZero** Policy
## Model Description
<!-- Provide a longer summary of what this model is. -->
This implementation applies **SampledEfficientZero** to the OpenAI/Gym/Atari **MsPacmanNoFrameskip-v4** environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine).
**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).
## Model Usage
### Install the Dependencies
<details close>
<summary>(Click for Details)</summary>
```shell
# install huggingface_ding
git clone https://github.com/opendilab/huggingface_ding.git
pip3 install -e ./huggingface_ding/
# install environment dependencies if needed
pip3 install DI-engine[common_env,video]
pip3 install LightZero
```
</details>
### Git Clone from Huggingface and Run the Model
<details close>
<summary>(Click for Details)</summary>
```shell
# running with trained model
python3 -u run.py
```
**run.py**
```python
from lzero.agent import SampledEfficientZeroAgent
from ding.config import Config
from easydict import EasyDict
import torch
# Pull model from files which are git cloned from huggingface
policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu"))
cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict)
# Instantiate the agent
agent = SampledEfficientZeroAgent(
env_id="PongNoFrameskip-v4", exp_name="PongNoFrameskip-v4-SampledEfficientZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
)
# Continue training
agent.train(step=5000)
# Render the new agent performance
agent.deploy(enable_save_replay=True)
```
</details>
### Run Model by Using Huggingface_ding
<details close>
<summary>(Click for Details)</summary>
```shell
# running with trained model
python3 -u run.py
```
**run.py**
```python
from lzero.agent import SampledEfficientZeroAgent
from huggingface_ding import pull_model_from_hub
# Pull model from Hugggingface hub
policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/PongNoFrameskip-v4-SampledEfficientZero")
# Instantiate the agent
agent = SampledEfficientZeroAgent(
env_id="PongNoFrameskip-v4", exp_name="PongNoFrameskip-v4-SampledEfficientZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
)
# Continue training
agent.train(step=5000)
# Render the new agent performance
agent.deploy(enable_save_replay=True)
```
</details>
## Model Training
### Train the Model and Push to Huggingface_hub
<details close>
<summary>(Click for Details)</summary>
```shell
#Training Your Own Agent
python3 -u train.py
```
**train.py**
```python
from lzero.agent import SampledEfficientZeroAgent
from huggingface_ding import push_model_to_hub
# Instantiate the agent
agent = SampledEfficientZeroAgent(env_id="PongNoFrameskip-v4", exp_name="PongNoFrameskip-v4-SampledEfficientZero")
# Train the agent
return_ = agent.train(step=int(2000000))
# Push model to huggingface hub
push_model_to_hub(
agent=agent.best,
env_name="OpenAI/Gym/Atari",
task_name="PongNoFrameskip-v4",
algo_name="SampledEfficientZero",
github_repo_url="https://github.com/opendilab/LightZero",
github_doc_model_url=None,
github_doc_env_url=None,
installation_guide='''
pip3 install DI-engine[common_env,video]
pip3 install LightZero
''',
usage_file_by_git_clone="./sampled_efficientzero/pong_sampled_efficientzero_deploy.py",
usage_file_by_huggingface_ding="./sampled_efficientzero/pong_sampled_efficientzero_download.py",
train_file="./sampled_efficientzero/pong_sampled_efficientzero.py",
repo_id="OpenDILabCommunity/PongNoFrameskip-v4-SampledEfficientZero",
platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)",
model_description="**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).",
create_repo=True
)
```
</details>
**Configuration**
<details close>
<summary>(Click for Details)</summary>
```python
exp_config = {
'main_config': {
'exp_name': 'MsPacmanNoFrameskip-v4-SampledEfficientZero',
'seed': 0,
'env': {
'env_id': 'MsPacmanNoFrameskip-v4',
'env_name': 'MsPacmanNoFrameskip-v4',
'obs_shape': [4, 96, 96],
'collector_env_num': 8,
'evaluator_env_num': 3,
'n_evaluator_episode': 3,
'manager': {
'shared_memory': False
}
},
'policy': {
'on_policy': False,
'cuda': True,
'multi_gpu': False,
'bp_update_sync': True,
'traj_len_inf': False,
'model': {
'observation_shape': [4, 96, 96],
'frame_stack_num': 4,
'action_space_size': 9,
'downsample': True,
'continuous_action_space': False,
'num_of_sampled_actions': 5,
'discrete_action_encoding_type': 'one_hot',
'norm_type': 'BN'
},
'use_rnd_model': False,
'sampled_algo': True,
'gumbel_algo': False,
'mcts_ctree': True,
'collector_env_num': 8,
'evaluator_env_num': 3,
'env_type': 'not_board_games',
'action_type': 'fixed_action_space',
'battle_mode': 'play_with_bot_mode',
'monitor_extra_statistics': True,
'game_segment_length': 400,
'transform2string': False,
'gray_scale': False,
'use_augmentation': True,
'augmentation': ['shift', 'intensity'],
'ignore_done': False,
'update_per_collect': 1000,
'model_update_ratio': 0.1,
'batch_size': 256,
'optim_type': 'SGD',
'learning_rate': 0.2,
'target_update_freq': 100,
'target_update_freq_for_intrinsic_reward': 1000,
'weight_decay': 0.0001,
'momentum': 0.9,
'grad_clip_value': 10,
'n_episode': 8,
'num_simulations': 50,
'discount_factor': 0.997,
'td_steps': 5,
'num_unroll_steps': 5,
'reward_loss_weight': 1,
'value_loss_weight': 0.25,
'policy_loss_weight': 1,
'policy_entropy_loss_weight': 0,
'ssl_loss_weight': 2,
'lr_piecewise_constant_decay': True,
'threshold_training_steps_for_final_lr': 50000,
'manual_temperature_decay': False,
'threshold_training_steps_for_final_temperature': 100000,
'fixed_temperature_value': 0.25,
'use_ture_chance_label_in_chance_encoder': False,
'use_priority': True,
'priority_prob_alpha': 0.6,
'priority_prob_beta': 0.4,
'root_dirichlet_alpha': 0.3,
'root_noise_weight': 0.25,
'random_collect_episode_num': 0,
'eps': {
'eps_greedy_exploration_in_collect': False,
'type': 'linear',
'start': 1.0,
'end': 0.05,
'decay': 100000
},
'cfg_type': 'SampledEfficientZeroPolicyDict',
'init_w': 0.003,
'normalize_prob_of_sampled_actions': False,
'policy_loss_type': 'cross_entropy',
'lstm_horizon_len': 5,
'cos_lr_scheduler': False,
'reanalyze_ratio': 0.0,
'eval_freq': 2000,
'replay_buffer_size': 1000000
},
'wandb_logger': {
'gradient_logger': False,
'video_logger': False,
'plot_logger': False,
'action_logger': False,
'return_logger': False
}
},
'create_config': {
'env': {
'type': 'atari_lightzero',
'import_names': ['zoo.atari.envs.atari_lightzero_env']
},
'env_manager': {
'type': 'subprocess'
},
'policy': {
'type': 'sampled_efficientzero',
'import_names': ['lzero.policy.sampled_efficientzero']
}
}
}
```
</details>
**Training Procedure**
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
- **Weights & Biases (wandb):** [monitor link](<TODO>)
## Model Information
<!-- Provide the basic links for the model. -->
- **Github Repository:** [repo link](https://github.com/opendilab/LightZero)
- **Doc**: [Algorithm link](<TODO>)
- **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/MsPacmanNoFrameskip-v4-SampledEfficientZero/blob/main/policy_config.py)
- **Demo:** [video](https://huggingface.co/OpenDILabCommunity/MsPacmanNoFrameskip-v4-SampledEfficientZero/blob/main/replay.mp4)
<!-- Provide the size information for the model. -->
- **Parameters total size:** 33030.28 KB
- **Last Update Date:** 2024-01-19
## Environments
<!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
- **Benchmark:** OpenAI/Gym/Atari
- **Task:** MsPacmanNoFrameskip-v4
- **Gym version:** 0.25.1
- **DI-engine version:** v0.5.0
- **PyTorch version:** 2.0.1+cu117
- **Doc**: [Environments link](<TODO>)