Tensorflow 2.x Agents(TF-Agents, Reinforcement Learning Module) & PySC2

Question

There are pysc2( https://github.com/deepmind/pysc2 ) & Tensorflow(1.x) and OpenAI-Baselines( https://github.com/openai/baselines ), like the following

https://github.com/chris-chris/pysc2-examples
https://github.com/llSourcell/A-Guide-to-DeepMinds-StarCraft-AI-Environment

The TF team has recently come up with a RL implementations(alternative to OpenAi-Baselines) called TF-Agents ( https://github.com/tensorflow/agents ). Examples:

https://github.com/tensorflow/agents/blob/master/docs/tutorials/1_dqn_tutorial.ipynb
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_12_05_apply_rl.ipynb
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_12_04_atari.ipynb

For TF-Agents, you do

env_name = 'CartPole-v0'
train_py_env = suite_gym.load(env_name)
eval_py_env = suite_gym.load(env_name)

q_net = q_network.QNetwork(
    train_env.observation_spec(),
    train_env.action_spec(),
    fc_layer_params=fc_layer_params)

optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=learning_rate)

agent = dqn_agent.DqnAgent(
    train_env.time_step_spec(),
    train_env.action_spec(),
    q_network=q_net,
    optimizer=optimizer,
    td_errors_loss_fn=common.element_wise_squared_loss,
    train_step_counter=train_step_counter)
agent.initialize()

For pysc2,

from pysc2.env import environment
from pysc2.env import sc2_env
from pysc2.lib import actions
from pysc2.lib import actions as sc2_actions
from pysc2.lib import features
mineral_env = sc2_env.SC2Env(
        map_name="CollectMineralShards",
        step_mul=step_mul,
        agent_interface_format=AGENT_INTERFACE_FORMAT,
        visualize=True)

How do I combine TF-Agents and Pysc2 together? They are both Google products.

Answer 1

I've recently stumbled on a very similar situation where I wanted to use the hanabi-learning-environment developed by DeepMind with TF-Agents. I'm afraid I have to tell that there is no nice solution to this.

What you must do is fork the DeepMind repo and modify the environment wrapper to be compatible with what TF-Agents requires. It's gonna be quite some work to do especially if you are not familiar with how environments are defined in TF-Agents, but this is def.netly something that can be done in about a week of work.

If you want to get an idea of what I did you can look at the original rl_env.py code in the Hanabi repo from DeepMind, and what I modified it into in my repo

I have no idea why DeepMind stick to their structure instead of making their code more compatible, but this is how it is.