Introduction
In December 2018, a team of researchers of DeepMind (owned by Google) published a paper in the journal Science, demonstrating the ability of their newly developed AlphaZero algorithm to beat the best game engines in Chess, Go, and Shogi. What is more, instead of relying on hand-crafted evaluation functions of board states, the AlphaZero algorithm contains no expert information on any of the games played: it autonomously learns to play each game only by playing the game many times against itself.
This AI breakthrough is exciting because Go and Chess are games where it is crucial to anticipate unknown moves of the opponent. When making logistics decisions, it is equally important to anticipate the arrival of new data (e.g., orders, delays, disruptions, etc.). For various dynamic data-driven decision problems, Deep Reinforcement Learning (DRL) algorithms like AlphaZero have been demonstrated to be game-changers. The logistics sector recognizes the opportunities and is eager to adopt AI for decision automation.
However, companies struggle to translate the abstract possibilities of AI into the tangible project plans, and employing DRL-based decision making requires expert algorithmic knowledge that is difficult to source.
To overcome these challenges, we started to develop the DynaPlex toolbox. In a similar fashion as AlphaZero was designed as a generic tool to solve various games, we created the DynaPlex toolbox to support the rapid development of automated decision making based on DRL. DynaPlex focuses on dynamic data-driven logistics challenges, with a focus on planning, scheduling, allocation and routing decisions, for example in transportation management, inventory management, and warehouse management.
Modelling Framework
The modelling framework acts as a common interface to define the problem to be solved by our DRL algorithms. This modelling architecture is the linking pin that connects all other constituents of the tool, such as the environment models and algorithms. The modelling framework is structured around the language of Markov Decision Processes (MDP), where the following problem elements have to be defined: stages (decision epochs), states (characterization of the problem at a decision epoch), decisions (possible decisions), rewards (costs or rewards corresponding with a decision in a state), and the transition function (how a decision in a state brings us to another state).
Algorithmic Framework
The algorithmic framework will be developed to solve the problems defined through our modelling framework. This algorithmic framework unifies the broad RL field: various problem types (e.g., model free versus model based, online versus offline), value or policy approximations (e.g., parametric, non-parametric, linear, kernel-based, neural networks), and algorithmic forms, such as policy gradient algorithms (e.g., REINFORCE), value based algorithms (e.g., Q-learning, Deep Q-Network), Actor-Critic Algorithms (e.g., A3C, TD3, SAC), trust region policy optimization (TRPO), proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and many more.
Algorithmic selector
Given the diversity in RL solution methodologies, insight is needed into the applicability of certain algorithmic forms to specific classes of logistics problems. This is achieved by creating a mapping based on certain problem features, as well as establishing hyper-heuristic forms that could automate this mapping. Furthermore, we support the user with parameter tuning, especially related to the tuning of hyper parameters in neural networks. This problem includes selecting good architectures (e.g., graph based, feedforward, or recurrent based) and the right numbers of layers and nodes for NNs.
