Posted: November 6th, 2023

Developing an Autonomous Navigation Framework for Commercial Vessels using Deep Reinforcement Learning

Developing an Autonomous Navigation Framework for Commercial Vessels using Deep Reinforcement Learning

Traditionally, commercial vessels have relied on human operators for navigation through open waters. However, with advancements in technologies such as computer vision, sensors and machine learning, autonomous navigation of commercial vessels is becoming increasingly viable. Deep reinforcement learning (DRL) is a promising approach for developing autonomous navigation capabilities as it allows agents to learn optimal navigation policies directly from interactions with complex, uncertain maritime environments.
This paper proposes a DRL-based framework for autonomous navigation of commercial vessels. A simulated environment models vessel dynamics and interactions with other vessels and obstacles. The state representation includes sensor readings while the action space controls propulsion and steering. A reward function encourages safe and efficient navigation. The DRL algorithm Deep Q-Network is used to approximate the optimal policy through trial-and-error learning. The agent is trained extensively in simulation and then evaluated on its ability to autonomously navigate in new scenarios. Once proficient in simulation, the agent’s policies can potentially be transferred to real vessels through domain adaptation techniques.
Framework Components

The simulated environment is created using data from studies modeling vessel dynamics and ocean conditions (Liu et al., 2020; Wang et al., 2021). It allows modeling interactions between the vessel, other traffic and obstacles realistically.
The state representation observed by the agent includes sensor readings like GPS, velocity, heading from radar/LIDAR as well as environmental conditions like wind. This high-dimensional input allows the agent to make informed decisions.
The action space consists of continuous control signals to manipulate the rudder and propeller in order to change the vessel’s heading and speed for navigation tasks.
The reward function encourages progress toward the destination while penalizing collisions and deviations from traffic lanes to guide safe and efficient autonomous navigation (Mnih et al., 2015).
The DRL algorithm Deep Q-Network (DQN) is implemented which uses a neural network to approximate the optimal Q-function and learn control policies through trial and error with the environment (Tzeng et al., 2017).
Training and Evaluation
For training, the DRL agent extensively interacts with the simulated environment over many episodes, using DQN to gradually refine its policies. The trained agent is then evaluated for its ability to autonomously navigate in new, previously unseen scenarios in the simulation to analyze its performance metrics like success rate, journey time and safety.
Once proficient in simulation, the agent’s policies can potentially be transferred to real commercial vessels through domain adaptation techniques, allowing deployment of autonomous navigation capabilities on real vessels.
Conclusion
This paper proposes a DRL-based framework for autonomous navigation of commercial vessels utilizing a simulated training and evaluation environment. The framework’s key components and their implementation were discussed. With further research, this approach shows promise for developing autonomous navigation solutions that could help improve safety, efficiency and sustainability of maritime transportation.