Real-time hierarchical fusion system for semantic segmentation in offroad scenes

Semantic segmentation is an important task for autonomous vehicle navigation in off road environments. However, several natural factors make this problem uniquely challenging. For example, road segmentation is often difficult under heavy shadow or steel terrain, and dangerous muddy water puddles may have the similar visual appearance to dirt road surfaces (and thus are hard to identify). To tacule these challenges, we present a semantic segmentation system based on a two-stage hierarchical fusion pipeline. The first stage improves the road segmentation by effectively fusing information from camera and 3D Lidar point cloud. The second stage is dedicated to detecting water puddles, based on the results from the first stage. Due to the parallelized architecture, our system can be deployed for real-time applications. We achieved an F1 score of around 93% for road segmentation and 80% for water puddle segmentation at more than 10 Hz.