Poor road conditions like cracks and potholes can cause inconvenience to passengers, damage to vehicles, and accidents. Detecting those obstacles has become relevant due to the rise of the autonomous vehicle. Although previous studies used various sensors and applied different image processing techniques, performance is still significantly lacking, especially when compared to the tremendous leaps in performance with computer vision and deep learning. This research addresses this issue with the help of deep learning-based techniques. We applied the You Only Look Once version 2 (YOLOv2) detector and propose a deep convolutional neural network (CNN) based on YOLOv2 with a different architecture and two models. Despite a limited amount of learning data and the challenging nature of pothole images, our proposed architecture is able to obtain a significant increase in performance over YOLOv2 (from 60.14% to 82.43% average precision).