Advanced Lane Detection

fig. 0 Lane Detection on a road in Annamalai Forest, Tamil Nadu, India

The field of computer vision has a vital role in helping autonomous systems to visualize their surroundings, make inferences from pixel data and use these inferences to predict regression-based values that can be used to feed controllers that have the responsibility to guide and control the autonomous vehicle. This article explains the process involved in lane detection using absolute computer vision techniques and walks you through the steps to recognize the lane area, calculate the RoC of the road and estimate the distance from the center of the lane. The supporting source code can be found here: Advanced Lane Detection

Camera Calibration (calibrateCamera.py)

As most cameras use a wide variety of lenses that in the plight of focusing (almost all) light rays on the sensor to capture a wide image, these light rays end up bending at the edges of these lenses due to refraction. This phenomenon leads to an effect that distorts the edges of images. The following video explains two major types of distortions with examples, highly recommended that you watch.

Assuming that now you understand what radial distortion is, we have to make use of distortion coefficients (k1, k2 & k3) to correct for radial distortion. The calibrateCamera.py handles the camera calibration process, which by default is not enabled. It is recommended that at least 20 chessboard images are used in the process to generate objpoints and imgpoints and once generated you can hardcode these in main.py for the future. The calibrate() in main is configured to look for the images in /data/calibration but you are free to point the function to any other directory of your choice.

fig. 1 Left: Distorted Image; Right: Undistorted Image

The entire math involved in this entire process of undistorting an image is quite interesting and to learn more about it OpenCV has a good tutorial that explains the concepts along with a few examples.

Perspective Transform (preprocess.py: 8–19)

The first step in detecting lanes is to align our visual systems to visualize the road ahead in a manner that they seem to be looking at it from a bird’s eye perspective, this will help in calculating the curvature of the road, as a result, will help us predict the steering angle for the next few hundred meters. The other benefit of top-down view is it fixes the issue where lane lines seem to merge whereas lane lines are infinitely parallel lines as long as the road runs.

The bird’s eye view can be achieved by applying the perspective transform, essentially mapping a set of four points bounding the lane area in our input image to the desired set of points, thus generating the desired top-down view. These points are determined on a case-to-case basis, the determining factors are mainly the position of the camera in the vehicle and its field of view. The pictures in fig. 2 represent the input and post-transformation output image respectively.

fig. 2 Left: Before, Right: After

Thresholding (preprocess.py: 22)

Now that lane lines are parallel, the next step is to segment them in the input image. The input image still contains R, G and B (3) channels, lanes are either white or yellow, based on this assumption, the input image can be converted into a single channel grayscale image thus eliminating channels that we don’t require. The other color space to convert to is HLS color space where S channel may provide with great results depending on the lighting situation. In my example, I have decided to use image thresholding because it worked adequately given the input file. The fig. 3 visualizes the output after thresholding:

fig. 3 cv2.threshold(image, 220, 225, cv2.THRESH_BINARY)

The lower threshold (220) and upper threshold (225) are to be manually tuned based on the input file, however, there is an advanced technique in OpenCV based on Holistic-Nested Edge Detection that employs a neural network to extract edges without any manual tuning of the thresholds but this article shall stick to the simple thresholding technique.

Finding Lane Pixels (laneDetection.py: 4–70)

Once the input image is preprocessed, the next step is to locate and map the lane lanes in the image space. The approach would be to plot a histogram of pixels that are non-zero in the lower half of the binary image (threshold image) to observe the following pattern:

fig. 4 Histogram of x = Pixels, y = Counts

As the pixel values are now binary, the peaks can represent where most of the non-zero pixels are located and thus are a good indicator of the lane lines. Thus, the x-coordinates from the histogram serve as a starting point to search for the respective lanes. The concept of sliding windows approach will be applied here, which is essentially a window with a margin being placed around the line’s center (one for each, i.e. left and right lane). The following video illustrates the concept of the sliding window, followed by the result in fig. 5.

fig. 5. The concept of sliding windows applied to result from fig. 4.

Recognize Lane Area (laneDetection.py: 85–149)

The sliding windows help in estimating center of each lane area, using these x and y pixel positions the function search_around_poly() can fit a second-order polynomial curve. The function fits for f(y) instead of f(x) because the lanes are vertical in the image. Fig. 6 is a good illustration of this step.

fig. 6 Fit a second order polynomial on these lanes detected

The next step is to compute the radius of curvature which can be calculated with a circle that closely fits nearby points on a local section of a curve as seen in fig. 7. The radius of curvature of the curve at a particular point can be defined as the radius of the approximating circle. This radius can be computed using the formula from fig. 7.

fig. 7 Radius of Curvature concept illustration and an equation to compute RoC

The last step would be to highlight the lane area by fitting a polygon between these points and projecting it back on to the original image .The lane area and radius of curvature have been computed in image space based on pixel values which is not the same as real world space, hence they have to be converted into real world values, this involves measuring the length and width of the section of lane on which we are projecting our warped image. For simplicity, we can assume that most lanes are going to be usually 30 meters long and 3.7 meters wide and the code uses these values to convert pixel space values to real-world meter measurements.

fig. 8 The final expected result highlighting the lane area

Action

The pipeline discussed in this article works really well on data such as the following:

The next clip is a recording of me driving through Annamalai Forest, Tamil Nadu, India and it is really impressive to observe that this pipeline is able to detect lanes even on roads with different lighting conditions and surroundings:

Conclusion

The steps discussed in this article are great steps, to begin with, that solve the lane detection problem with absolute computer vision techniques. Several even more advanced techniques take advantage of neural networks to provide consistent, and comprehensive results even when the input data consists of varying lighting conditions and surroundings. I hope this article serves as a great starting point for you to explore the field of computer vision in autonomous systems.