Extract Image Data from Waymo OD

Waymo released their Open Dataset in August 2019 followed by a Open Dataset Challenge in March for researchers like us in the field of autonomous vehicles, computer vision and graphics.

The data released is one of the largest, richest, and most diverse ones as opposed to prior ones as there are almost 1000 segments each consisting 20 seconds of continuous driving allowing for object recognition and detection in both 2D and 3D space. If you’d like to explore more about the dataset then follow this link for a brief overview and a detailed research publication can be found here.

This tutorial shows how to extract images as png and labels using the library which was developed to explore the dataset, for training models developed using other deep learning libraries or if you have storage issues as the entire dataset measures around 2TB then this library will help you retrieve each segment and extract as well, if required.

This article is based on my WaymoDataToolkit v1.0. Please check for the updated versions in the repository before proceeding.

Set up the Environment

As this library requires access to Waymo Open Dataset’s storage bucket which is hosted on Google Cloud Platform, there are a series of steps involved as a precursor to using the library in your desired environment.

1. Sign in with your Google credentials here and you will receive an email that you now have access to the dataset which also includes the Google Cloud Storage Bucket hosted here.
2. Let’s now set up the environment — Proceed to preferably create a virtual environment for your project with Python 3.6, make sure to install python3-dev, python3-pip and virtualenv as well.
3. Install gcloud and initialize with the Google account that was used to register on the Waymo Open Dataset website.
4. Clone the WaymoDataToolkit repository.
5. pip install -r requirements.txt
6. pip install waymo-open-dataset-tf-2–1–0==1.2.0

Extract Images and Labels

The repository includes demo.py that provides you with an abstraction to retrieve and extract both images and labels to a directory. The following variables can be modified:

imageDir — Directory to store extracted images
labelDir — Directory to store the labels
imageCounter - Keeps a count of images within a segment

The demo.py is programmed to retrieve training files and Google Storage link can be edited to retrieve either validation or test data, if required. The demo will load the first file from the list it generates after reading the list of links.

The first step is to initialize the WaymoDataToolkit with URL, the following does the job:

dataset = WaymoDataToolkit.WaymoDataToolkit(url)

The file can be retrieved by calling the following function on the object created:

dataset.dataRetriever()

The retrieved file now needs to be extracted into images and labels, which is set using the variables defined previously:

dataset.dataExtractor()

Now you have image data and their corresponding labels!

Future

The library currently extracts just images and their labels as I was curious to explore the image data, however, I am planning on adding abstraction functions to visualize image data with bounding boxes, extract LiDAR data and visualize LiDAR data as well.