Advanced Aerial Imagery Analysis with Deep Neural Networks Explained in 5 Minutes.

Written by Alexander Usoltsev On August 14th of 2018

There is no secret that when dealing with aerial images the best state-of-the-art results are achieved with deep learning models which come at the cost of their complexity. At the same time thanks to the Open Data, we can explore in a creative way even the most sophisticated techniques.

At nam.R we are working hard to build a Digital Twin of France, and to achieve that we use a lot of sources of information. One of the richest of them being aerial images. For us, humans, “reading” images is easy, but to teach computers how to deal with it is sometimes a real challenge (and fun!). In this post, we will show how we extract a rich description of buildings’ roofs from aerial images, particularly detecting their slopes. In Computer Vision jargon this task is called “object segmentation”, and to do that we chose to use a deep learning approach.

One of the current state-of-the-art segmentation models is the Mask R-CNN model published by researchers from Facebook. And we used this architecture implemented with a Keras framework.

To sum up, our deep learning model should be able to analyze the aerial images and detect roof slopes. This can help us understand the solar energy potential of the roof, and ultimately lead to a progress in nam.R’s vision: accelerating the ecological transition.

What Data Do We Need?

First of all, we need to define what kind of data is suitable for this task. We have a choice between satellite and aerial images. The main difference between them, in the context of our work, is the image resolution. Openly accessible satellite images have a resolution of several meters per pixel, while one can find aerial images with resolutions around 15-20 cm per pixel.

Because we would like to find some fine details on the images, roof slopes and ridges, we have gone with aerial images.

Training a deep learning model to detect roof slopes is a “supervised learning” task, so we need not only the images but also the labels of the slopes. So we created some labels ourselves. This is not a very exciting task, but it is a necessary steps to train a decent model.

This way we obtained two types of data to train the machine learning model: images of roofs and the labels for roof slopes.

A train data “image-label” pair looks like this:

To train a good model we need as much data as possible. Of course, we can label more roofs by hand, but it also possible to generate new samples just with the use of some simple transformations of original images and labels (“data augmentation”). This could be a rotation, a vertical or a horizontal flip and so on.

This way we can obtain a big enough dataset to train our deep learning model.

Deep Learning Model Which Fits Our Goal

Last few years, many high-performance deep neural networks were developed, and achieved impressive results on tasks of object detection. We chose Mask RCNN, a high-performance object segmentation network that was released in 2017. We adapted Matterport's implementation to be compatible with our aerial images and labels data source.

During the training, the model takes images and corresponding labels and learns its internal parameters to detect roof slopes on any new image. Because our dataset is quite small, a couple of hours of training already produce decent results.

One of the indicators that our model learned is the value of its loss function. The loss function is the criterion which the model tries to minimize, and it is usually an average of error between the real label and the predicted one.

Below are the loss function values for our model, which are constantly decreasing. The figures mean that the model learns well how to detect roof slopes:

Detect Roof Slopes on New Image

During the prediction phase, the model reads only aerial image and predicts the contours of the roof slopes in the image. Because the prediction step does not require complex calculations, it is possible, for example, to copy trained model to the production server and use it to analyze images in a real time.

We found roof slopes on the new image

We can see, that predicted labels for roof slopes are quite accurate, but, as always, there is some space for improvement. From the image above we can see how the roof slope detection doesn’t work well on roofs uncommon material such as metal.

One can imagine several ways to improve this model. For example, we can try to add more training samples for this type of roof material or do more data transformations to generate new samples.

But for the goal of our exploration of deep learning in advanced aerial image analysis, it is already a great result.

This post was just one example of a deep learning model used by nam.R to make France’s Digital Twin richer and closer to the reality. We will share our other techniques in future posts.

Stay tuned!

Developing a complex computer vision system, a case study : solar panels equipped roofs.

Written by Bastien Hell On July 5th of 2018

At nam.R we are working hard to build the Digital Twin of France. To this end, we use a lot of sources of information such as aerial images. Extracting useful information from the unstructured data that are images? Sounds like a job for the computer vision team !

To detect all solar panels on the roofs of french buildings, we used aerial imagery and the known outlines of the buildings, through a pipeline consisting of a solar panel outline detector and a filtering algorithm.

We took inspiration from the projects revolving around the state-of-the-art object segmentation deep learning algorithm known as Mask-RCNN. This algorithm is the newborn of a family of algorithms developed by Ross Girshick & al., in direct continuation of RCNN, Fast-RCNN and Faster-RCNN.

The chosen pipeline consists in two complementary parts :
- an object detector, more specifically an instance segmentation algorithm, meant to detect solar panels and extract their contours
- a filtering algorithm that takes all detections and filters out those that don’t match our business rules

While the filtering algorithm can easily be developed using the expert rules we chose to consider (size of the detected solar panels, their position regarding to the considered roof), the deep learning model depends directly of the data we will feed it with.

The first part of the project was, accordingly, to generate a dataset of roofs equipped with solar panels, and the matching labels. There are multiple existing tools for image annotations (VGG VIA, MIT LabelMe,...) that can be used as is. We chose the VGG Image Annotator. After a few (hundreds of) clicks, we ended up with a dataset we’re quite proud of.

Only then, we were able to train the Mask-RCNN model to detect solar panels.

The first version of our model wasn’t performing all that well and if was necessary to add more data to the training stage. We realized semi-supervised learning with automatic labelling.

This technique consists in using the model as a way to compute more labels that are then controlled by human operators and used as training data for a new, more robust version of the model. Controlling whether the proposed labels were right and correcting the wrong ones was way simpler than labelling by hand hundreds of images. Basically, we used our first model as a replacement for crowdsourcing !

After a few loops we fetched more data and matching labels and were able to train a model that had acceptable performances.

We transformed the raw output of the model into polygons in the same format and projection as our building polygons using geometric algorithms (Marching Squares, Douglas-Peucker) and geographic transformations. This enables us to directly filter out the potential false positives. We found out that roof windows, glass roofs and blue awning fabric were likely to be mistaken for solar panels due to their similar visual textures.

Fortunately, most false positives can be filtered out using the information of the building shape and position, but also expert rules concerning the minimal surfaces for solar panels.

An example of a false positive that was filtered out using its relative position to the building

The first conclusion we draw is that the combination of machine learning and expert rules can become a reliable framework, harnessing the power of machine learning algorithms and the robustness of business rules.

The second one was the use of our first imperfect model to help us label more data. Real data but synthetic labels, a great example of human-machine cooperation, isn’t it?

Finally, there are many different ways for computing the performances of this kind of pipelines. The deep learning model itself can be evaluated using metrics such as its mean average precision but we were mainly interested in the performances of the whole flow. Thus, we chose metrics that are less image-centric and oriented more towards information retrieval : precision, recall and overall accuracy. We added a geometric metric that indicates how well our predicted panels matched with the actual ones, the Intersection over Union (IoU).

Examples of the IoU computing : better overlaps mean better scores

We achieved the performances of 96% overall algorithm accuracy and 84% IoU on our test set, values we’re quite proud of.

The predictions of solar panels were integrated in nam.R’s Digital Twin and the information is already put to good use !