深度详解：对象检测和图像分割的数据探索过程-FlyAI

深度详解：对象检测和图像分割的数据探索过程

图像分类

最后更新 2020-09-17 16:52 阅读 416

最后更新 2020-09-17 16:52

阅读 416

图像分类

##FlyAI文献翻译
**英文原文：**[Data Exploration Process for Object Detection And Image Segmentation](https://hackernoon.com/data-exploration-process-for-object-detection-and-image-segmentation-qn113u24 "Data Exploration Process for Object Detection And Image Segmentation")
> **标签**：图像分割

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599121946692466.jpeg)

I’ve been working with object detection and image segmentation problems for many years. An important realization I made is that people don’t put the same amount of effort and emphasis on data exploration and results analysis as they would normally in any other non-image machine learning project.

**Why is it so?**
I  believe there are two major reasons for it:

People don’t understand object detection and image segmentation models in depth and treat them as black boxes, in that case they don’t even know what to look at and what the assumptions are.

It can be quite tedious from a technical point of view as we don’t have good image data exploration tools.

In my opinion image datasets are not really an exception, understanding how to adjust the system to match our data is a critical step to success.

In this article, I will share with you how I approach data exploration for image segmentation and object detection problems. Specifically:

Why you should care about image and object dimensions,

Why small objects can be problematic for many deep learning architectures,

Why tackling class imbalances can be quite hard,

Why a good visualization is worth a thousand metrics,

The pitfalls of data augmentation.

>多年来，我一直致力于研究目标检测和图像分割问题。我的一个重要认识是，人们在数据探索和结果分析上没有像在其他非图像机器学习项目中那样投入同样的精力和重心。

>**为什么会这样呢？**
>我觉得有两个主要的原因：

>人们不深入了解目标检测和图像分割模型，而是把它们当作黑盒子，在这种情况下，他们甚至不知道要看的是什么和假设是什么。
从技术角度来看，这是相当乏味的，因为我们没有好的图像数据探测工具。
在我看来，图像数据集并不是真正的例外，理解如何调整系统来匹配我们的数据是成功的关键一步。

>在本文中，我将与您分享我是如何处理图像分割和目标检测问题的数据探索的。具体地说：

> 为什么要关注图像和物体的尺寸，
>为什么对于许多深度学习架构来说，小对象可能是一个问题，
>为什么解决类别失衡可能非常困难，
>为什么一个好的可视化效果值一千个技巧
>数据扩充的陷阱。

**The Need for Data Exploration for Image Segmentation and Object Detection**
Data exploration is key to a lot of machine learning processes. That said, when it comes to object detection and image segmentation datasets there is no straightforward way to systematically do data exploration.

There are multiple things that distinguish working with regular image datasets from object and segmentation ones:

The label is strongly bound to the image. Suddenly you have to be careful of whatever you do to your images as it can break the image-label-mapping.

Usually much more labels per image.

Much more hyperparameters to tune (especially if you train on your custom datasets)

This makes evaluation, results exploration and error analysis much harder. You will also find that choosing a single performance measure for your system can be quite tricky – in that case manual exploration might still be  a critical step.

**Data Quality and Common Problems**
The first thing you should do when working on any machine learning problem (image segmentation, object detection included)  is assessing quality and understanding your data.

Common data problems when training Object Detection and Image Segmentation models include:

Image dimensions and aspect ratios (especially dealing with extreme values)

Labels composition – imbalances, bounding box sizes, aspect ratios (for instance a lot of small objects)

Data preparation not suitable for your dataset.

Modelling approach not aligned with the data.

Those will be especially important if you train on custom datasets that are significantly different from typical benchmark datasets such as COCO.  In the next chapters, I will show you how to spot the problems  I mentioned and how to address them.

>**数据挖掘对于图像分割和目标检测的需要**

>数据探索是很多机器学习过程的关键。也就是说，当涉及到目标检测和图像分割数据集时，没有直接的方法进行系统地数据探索。

>在处理常规图像数据集和分割图像数据集时，有很多东西是可以区分的：

>标签被强绑定在图像上。您必须非常小心对图像所做的任何操作，因为它可能破坏图像-标签-映射。
通常每个图像有更多的标签。
>还有更多需要调整的超参数（特别是如果您对自定义数据集进行训练）
>这使得评价、结果探索和错误分析变得更加困难。你还会发现，给你的系统选择单一的性能度量可能会变得非常棘手，在这种情况下，手工探索可能仍然是关键的一步。

>**数据质量和常见问题**
>在处理任何机器学习问题(包括图像分割，对象检测)时，你应该做的第一件事是评估质量并理解你的数据。

>图像尺寸和纵横比(特别是处理极值时)。
标签组合不平衡，边框大小，长宽比(例如许多小对象)。
准备的数据不适合你的数据集。
建模方法与数据不一致。
如果您训练的自定义数据集与经典基准数据集（例如COCO）有显着差异，那么这些将特别重要。 在下一章中，我将向您展示如何发现我提到的问题以及如何解决它们。

**General Data Quality**

This one is simple and rather obvious, also this step would be the same for all image problems not just object detection or image segmentation. What we need to do here is:

get the general feel of a dataset and inspect  it visually.

make sure it’s not corrupt and  does not contain any obvious artifacts (for instance  black only images)

make sure that all the files are readable – you don’t want to find that out in the middle of your training.

My tip here is to  visualize as many pictures as possible. There are multiple ways of doing this. Depending on the size of the datasets some might be more suitable than the others.

Plot them in a jupyter notebook using matplotlib.Use dedicated tooling like google facets to explore image data (https://pair-code.github.io/facets/)

**Use HTML rendering to visualize and explore in a notebook.**

I’m a huge fan of the last option, it works great in jupyter notebooks (even for thousands of pictures at the same time!) Try doing that with matplotlib. There is even more: you can install a hover-zoom  extension that will allow you to  zoom in into individual pictures to inspect them in high-resolution.

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122162959443.gif)

>**通用数据质量**
这一步很简单，也很明显，而且这一步对所有的图像问题都是一样的，不仅仅是目标检测或图像分割。我们要做的是：

>获得数据集的总体感觉并进行检查。
确保它没有损坏并且不包含任何明显的伪像（例如仅黑色图像）。
确保所有的文件都是可读的，你可不想在训练过程中发现这个问题。
我的建议是要尽可能多地可视化图片。 有多种方法可以做到这一点。 根据数据集的大小，某些数据可能比其他数据更合适。

>使用matplotlib将它们绘制在jupyter笔记本中。使用诸如Google facets之类的专用工具来浏览图像数据（https://pair-code.github.io/facets/）

>**使用HTML渲染在笔记本中可视化和探索**

>我是最后一个选择的忠实粉丝，它在jupyter note电脑上非常有效（甚至可以同时处理数千张图片！）请尝试使用matplotlib进行操作。 还有更多功能：您可以安装一个悬停缩放扩展程序，以便将其放大到单个图片中以高分辨率对其进行检查。

**Image sizes and aspect Ratios**

In the real world, datasets are unlikely to contain images of the same  sizes and aspect ratios. Inspecting basic datasets statistics such as aspect ratios, image widths and heights will help you make important decisions:

Can you and should you? do destructive resizing ? (destructive means resizing that changes the AR)

For non-destructive resizing what should be your desired output resolution and amount of padding?

Deep Learning models might have hyper parameters you have to tune depending on the above  (for instance anchor size and ratios)  or they might even have strong requirements when it comes to minimum input image size.

Good resources about anchors.

A special case would be if your dataset consists of images that are really big (4K+), which is not that unusual in satellite imagery or some medical modalities. For most cutting edge models in 2020, you will not be able to fit even a single 4K image per (server grade) GPU due to memory constraints. In that case, you need to figure out what realistically will be useful for your DL algorithms.

Two approaches that I saw are:

Training your model on image patches (randomly selected during training or extracted before training)

resizing the entire dataset to avoid  doing this every time you load your data.

>图像大小和长宽比
在现实世界中，数据集不太可能包含相同大小和高宽比的图像。检查基本的数据集统计信息，如纵横比、图像宽度和高度，将帮助您做出重要的决定：

>你能和应该吗?破坏性的大小调整?(破坏性意味着改变AR的大小)
对于非破坏性的大小调整应该是你想要的输出分辨率和填充量
深度学习模型可能会有一些你必须根据上面的参数来调整的超参数(例如锚的大小和比率)，或者当涉及到最小输入图像大小时，它们甚至可能有很强的要求。
关于锚点的资源

>一种特殊情况是，如果您的数据集包含非常大的图像（4K +），这在卫星图像或某些医疗模式中并不罕见。 对于2020年的大多数尖端型号，由于内存限制，每个（服务器级）GPU甚至无法容纳单个4K图像。 在这种情况下，您需要弄清楚什么对你的深度学习算法有用。

>我使用的两个方法：

>在图像补丁上训练你的模型(在训练中随机选择或在训练前提取)
调整整个数据集的大小，以避免每次加载数据时都这样做

![Fig 2. Histogram of image aspect ratios in the coco dataset ](https://static.leiphone.com/uploads/new/sns/article/202009/1599122277769894.png "Fig 2. Histogram of image aspect ratios in the coco dataset ")

In general I would expect most datasets to fall into one of 3 categories.

Uniformly distributed where most of the images have the same dimensions – here  the only decision you will have to make is how much to resize (if at all) This will mainly depend on objects area, size and aspect ratios)

Slightly bimodal distribution but most of the images are in the aspect ratio  range of (0.7  …  1.5) similar to the COCO dataset. I believe other “natural-looking” datasets would follow a similar distribution –  for those type of datasets you should be fine by going with a non-destructive resize -> Pad approach. Padding will be necessary but to a degree that is manageable and will not blow the size of the dataset too much

Dataset with a lot of extreme values (very wide images mixed with very narrow ones) – this case is much more tricky and there are more advanced techniques to avoid excessive padding. You might consider sampling batches of images based on the aspect ratio. Remember that this can introduce a bias to your sampling process – so make sure its acceptable or not strong enough.

The mmdetection framework supports this out of the box by implementing a GroupSampler that samples based on AR’s

>均匀分布在大多数图像具有相同尺寸的地方–在这里，您唯一要做的决定就是调整大小（如果有的话），这主要取决于对象的面积，大小和纵横比）。

>有点双峰分布，但大多数图像的纵横比在（0.7…1.5）范围内，与COCO数据集相似。 我相信其他``看起来自然''的数据集也会遵循类似的分布-对于那些类型的数据集，您应该使用无损调整大小->填充方法来解决问题。 填充是必需的，但在一定程度上是可以控制的，不会过多地破坏数据集的大小。

>具有很多极值的数据集（非常宽的图像和非常狭窄的图像混合在一起）–这种情况更加棘手，并且有更先进的技术来避免过度填充。 您可能考虑根据宽高比对图像批次进行采样。 请记住，这可能会给您的采样过程带来偏差-因此请确保其可接受程度或不足。
mmdetection框架通过实现一个GroupSampler来支持此功能，该GroupSampler基于AR的采样。

![Fig 3 and 4. Example Images (resized and padded) with a extreme aspect ratios from the coco dataset](https://static.leiphone.com/uploads/new/sns/article/202009/1599122337273715.png "Fig 3 and 4. Example Images (resized and padded) with a extreme aspect ratios from the coco dataset")

**Label (objects) sizes and dimensions**

Here we start looking at our targets (labels). Particularly we are interested in knowing how the sizes and aspect ratios are distributed.

Why is this important?

Depending on your modelling approach most of the frameworks will have design limitations. As I mentioned earlier, those models are designed to perform well on benchmark datasets. If for whatever reason your data is different, training them might be impossible.  Let’s have a look at a default config for Retinanet from detectron2:

>**标签(对象)的大小和尺寸**

>在这里，我们开始查看目标（标签），我们特别想知道尺寸和纵横比的分布方式。

>为什么这很重要？

>根据您的建模方法，大多数框架都会有设计限制。 如前所述，这些模型旨在在基准数据集上表现良好。 如果出于任何原因您的数据不同，则可能无法对其进行训练。让我们看一下detectron2中Retinanet的默认配置：

```
ANCHOR_GENERATOR:
    SIZES: !!python/object/apply:eval ["[[x, x * 2**(1.0/3), x * 2**(2.0/3) ] for x in [32, 64, 128, 256, 512 ]]"]
	```

What you can see there is, that for different feature maps the anchors we generate will have a certain size range:

for instance, if your dataset contains only really big objects – it might be possible to simplify the model a lot,

on the other side let’s assume you have small images with small objects (for instance 10x10px) given this config it can happen you will not be able to train the model.

The most important things to consider when it comes to box or mask dimensions are:

Aspect ratios

Size (Area)

>你可以看到的是，对于不同的特征地图，我们生成的锚将有一定的大小范围：

>例如，如果您的数据集只包含真正大的对象，那么就有可能大大简化模型。
另一方面，让我们假设你有小的图像和小的对象(例如10x10px)，在这种配置下，你可能无法训练模型。
关于盒子或遮罩尺寸，要考虑的最重要的事情是：

>纵横比
>区域大小

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599124379887379.png)

The tail of this distribution (fig. 3)  is quite long. There will be instances with extreme aspect ratios.  Depending on the use case and dataset it might be fine to ignore it or not, this should be further inspected.

>这种分布的尾巴很长（图3）。 某些情况下会出现极高的宽高比。 根据用例和数据集，是否忽略它可能会很好，应对此进行进一步检查。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122463727437.png)

This is especially true for anchor-based models (most of object detection / image segmentation models)  where there is a step of matching ground truth labels with predefined anchor boxes (aka. Prior boxes).

Remember that you control how those prior boxes are generated with hyperparameters like the number of boxes, their aspect ratio, and size. Not surprisingly you need to make sure those settings are aligned with your dataset distributions and expectations.

>对于基于锚点的模型（大多数目标检测/图像分割模型）尤其如此，其中有一个步骤将真实标签与预定义的锚点框（也称为先验框）匹配。

>请记住，您可以使用框的数量、它们的高宽比和大小等超参数来控制这些框的生成方式。毫不奇怪，您需要确保这些设置与您的数据集分布和期望保持一致。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122500810396.png)

An important thing to keep in mind is that labels will be transformed together with the image. So if you are making an image smaller during a preprocessing step the absolute size of the ROI’s will also shrink.

If you feel that object size might be an issue in your problem and you don’t want to enlarge the images too much (for instance to keep desired performance or memory footprint)  you can try to solve it with a Crop -> Resize approach.  Keep in mind that this can be quite tricky (you need to handle cases what happens if you cut through a bounding box or segmentation mask)

Big objects on the other hand are usually not problematic from a modelling perspective (although you still have to make sure that will be matched with anchors). The problem with them is more indirect, essentially the more big objects a class has the more likely it is that it will be underrepresented in the dataset. Most of the time the average area of objects in a given class will be inversely proportional to the (label) count.

>需要记住的一件重要的事情是，标签将与图像一起转换。因此，如果你在预处理过程中缩小图像，ROI的绝对尺寸也会缩小。

>如果您觉得问题可能是对象大小的问题，并且不想太大放大图像（例如保持所需的性能或内存占用），则可以尝试使用Crop-> Resize方法解决它。 请记住，这可能非常棘手（您需要处理穿过边界框或分割蒙版时发生的情况）。

>另一方面，从建模的角度来看，大对象通常没有问题（尽管您仍然必须确保将其与锚点匹配）。 它们的问题更加间接，本质上说，一类具有的大对象越多，它在数据集中的代表性就越低。 在大多数情况下，给定类中对象的平均面积将与（标签）计数成反比。

**Partially labeled data**

When creating and labeling an image detection dataset missing annotations are potentially a huge issue. The worst scenario is when you have false negatives already in your ground truth. So essentially you did not annotate objects even though they are present in the dataset.

In most of the modeling approaches, everything that was not labeled or did not match with an anchor is considered background. This means that it will generate conflicting signals that will hurt the learning process a LOT.

This is also a reason why you can’t really mix datasets with non-overlapping classes and train one model (there are some way to mix datasets though – for instance by soft labeling one dataset with a model trained on another one)

>**部分标记的数据**

>在创建和标记图像检测数据集时，缺少注释是一个潜在的大问题。最糟糕的情况是，在真实标记框中标记成负类样本。所以即使它们在数据集中出现，本质上你并没有注释对象。

>在大多数建模方法中，没有标记或与锚不匹配的所有东西都被认为是背景。这意味着它会产生相互矛盾的信号，这会对学习过程造成很大的伤害。

>这也是为什么您不能真正地将数据集与不重叠的类混合并训练一个模型的原因（尽管有一些方法可以混合数据集，例如，用一个训练有素的模型对一个数据集进行软标记）

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122566929216.png)
Fig 8.  Shows the problem of mixing datasets – notice for example that on the right image a person is not labeled. One way to solve this problem is to soft label the dataset with a model trained on the other one. Source

>图8所示。显示混合数据集的问题，注意例如，在右边的图像上一个人没有标签。解决这个问题的一种方法是用对另一个数据集进行训练的模型对数据集进行软标记。源图片

**Imbalances**

Class imbalances can be a bit of a problem when it comes to object detection.  Normally in image classification for example, one can easily oversample or downsample the dataset and  control each class contribution to the loss.

>**不平衡问题**

>当涉及到目标检测时，类不平衡可能是一个问题。通常在图像分类中，例如，人们可以很容易地对数据集进行过采样或向下采样，并控制每个类对损失的贡献。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122599990308.png)

Fig 9. Object counts per class

You can imagine this is more challenging when you have co-occurring classes object detection dataset since you can’t really drop some of the labels (because you would send mixed signals as to what the background is).

In that case you end up having the same problem as shown in the partially labeled data paragraph. Once you start resampling on an image level you have to be aware of the fact that multiple classes will be upsampled at the same time.

Note:

You may want to try other solutions like:

Adding weights to the loss (making the contributions of some boxes or pixels higher)

Preprocessing your data differently: for example you could do some custom cropping that rebalances the dataset on the object level

>图9所示 每个类的对象数量

>您可以想象，在同时存在类对象检测数据集的情况下，这将更具挑战性，因为您无法真正删除某些标签（因为您会发送关于背景是什么的混合信号）。

>在这种情况下，您最终会遇到与部分标记的数据段落中所示相同的问题。 一旦开始在图像级别上重新采样，就必须意识到多个类将同时被上采样的事实。

>注意：

>你可能试过这样的方法：

>增加损失的权重（使某些方框或像素的贡献更高）。
>以不同的方式预处理数据：例如，您可以进行一些自定义裁剪，以在对象级别重新平衡数据集。

**Understanding Augmentation and Preprocessing Sequences**

Preprocessing and data augmentation is an integral part of any computer vision system. If you do it well you can gain a lot but if you screw up it can really cost you.

Data augmentation is by far the most important and widely used regularization technique (in image segmentation / object detection ).

Applying it to object detection and segmentation problems is more challenging than in simple image classification because some transformations (like rotation, or crop) need to be applied not only to the source image but also to the target (masks or bounding boxes).  Common transformations that require a target transform include:

Affine transformations,

Cropping,

Distortions,

Scaling,

Rotations

and many more.

It is crucial to do data exploration on batches of augmented images and targets to avoid costly mistakes (dropping bounding boxes, etc).

Note:

Basic augmentations are a part of deep learning frameworks like PyTorch or Tensorflow but if you need more advanced functionalities you need to use one of the augmentation libraries available in the python ecosystem. My recommendations are:

Albumentations (I’ll use it in this post)

Imgaug

Augmentor

>**理解增强和预处理序列**

>预处理和数据增强是任何计算机视觉系统的组成部分。 如果做得好，你会收获很多，但是如果搞砸了，那确实会花钱。

>迄今为止，数据增强是最重要且使用最广泛的正则化技术（在图像分割/目标检测中）。

>将其应用于目标检测和分割问题比简单的图像分类更具挑战性，因为一些转换(如旋转或裁剪)不仅需要应用于源图像，还需要应用于目标图像(遮罩或边框)。需要目标转换的常见转换包括：

>仿射变换
>裁剪
>扭曲
>尺度变化
>旋转
>等等
>对成批的增强图像和目标进行数据探索，以避免代价高昂的错误（丢弃边界框等）至关重要。

>注意：

>基本增强是PyTorch或Tensorflow等深度学习框架的一部分，但如果您需要更高级的功能，则需要使用python生态系统中可用的扩充库之一。 我的建议是：

>Albumentations (本文中我使用的是这个)

>Imgaug

>Augmentor

**The Minimal Preprocessing Setup**

Whenever I’m building a new system I want to keep it very basic on the preprocessing and augmentation level to minimize the risk of introducing bugs early on. Basic principles I would recommend you to follow is:

Disable augmentation

Avoid destructive resizing

Always inspect the outputs visually

Let’s continue our COOC example. From the previous steps we know that:the majority of our images have:

aspect ratios = width / height = 1.5

the average avg_width is  = 600 and avg_height  = 500.

Setting the averages as our basic preprocessing resize values  seems to be a reasonable thing to do ( unless there is a strong requirement on the model side  to have bigger pictures )  for instance a resnet50 backbone model  has a minimum size requirement of 32×32 (this is related to the number of downsampling layers)

In Albumentations the basic setup implementation will look something like this:

LongestMaxSize(avg_height) – this will rescale the image based on the longest side preserving the aspect ratio

PadIfNeeded(avg_height, avg_width, border_mode=’FILL’, value=0)

>**最小的预处理设置**

>每当我构建新系统时，我都希望在预处理和扩充级别上保持非常基础，以最大程度地降低早期引入错误的风险。 我建议您遵循的基本原则是：

>禁用增强
>避免破坏性的调整
>总是可视化地检查输出
>让我们继续COOC的例子。从前面的步骤我们知道:我们的大多数图像：

>宽高比=宽度/高度= 1.5
>平均宽度= 600,avg高度= 500
>将平均值设置为我们的基本预处理调整大小值似乎是一件合理的事情（除非模型方面有很强的要求才能拥有更大的图片），例如resnet50主干模型的最小大小要求为32×32（这是 与下采样层数有关）

>在Albumentations中，基本设置实现将如下所示：

>这将根据保持长宽比的最长边来缩放图像
>PadIfNeeded(avg height, avg width, border mode= FILL , value=0)

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122773627125.png)

Fig 10 and 11. MaxSize->Pad output for two pictures with drastically different aspect ratios

As you can see on figure 10 and 11 the preprocessing results in an image of 500×600 with reasonable 0-padding for both pictures.

When you use padding there are many options in which you can fill the empty space. In the basic setup I suggest that you go with default constant 0 value,

When you experiment with more advanced methods like reflection padding always explore your augmentations visually. Remember that you are running the risk of introducing false negatives especially in object detection problems (reflecting an object without having a label for it)

>图10和11 两张图片的纵横比截然不同的MaxSize-> Pad输出

>正如您在图10和图11中看到的，预处理结果是500600的图像，这两幅图像都有合理的0填充。

>当您使用填充时，有许多选项可以填充空白区域。在基本设置中，我建议您使用默认常量0值。

>当您尝试使用诸如反射填充之类的更高级的方法时，请始终以可视方式探索您的增强。 请记住，您冒着引入假阴性的风险，尤其是在物体检测问题中（反射物体而没有标签）。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122826388589.png)

Fig 12. Notice how reflection-padding creates false negative errors in our annotations. The cat’s reflection (top of the picture) has no label!

>图12.注意反射填充如何在我们的注释中产生假的负错误。 猫的倒影（图片顶部）没有标签！

**Augmentation – Rotations**

Rotations are powerful and useful augmentations but they should be used with caution. Have a look at fig 13. below which was generated using a Rotate(45)->Resize->Pad pipeline.

>**数据增强 --- 旋转**

>旋转是强大和有用的增强，但他们应该谨慎使用。请看图13。下面是使用一个Rotate(45)->调整大小->填充生成的。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122862267040.png)

Fig 13. Rotations can be harmful to your bounding box labels

The problem is that if we use standard bounding boxes (without an angle parameter), covering a rotated object can be less efficient (box-area to object-area will increase). This happens during rotation augmentations and it can harm the data. Notice that we have also introduced false positive labels in the top left corner. This is because we crop-rotated the image.

My recommendation is:

You might want to give up on those if you have a lot of objects with aspect ratios far from one.

Another thing you can consider  is using 90,180, 270 degree non-cropping rotations (if they make sense) for your problem (they will not destroy any bounding boxes)

>图13所示。旋转可能对你的边框标签有害

>问题是，如果我们使用标准的边界框（不带角度参数），则覆盖旋转的对象的效率可能会降低（框区域到对象区域的面积会增加）。 这种情况在旋转增强期间发生，并且可能会损坏数据。 请注意，我们还在左上角引入了误报标签。 这是因为我们裁剪了图像。

>我的建议是：

>如果您有许多对象的宽高比远非一个，则可能要放弃这些对象。
>你可以考虑使用90,180,270度的非裁剪旋转(如果它们有意义的话)来解决你的问题(它们不会破坏任何边框)。

**Augmentations – Key takeaways**

As you see, spatial transforms can be quite tricky and a lot of unexpected things can happen (especially  for object detection problems).

So if you decide to use those spatial augmentations make sure to do some data exploration and visually inspect your data.

Note:

Do you really need spatial augmentations? I believe that in many scenarios you will not need them and as usual keep things simpler and gradually add complexity.

From my experience a good starting point (without spatial transforms) and for natural looking datasets (similar to coco) is the following pipeline:

>**数据增强 --- 关键点**

>正如您所看到的，空间转换非常棘手，可能会发生许多意想不到的事情(特别是对于目标检测问题)。

>因此，如果您决定使用这些空间扩展，请确保进行一些数据探索并可视化地检查您的数据。

>注意：

>你真的需要增加空间吗?我相信在许多情况下，您将不需要它们，并像通常那样使事情更简单并逐渐增加复杂性。

>根据我的经验，对于看起来自然的数据集(类似coco)来说，下面的通道是一个很好的起点(没有空间转换)：

```
transforms = [
    LongestMaxSize(max_size=500),
    HorizontalFlip(p=0.5),
    PadIfNeeded(500, 600, border_mode=0, value=0),
    JpegCompression(quality_lower=70, quality_upper=100, p=1),
    RandomBrightnessContrast(0.3, 0.3),
    Cutout(max_h_size=32, max_w_size=32, p=1)
]
```
Of course things like max_size or cutout sizes are arbitrary and have to be adjusted.

>当然，像最大尺寸或切断尺寸是任意的，必须调整。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599122968297781.png)

Fig 14. Augmentation results  with cutout, jpeg compression and contrast/brightness adjustments

Best Practice:
One thing I did not mention yet that I feel is pretty important: Always load the whole dataset (together with your preprocessing and augmentation pipeline) .

>图14.带有剪切，jpeg压缩和对比度/亮度调整的增强结果

>最佳做法：
>我没有提到的一件事我觉得非常重要：始终加载整个数据集（连同您的预处理和扩充通道）。

```
%%timeit -n 1 -r 1for b in data_loader: pass
```

Two lines of code that will save you a lot of time. First of all, you will understand what the overhead of the data loading is and if you see a clear performance bottleneck you might consider fixing it right away. More importantly, you will catch potential issues with:

corrupted files, labels that can’t be transformed etc anything fishy that can interrupt training down the line.

>两行代码将节省您很多时间。首先，您将了解数据加载的开销是什么，如果您看到一个明显的性能瓶颈，您可能会考虑立即修复它。更重要的是，您将捕获潜在的问题：

>损坏的文件，无法转换的标签等任何会干扰训练的可疑内容。

**Understanding Results**

Inspecting model results and performing error analysis can be a tricky process  for those types of problems. Having one metric rarely tells you the whole story and  if you do have one interpreting it can be a relatively hard task.

Let’s have a look at the official coco challenge and how  the evaluation process looks there (all the results i will be showing are for a MASK R-CNN model with a resnet50 backbone).

>**看懂结果**

>对于这些类型的问题，检查模型结果和执行错误分析可能是一个棘手的过程。只有一个度量标准很少能告诉你整个故事，如果你有一个度量标准来解释它，那将是一项相对困难的任务。

>让我们看看官方的coco挑战和评估过程是如何进行的(我将展示的所有结果都是一个带有resnet50主干的MASK R-CNN模型)。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123054809596.png)

Fig 15. Coco evaluation output

It returns the AP and AR for various groups of observations partitioned by IOU (Intersection over Union of predictions and ground truth) and Area. So even the official COCO evaluation is not just one metric and there is a good reason for it.

Lets focus on the IoU=0.50:0.95 notation.

What this means is the following: AP and AR is calculated as the average of precisions and recalls calculated for different IoU settings (from 0.5 to  0.95 with a 0.05 step).  What we gain here is a more robust evaluation process, in such a case a model will score high if its pretty good at both (localizing and classifying)

Of course, your problem and dataset might be different. Maybe you need an extremely accurate detector, in that case, choosing AP@0.90IoU might be a good idea.

The downside (of the coco eval tool)  is that by default all the values are averaged for all the classes and all images. This might be fine in a competition-like setup where we want to evaluate the models on all the classes but in real-life situations where you train models on custom datasets (often with fewer classes) you really want to know how your model performs on a per-class basis.

Looking at per-class metrics is extremely valuable, as it might give you important insights:

help you compose a new dataset better

make better decisions when it comes to data augmentation, data sampling etc.

>图15所示。COCO评价输出

>它返回由IOU和区域划分的各种观测组的AP和AR。因此，即使是官方的COCO评估也不仅仅是一种衡量标准，这样做也是有充分理由的。

>让我们关注IoU=0.50 - 0.95的符号。

>这意味着以下内容:AP和AR是根据不同IoU设置(从0.5到0.95，步骤为0.05)计算出的精度和召回率的平均值。我们从中得到的是一个更稳健的评估过程，在这种情况下，如果一个模型在这两方面都做得很好(定位和分类)，它就会得分很高。

>当然，您的问题和数据集可能有所不同。 也许您需要一个非常精确的检测器，在这种情况下，选择AP@0.90IoU可能是一个好主意。

>查看每个类的度量是非常有价值的，因为它可能会给您重要的见解。

>帮助您更好地构建新的数据集
>在数据扩充、数据采样等方面做出更好的决策
 
![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123107208168.png)

Fig 16. Per class AP

Figure 16. gives you a lot of useful informations there are few things you might consider:

Add more data to low performing classes

For classes that score well, maybe you can consider downsampling them  to speed up the training and maybe help with the performance of other less frequent classes.

Spot any obvious correlations for instance classes with small objects performing poorly.

>图16所示每个类的AP

>图16.为您提供了许多有用的信息，您可能需要考虑以下几点：

>向性能较差的类添加更多数据
>对于那些得分较高的类别，或许你可以考虑对它们进行抽样，以加快训练速度，或许还可以帮助提高其他不那么频繁的类别的表现
>找出带有小对象性能差的实例类任何明显的关联

**Visualizing results**

Ok, so if looking at single metrics is not enough what should you do?

I would definitely suggest spending some time on manual results exploration, with the combination of hard metrics from the previous analysis – visualizations will help you get the big picture.

Since exploring predictions of image detection and image segmentation models can get quite messy I would suggest you do it step by step. On the gif below I show how this can be done using the coco inspector tool.

>**可视化结果**

>那么，如果仅看一个指标还不够，该怎么办？

>我绝对建议您花一些时间来进行手动结果探索，并结合之前的分析中的严格指标–可视化将帮助您全面了解。

>由于探索图像检测和图像分割模型的预测会变得非常混乱，因此建议您逐步进行操作。 在下面的gif上，我显示了如何使用coco检查器工具完成此操作。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123336913457.gif)

Fig 17. All the predictions and ground-truths visualized

On the gif we can see how all the important information is visualized:

Red masks – predictions

Orange masks – overlap of predictions and ground truth masks

Green masks – ground truth

Dashed bounding boxes – false positives (predictions without a match)

Orange boxes  true positiveGreen boxes – ground truth

>图17.可视化的所有预测和真实值

>在gif中，我们可以看到所有重要的信息是如何可视化的：

>红色遮罩--预测值
>橙色遮罩--预测和真实值的重叠
>绿色遮罩--真实值
>虚线边框假阳性(没有匹配的预测)
>橙色盒子正类绿色盒子-真实值

**Understanding Results – per image scores**

By looking at the hard metrics and inspecting images visually we most likely have a  pretty good idea of what’s going on. But looking at results of random images (or grouped by class)  is likely not an optimal way of doing this. If you want to really dive in and spot edge cases of your model, I suggest calculating per image metrics (for instance AP or Recall).

Below and example of an image  I found by doing exactly that.

>**看懂结果--每个图像的值**

>通过观察硬指标和检查图像，我们很可能有一个很好的想法，发生了什么。但是观察随机图像(或按类分组)的结果可能不是这样做的最佳方式。如果您真的想深入了解模型的边缘情况，我建议计算每个图像的度量(例如AP或Recall)。

>下面是我找到的一个图片的例子。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123404837450.png)

Fig 18. Image with a very low AP score

In the example above (Fig 18.) we can see two false positive stop sign predictions –  from that we can deduce that our model understands what a stop sign is but not what other traffic signs are.

Perhaps we can add new classes to our dataset or use our “stop sign detector” to label other traffic signs and then create a new “traffic sign” label to overcome this problem.

>图18所示。图像与非常低的AP分数

>在上面的例子中(图18)，我们可以看到两个假阳性停车标志的预测，由此我们可以推断，我们的模型了解什么是停车标志，但不了解其他交通标志。

>也许我们可以在数据集中添加新的类，或者使用我们的停止标志检测器来标记其他交通标志，然后创建一个新的交通标志标签来克服这个问题。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123447314760.png)

Fig 19. Example of an image with a good score > 0.5 AP

Sometimes we will also learn that our model is doing better that it would seem from the scores alone. That’s also useful information, for instance in the example above our model detected a keyboard on the laptop but this is actually not labeled in the original dataset.

>图19.得分大于0.5 AP的图像示例

>有时，我们还会发现我们的模型表现得更好，仅从分数来看就可以了。 这也是有用的信息，例如在我们的模型的上例中，检测到了笔记本电脑上的键盘，但实际上未在原始数据集中标记。

**COCO format**

The way a coco dataset is organized can be a bit intimidating at first.

It consists of a set of dictionaries mapping from one to another.  It’s also intended to be used together with the pycocotools / cocotools  library that builds a rather confusing API on top of the dataset metadata file.

Nonetheless, the coco dataset (and the coco format)  became a standard way of organizing object detection and image segmentation datasets.

In COCO we follow the xywh convention for bounding box encodings or as I like to call it tlwh: (top-left-width-height) that way you can not confuse it with for instance cwh: (center-point, w, h). Mask labels (segmentations) are  run-length  encoded (RLE explanation).

>**COCO格式**

>coco数据集的组织方式一开始可能有点吓人。

>它由一组相互映射的字典组成。 它也打算与pycocotools / cocotools库一起使用，该库在数据集元数据文件的顶部构建了一个令人困惑的API。

>然而，coco数据集(以及coco格式)成为组织目标检测和图像分割数据集的标准方法。

>在COCO中，我们遵循xywh规则来进行边框编码，或者我喜欢称之为tlwh:(左上宽高)，这样你就不会与cwh:(中心点，w, h)相混淆。掩码标签(分段)是行距编码(RLE解释)。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123510923585.png
)

Fig 20. The coco dataset annotations format

There are still very important advantages of having a widely adopted standard:

Labeling tools and services  export and import COCO-like datasets

Evaluation and scoring code (used for the coco competition) is pretty well optimized and battle tested.

Multiple open source datasets follow it.

In the previous paragraph, I used the COCO eval functionality which is another benefit of following the COCO  standard. To take advantage of that you need to format your predictions in the same way as your coco dataset is constructed- then calculating metrics is as simple as calling: COCOeval(gt_dataset, pred_dataset)

>  图20所示。coco数据集注释格式

>拥有一个被广泛采用的标准仍然有非常重要的优势：

>标签工具和服务导出和导入类似coco的数据集。
>评估和评分代码（用于可可比赛）经过了很好的优化和测试。
>随之而来的是多个开源数据集。

>在前一段中，我使用了COCO eval功能，这是遵循COCO标准的另一个好处。要利用这一点，您需要按照构建您的coco数据集的相同方式格式化您的预测——那么计算指标就像调用:COCOeval(gt dataset, pred dataset)一样简单。

**COCO dataset explorer**

In order to streamline the process of data and results exploration (especially for object detection) I wrote a tool that operates on COCO datasets.

Essentially you provide it with the ground truth dataset and the predictions dataset (optionally) and it will do the rest for you:

Calculate most of the metrics I presented in this post

Easily visualize the datasets ground truths and predictions

Inspect coco metrics, per class AP metrics

Inspect per-image scores

>**浏览COCO数据集**

>为了简化数据和结果探索的过程(特别是对象检测)，我编写了一个在COCO数据集上操作的工具。

>实际上，你向它提供真实标记的数据集和预测的数据集(可选)，它将为您完成其余的工作：

>计算我在这篇文章中提出的大部分指标
>轻松可视化数据集的真实值和预测值
>检查coco度量，每个类AP度量
>检查每张图象分数

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599123603165489.gif)

To use COCO dataset explorer tool you need to:

Clone the project repository

>要使用COCO数据集资源管理器工具，您需要：
>克隆项目存储库

```
git clone https://github.com/i008/COCO-dataset-explorer.git

```
Download example data I used for the examples or use your own data in the COCO format:

Example COCO format dataset with predictions.

If you downloaded the example data you will need to extract it.

>下载我用于示例的示例数据，或者使用您自己的COCO格式的数据
>示例COCO格式化数据集与预测。

>如果下载了示例数据，则需要提取它。

```
tar -xvf coco_data.tar
```
Download example data I used for the examples or use your own data in the COCO format:

Example COCO format dataset with predictions.

If you downloaded the example data you will need to extract it.

```
tar -xvf coco_data.tar
```
You should have the following directory structure:

Set up the environment with all the dependencies

>设置带有所有依赖项的环境

```
conda env update;
conda activate cocoexplorer
```
Run streamlit app specifying a file with ground truth and predictions in the COCO format and the image directory:

```
streamlit run coco_explorer.py -- \
    --coco_train coco_data/ground_truth_annotations.json \
    --coco_predictions coco_data/predictions.json  \
    --images_path coco_data/images/
	```
>运行streamlit应用程序，指定一个文件的真实值和预测在COCO格式和图像目录

Note: You can also run this with docker:
>注意:你也可以用docker来运行它

```
sudo docker run -p 8501:8501 -it -v "$(pwd)"/coco_data:/coco_data i008/coco_explorer  \
    streamlit run  coco_explorer.py -- \
    --coco_train /coco_data/ground_truth_annotations.json \
    --coco_predictions /coco_data/predictions.json  \
    --images_path /coco_data/images/
	```
explore the dataset in the browser. By default, it will run on http://localhost:8501/
>在浏览器中浏览数据集。 默认情况下，它将在http：// localhost：8501 /上运行

**Final words**

I hope that with this post I convinced you that data exploration in object detection and image segmentation is as important as in any other branch of machine learning.

I’m confident that the effort we make at this stage of the project pays off in the long run.

The knowledge we gather allows us to make better-informed modeling decisions, avoid multiple training pitfalls and gives you more confidence in the training process, and the predictions your model produces.

This article was originally written by Jakub Cieślik and posted on the Neptune blog. You can find more in-depth articles for machine learning practitioners there.

>**结语**

>我希望通过这篇文章让你相信，在目标检测和图像分割中的数据探索与机器学习的其他分支一样重要。

>我相信我们在这个项目现阶段所做的努力最终会得到回报的。

>我们收集的知识使我们能够做出更明智的建模决策，避免多个训练陷阱，并使您对训练过程和模型产生的预测更有信心。

>本文最初由JakubCieślik撰写并发布在Neptune博客上。 您可以在此处找到针对机器学习从业人员的更多深入文章。

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://www.flyai.com/article/703

讨论

500字

表情

每日优质讨论奖励 20FAI

发送

每日优质讨论奖励 20FAI

删除确认

是否删除该条评论？

取消删除