计算机视觉：为什么AI感知与人类感知无法直接比较?-FlyAI

计算机视觉：为什么AI感知与人类感知无法直接比较?

图像分类

最后更新 2020-09-09 11:41 阅读 219

最后更新 2020-09-09 11:41

阅读 219

图像分类

##FlyAI文献翻译
**英文原文：**[Computer vision: Why it’s hard to compare AI and human perception](https://bdtechtalks.com/2020/08/10/computer-vision-deep-learning-vs-human-perception/)
> **标签**：图像识别

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599036491445351.jpg)

This article is part of our reviews of AI research papers, a series of posts that explore the latest findings in artificial intelligence.

Human-level performance. Human-level accuracy. Those are terms you hear a lot from companies developing artificial intelligence systems, whether it’s facial recognition, object detection, or question answering. And to their credit, the recent years have seen many great products powered by AI algorithms, mostly thanks to advances in machine learning and deep learning.

But many of these comparisons only take into account the end-result of testing the deep learning algorithms on limited data sets. This approach can create false expectations about AI systems and yield dangerous results when they are entrusted with critical tasks.

> 本文是我们对人工智能研究论文评论的一部分，这是一个探索人工智能最新研究成果的系列文章。

> 人类层面的性能。人类级的准确性。这些术语你经常从开发人工智能系统的公司那里听到，无论是面部识别、物体检测还是问题回答。而值得称道的是，最近几年看到了许多由人工智能算法驱动的优秀产品，这主要得益于机器学习和深度学习的进步。

> 但其中很多比较只考虑到在有限的数据集上测试深度学习算法的最终结果。这种方法可能会对人工智能系统产生错误的期望，当它们被委托执行关键任务时，会产生危险的结果。

In a recent study, a group of researchers from various German organizations and universities have highlighted the challenges of evaluating the performance of deep learning in processing visual data. In their paper, titled, “The Notorious Difficulty of Comparing Human and Machine Perception,” the researchers highlight the problems in current methods that compare deep neural networks and the human vision system.

In their research, the scientist conducted a series of experiments that dig beneath the surface of deep learning results and compare them to the workings of the human vision system. Their findings are reminder that we must be cautious when comparing AI to humans, even if it shows equal or better performance on the same task.

>在最近的一项研究中，一组来自德国不同组织和大学的研究人员强调了评估深度学习在处理视觉数据方面的性能的挑战。在他们题为 "比较人类和机器感知的臭名昭著的困难 "的论文中，研究人员强调了当前比较深度神经网络和人类视觉系统的方法中存在的问题。

>在研究中，科学家们进行了一系列实验，挖掘深度学习结果的表面之下，并将其与人类视觉系统的工作原理进行比较。他们的研究结果提醒我们，在将人工智能与人类进行比较时，我们必须谨慎，即使它在相同的任务上表现出同等或更好的性能。

###The complexity of human and computer vision

In the seemingly endless quest to reconstruct human perception, the field that has become known as computer vision, deep learning has so far yielded the most favorable results. Convolutional neural networks (CNN), an architecture often used in computer vision deep learning algorithms, are accomplishing tasks that were extremely difficult with traditional software.

However, comparing neural networks to the human perception remains a challenge. And this is partly because we still have a lot to learn about the human vision system and the human brain in general. The complex workings of deep learning systems also compound the problem. Deep neural networks work in very complicated ways that often confound their own creators.

In recent years, a body of research has tried to evaluate the inner workings of neural networks and their robustness in handling real-world situations. “Despite a multitude of studies, comparing human and machine perception is not straightforward,” the German researchers write in their paper.

In their study, the scientists focused on three areas to gauge how humans and deep neural networks process visual data.

>###人类和计算机视觉的复杂性

>在重构人类感知这一看似无止境的探索中，在被称为计算机视觉的领域，深度学习迄今为止取得了最有利的成果。卷积神经网络（CNN）是计算机视觉深度学习算法中经常使用的一种架构，它正在完成传统软件极其困难的任务。

>然而，将神经网络与人类的感知进行比较仍然是一个挑战。而这部分是因为我们对人类视觉系统和整个人类大脑仍有很多需要了解的地方。深度学习系统的复杂工作原理也使问题更加复杂。深度神经网络的工作方式非常复杂，经常会让自己的创造者感到困惑。

>近年来，一系列研究试图评估神经网络的内部工作原理及其在处理实际情况时的鲁棒性。"尽管有大量的研究，但比较人类和机器的感知并不直接。"德国研究人员在论文中写道。

>在他们的研究中，科学家们专注于三个领域，以衡量人类和深度神经网络如何处理视觉数据。

###How do neural networks perceive contours?

The first test involves contour detection. In this experiment, both humans and AI participants must say whether an image contains a closed contour or not. The goal here is to understand whether deep learning algorithms can learn the concept of closed and open shapes, and whether they can detect them under various conditions.

>###神经网络是如何感知轮廓的？

>第一个测试涉及轮廓检测。在这个实验中，人类和人工智能参与者都必须说出一幅图像是否包含一个封闭的轮廓。这里的目标是了解深度学习算法是否可以学习封闭和开放形状的概念，以及它们是否可以在各种条件下检测它们。

![Can you tell which one of the above images contains a closed shape?](https://static.leiphone.com/uploads/new/sns/article/202009/1599036511169667.png "Can you tell which one of the above images contains a closed shape?")

“For humans, a closed contour flanked by many open contours perceptually stands out. In contrast, detecting closed contours might be difficult for DNNs as they would presumably require a long-range contour integration,” the researchers write.

For the experiment, the scientists used the ResNet-50, a popular convolutional neural network developed by AI researchers at Microsoft. They used transfer learning to finetune the AI model on 14,000 images of closed and open contours.

They then tested the AI on various examples that resembled the training data and gradually shifted in other directions. The initial findings showed that a well-trained neural network seems to grasp the idea of a closed contour. Even though the network was trained on a dataset that only contained shapes with straight lines, it could also performed well on curved lines.

>"对于人类来说，一个封闭的轮廓两侧有许多开放的轮廓，在感知上很突出。相比之下，检测封闭轮廓对于DNNs来说可能是困难的，因为它们大概需要长距离的轮廓整合。"研究人员写道。

>在实验中，科学家们使用了ResNet-50，这是微软的AI研究人员开发的一种流行的卷积神经网络。他们使用转移学习在14000张封闭和开放轮廓的图像上对AI模型进行微调。

>然后，他们在各种与训练数据相似的例子上测试了AI，并逐渐向其他方向转移。初步研究结果表明，一个训练有素的神经网络似乎能够掌握封闭轮廓的概念。即使该网络是在一个只包含直线形状的数据集上进行训练的，但它也可以在曲线上表现良好。

“These results suggest that our model did, in fact, learn the concept of open and closed contours and that it performs a similar contour integration-like process as humans,” the scientists write.
>"这些结果表明，我们的模型事实上确实学习了开放和封闭轮廓的概念，并且它执行了与人类类似的轮廓整合过程，"科学家写道。

![The ResNet neural network was able to detect various open- and closed-contour images, despite having been trained on straight-line examples only.](https://static.leiphone.com/uploads/new/sns/article/202009/1599036545722331.png "The ResNet neural network was able to detect various open- and closed-contour images, despite having been trained on straight-line examples only.")
However, further investigation showed that other changes that didn’t affect human performance degraded the accuracy of the AI model’s results. For instance, changing the color and width of the lines caused a sudden drop in the accuracy of the deep learning model. The model also seemed to struggle with detecting shapes when they became larger than a certain size.
>然而，进一步的调查显示，其他不影响人类性能的变化降低了人工智能模型结果的准确性。例如，改变线条的颜色和宽度会导致深度学习模型的准确性突然下降。当形状变得大于一定尺寸时，该模型似乎也难以检测到它们。

![The ResNet-50 neural network struggled when presented with images that contained lines of different colors and thickness, and where shapes were larger than the training set.](https://static.leiphone.com/uploads/new/sns/article/202009/1599036565883222.png "The ResNet-50 neural network struggled when presented with images that contained lines of different colors and thickness, and where shapes were larger than the training set.")

The neural network was also very sensitive to adversarial perturbations, carefully crafted changes that are imperceptible to the human eye but cause disruption in the behavior of machine learning systems.
>神经网络对对抗性扰动也非常敏感，这些精心设计的变化是人眼无法察觉的，但会对机器学习系统的行为造成干扰。

![The image on the right has been modified with adversarial perturbations, noise that is imperceptible by humans. To the human eye, both images are identical. But to a neural network, they are different images.](https://static.leiphone.com/uploads/new/sns/article/202009/1599036592435488.png "The image on the right has been modified with adversarial perturbations, noise that is imperceptible by humans. To the human eye, both images are identical. But to a neural network, they are different images.")
To further investigate the decision-making process of the AI, the scientists used a Bag-of-Feature network, a technique that tries to localize the bits of data that contribute to the decision of a deep learning model. The analysis proved that “there do exist local features such as an endpoint in conjunction with a short edge that can often give away the correct class label,” the researchers found.

>为了进一步研究人工智能的决策过程，科学家们使用了Bag-of-Feature网络，这种技术试图将有助于深度学习模型决策的数据位进行本地化。分析证明，"确实存在局部特征，比如端点与短边的结合，往往可以泄露正确的类标签，"研究人员发现。

###Can machine learning reason about images?
The second experiment tested the abilities of deep learning algorithms in abstract visual reasoning. The data used for the experiment is based on the Synthetic Visual Reasoning Test (SVRT), in which the AI must answer questions that require understanding of the relations between different shapes in the picture. The tests include same-different tasks (e.g., are two shapes in a picture identical?) and spatial tasks (e.g., is the smaller shape in the center of the larger shape?). A human observer would easily solve these problems.
>###机器学习能对图像进行推理吗？

>第二个实验测试了深度学习算法在抽象视觉推理方面的能力。实验使用的数据基于合成视觉推理测试（SVRT），人工智能必须回答需要理解图片中不同形状之间关系的问题。测试包括同异任务(例如，图片中的两个形状是否相同?)和空间任务(例如，较小的形状是否在较大形状的中心?)。人类观察者会很容易解决这些问题。

![The SVRT challenge requires the participating AI to solve same-different and spatial tasks.](https://static.leiphone.com/uploads/new/sns/article/202009/1599036615730912.png "The SVRT challenge requires the participating AI to solve same-different and spatial tasks.")

For their experiment, the researchers use the ResNet-50 and tested how it performed with different sizes of training dataset. The results show that a pretrained model finetuned on 28,000 samples performs well both on same-different and spatial tasks. (Previous experiments trained a very small neural network on a million images.) The performance of the AI dropped as the researchers reduced the number of training examples, but degradation in same-different tasks was faster.

“Same-different tasks require more training samples than spatial reasoning tasks,” the researchers write, adding, “this cannot be taken as evidence for systematic differences between feed-forward neural networks and the human visual system.”

The researchers note that the human visual system is naturally pre-trained on large amounts of abstract visual reasoning tasks. This makes it unfair to test the deep learning model on a low-data regime, and it is almost impossible to draw solid conclusions about differences in the internal information processing of humans and AI.

“It might very well be that the human visual system trained from scratch on the two types of tasks would exhibit a similar difference in sample efficiency as a ResNet-50,” the researchers write.

>在实验中，研究人员使用ResNet-50，并测试了它在不同规模的训练数据集上的表现。结果显示，在28000个样本上微调的预训练模型在同异和空间任务上都表现良好。之前的实验在100万张图片上训练了一个非常小的神经网络）。随着研究人员减少训练例子的数量，人工智能的性能下降，但在同差任务中的退化更快。

>"与空间推理任务相比，同异任务需要更多的训练样本，"研究人员写道，"这不能作为前馈神经网络和人类视觉系统之间系统性差异的证据。"

>研究人员指出，人类视觉系统天然地在大量的抽象视觉推理任务上进行了预训练。这使得在低数据制度上测试深度学习模型是不公平的，几乎不可能对人类和人工智能内部信息处理的差异得出可靠的结论。

>研究人员写道："很可能是，在这两类任务上从头训练的人类视觉系统会表现出与ResNet-50相似的样本效率差异"。

###Measuring the recognition gap of deep learning

The recognition gap is one of the most interesting tests of visual systems. Consider the following image. Can you tell what it is without scrolling further down?

>衡量深度学习的识别度差距

>识别差距是视觉系统最有趣的测试之一。考虑以下图片。如果不继续往下滚动，你能判断出它是什么吗？

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599036637824608.jpg)
Below is the zoomed-out view of the same image. There’s no question that it’s a cat. If I showed you a close-up of another part of the image (perhaps the ear), you might have had a greater chance of predicting what was in the image. We humans need to see a certain amount of overall shapes and patterns to be able to recognize an object in an image. The more you zoom in, the more features you’re removing, and the harder it becomes to distinguish what is in the image.

>下图是同一张图片的放大图。毫无疑问，这是一只猫。如果我给你看图像的另一部分特写（也许是耳朵），你可能会有更大的机会预测出图像中的东西。我们人类需要看到一定数量的整体形状和图案，才能识别图像中的物体。你放大得越多，去除的特征就越多，就越难分辨图像中的东西。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599036655914653.jpg)
Depending on the features they contain, close-up views of different sections of the cat image have different effects on our perception.

Deep learning systems also operate on features, but they work in subtler ways. Neural networks sometimes the find minuscule features that are imperceptible to the human eye but remain detectable even when you zoom in very closely.

In their final experiment, the researchers tried to measure the recognition gap of deep neural networks by gradually zooming in images until the accuracy of the AI model started to degrade considerably.

>根据它们所包含的特征，猫咪图像的不同部分的特写视图对我们的感知有不同的影响。

>深度学习系统也对特征进行操作，但它们的工作方式更微妙。神经网络有时会发现一些微不足道的特征，这些特征对人眼来说是无法察觉的，但即使你把图像放大到很近的地方，仍然可以检测到。

>在最后的实验中，研究人员试图通过逐渐放大图像来测量深度神经网络的识别差距，直到人工智能模型的准确性开始大幅下降。

Previous experiments show a large difference between the image recognition gap in humans and deep neural networks. But in their paper, the researchers point out that most previous tests on neural network recognition gaps are based on human-selected image patches. These patches favor the human vision system.

When they tested their deep learning models on “machine-selected” patches, the researchers obtained results that showed a similar gap in humans and AI.

>之前的实验显示，人类和深度神经网络的图像识别差距存在较大差异。但研究人员在论文中指出，之前大多数关于神经网络识别差距的测试都是基于人类选择的图像补丁。这些补丁有利于人类视觉系统。

>当他们在 "机器选择 "的补丁上测试他们的深度学习模型时，研究人员得到的结果显示，人类和人工智能也存在类似的差距。

![](https://static.leiphone.com/uploads/new/sns/article/202009/1599036671449201.png)

The recognition gap test evaluates how zooming in on an image affects the precision of an AI’s

“These results highlight the importance of testing humans and machines on the exact same footing and of avoiding a human bias in the experiment design,” the researchers write. “All conditions, instructions and procedures should be as close as possible between humans and machines in order to ensure that all observed differences are due to inherently different decision strategies rather than differences in the testing procedure.”

>识别差距测试评估了放大图像对AI的精度的影响。

>"这些结果强调了在完全相同的基础上测试人类和机器的重要性，以及避免实验设计中的人类偏见，"研究人员写道。"所有的条件、指令和程序都应该在人类和机器之间尽可能地接近，以确保所有观察到的差异都是由于本质上不同的决策策略而不是测试程序的差异。"

###Closing the gap between AI and human intelligence
As our AI systems become more complex, we will have to develop more complex methods to test them. Previous work in the field shows that many of the popular benchmarks used to measure the accuracy of computer vision systems are misleading. The work by the German researchers is one of many efforts that attempt to measure artificial intelligence and better quantify the differences between AI and human intelligence. And they draw conclusions that can provide directions for future AI research.

“The overarching challenge in comparison studies between humans and machines seems to be the strong internal human interpretation bias,” the researchers write. “Appropriate analysis tools and extensive cross checks – such as variations in the network architecture, alignment of experimental procedures, generalization tests, adversarial examples and tests with constrained networks – help rationalizing the interpretation of findings and put this internal bias into perspective. All in all, care has to be taken to not impose our human systematic bias when comparing human and machine perception.”

>###缩小人工智能与人类智能之间的差距

>随着我们的人工智能系统变得越来越复杂，我们将不得不开发更复杂的方法来测试它们。该领域以前的工作表明，许多用于衡量计算机视觉系统准确性的流行基准都具有误导性。德国研究人员的工作是众多试图衡量人工智能并更好地量化人工智能与人类智能之间差异的努力之一。而他们得出的结论，可以为未来的人工智能研究提供方向。

>"人类和机器之间比较研究的首要挑战似乎是人类内部强烈的解释偏差，"研究人员写道。"适当的分析工具和广泛的交叉检查--比如网络架构的变化、实验程序的调整、泛化测试、对抗性的例子和受限网络的测试--有助于合理化解释研究结果，并将这种内部偏见纳入视野。总而言之，在比较人类和机器的感知时，必须注意不要将我们人类的系统性偏见强加于人。"

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://www.flyai.com/article/697

讨论

500字

表情

每日优质讨论奖励 20FAI

发送

每日优质讨论奖励 20FAI

删除确认

是否删除该条评论？

取消删除