如何将 Pandas 速度加快100倍？-FlyAI

如何将 Pandas 速度加快100倍？

深度学习机器学习

最后更新 2021-01-06 16:27 阅读 149

最后更新 2021-01-06 16:27

阅读 149

深度学习机器学习

##FlyAI文献翻译

**英文原文：**[：How to make your Pandas operation 100x faster
](https://towardsdatascience.com/how-to-make-your-pandas-operation-100x-faster-81ebcd09265c)

> **标签** ：数据挖掘应用

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608878841177539.png)

Photo by Matthew Brodeur on Unsplash

> Matthew Brodeur摄于Unsplash

Pandas is a great tool for exploring and working with data. As such, it is deliberately optimized for versatility and ease of use, instead of performance. There are often many different ways to do the exact same operation, some of which are far more performant than others. This is not a problem if your data is relatively small or your transformation is relatively simple, but it can quickly becomes a bottleneck as the scale and complexity increase. In this article, I will walk through a few different approaches to optimizing the performance ranging from vectorization to parallelization.

Let’s start by making this issue more concrete with an example.

A common operation when using Pandas is to creat a new derived value based on one or more existing columns. The examples below (inspired by [1]) illustrates the various different ways of doing this using Pandas, and measure the resulting performance using the magic command in Jupyter notebooks.%timeit

>Pandas 是探索和处理数据的好工具。因此，它被有意地优化为通用性和易用性，而不是性能。通常有许多不同的方法来完成完全相同的操作，其中一些比其他的性能要好得多。如果数据相对较小或转换相对简单，这就不是问题，但随着规模和复杂性的增加，这可能很快成为瓶颈。在本文中，我将介绍几种不同的性能优化方法，从矢量化到并行化。

>让我们首先通过一个示例使这个问题更加具体。

>使用Pandas时的一个常见操作是根据一个或多个现有列创建新的派生值。下面的例子(受[1]启发)说明了使用Pandas完成此任务的各种不同方法，并使用jupiter notebook .%timeit中的magic命令测量结果性能

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608878819522267.png)

###The first approach
```
[sum_square(row[0], row[1]) for _, row in df.iterrows()]
```
uses list comprehension along with the method , and is the slowest by a long shot. This is because it is effectively using a simple for loop and incurring the heavy overhead of using the pandas series object in each iteration. It is rarely necessary to use series object in your transformation function, so you should almost never use the method.iterrowsiterrows

>与方法一起使用列表理解，是最慢的方法。这是因为它有效地使用了简单的for循环，并在每次迭代中使用pandas系列对象带来了沉重的开销。在转换函数中很少需要使用series object，所以你几乎不应该使用 iterrows  iterrows

###The second approach
```
[sum_square(a, b) for a, b in df[[0, 1]].itertuples(index=False)]
```
replaces the method of the first approach with method. The improvement in performance is a whopping 60x (575ms -> 9.53ms). This is because the method bypasses the overhead associated with the Pandas series object and uses the simple tuples instead. If you need to loop through a Pandas dataframe, you should almost always instead of the iterrowsitertuplesitertuplesitertuplesiterrows

>用method替换第一种方法的方法。性能的提升是惊人的60倍(575ms -> 9.53ms)。这是因为该方法绕过了与Pandas系列对象相关的开销，而是使用简单的元组。如果你需要循环使用Pandas数据帧，你应该总是使用iterrows  itertuples  itertuples  itertuples iterrows

###The third approach
```
df.apply(lambda row: sum_square(row[0], row[1]), axis=1 )
```

uses the method instead of list comprehension to apply the function to each row of the dataframe. It still incurs the overhead of the Pandas series object when working with each row, but is 6x faster than the first approach because it is able to leverage some more efficient pre-compiled looping methods underneath the hood. It is, however, still effectively a for loop, just more efficient way to do so.apply

>使用该方法而不是列表理解将函数应用到数据帧的每一行。在处理每一行时，它仍然会招致Pandas series对象的开销，但比第一种方法快6倍，因为它能够在底层利用一些更有效的预编译循环方法。然而，它仍然是一个有效的for循环，只是更有效的方法   apply

###The fourth approach

```
df.apply(lambda row: sum_square(row[0], row[1]), raw=True, axis=1 )
```
is able to achieve a 4x speed up relative to the third approach, with a very simple parameter tweak in adding . This is telling the method to bypass the overhead associated with the Pandas series object and use simple map objects instead. It is interesting note that this is still slower than the second approach with raw=Trueapplyitertuples

>与第三种方法相比，能够实现4倍的速度，只需在添加中进行非常简单的参数调整。这告诉该方法绕过与Pandas系列对象相关的开销，而使用简单的map对象。有趣的是，这仍然比第二种方法慢  raw=True    apply     itertuples

###The fifth approach

```
np.vectorize(sum_square)(df[0], df[1])
```
uses to attempt to make the function provided more “vectorized”. This basically means it tries to leverage pre-compiled and native methods that are SIMD (Single Instruction Multiple Data) and significantly faster [2], as much as possible. Depending on the function, it may or may not actually yield a speed up, but it never hurts to try given the relative simplicity. In this particularly example, it worked quite well and yielded a 171x speed up relative to the first approach. It is even 3x faster than the itertuple method in the second approach.np.vectorize

>用于尝试使提供的函数更“向量化”。这基本上意味着它尽量利用预编译和本地方法，如SIMD(单指令多数据)和显著更快的[2]。根据函数的不同，它可能会也可能不会实际产生加速，但考虑到相对的简单性，尝试一下也不会有坏处。在这个特别的例子中，它工作得很好，相对于第一种方法产生了171x的速度。它甚至比第二种方法中的itertuple方法快3倍。  np.vectorize

###The last approach

````
np.power(df[0] + df[1], 2)
```

fully takes advantage of the vectorized native Numpy methods and is 1888x faster than the first approach we started with. Just like the method, it may not always be possible (or clear as to how) to vectorize the given function, but when you can you should definitely try. A good rule of thumb for assessing whether you can vectorize the function is if you can represent it as linear algebra operations on matrices and scalar values. If you can, you can likely find native Numpy methods to vectorize those operations.np.vectorize

What if your function cannot be vectorized or you don’t know how to?

There are transformations of the data that simply cannot be vectorized, and therefore cannot benefit from the dramatic improvements that vectorization provided above (but you should still never use ). One example of this type of transformation is regex operations on strings like extracting the top level domain from URLs. In these cases, there are a few different approaches like compiling your function using Cython or run-time compilers like Numba, or using parallel processing. I prefer parallel processing because it requires the least amount of rewrite of your existing code. You simply have to add a few lines wrapper code to make it work. The example below illustrates how you can do thisiterrows

>充分利用了向量化的本机Numpy方法，比我们开始使用的第一种方法快1888x。就像这个方法一样，它可能不总是可能(或清楚如何)向量化给定的函数，但如果你可以，你一定要尝试。评估是否可以将函数向量化的一个很好的经验法则是，是否可以将其表示为矩阵和标量值上的线性代数运算。如果可以，可以找到原生Numpy方法来对这些操作进行向量化

>如果你的函数不能向量化或者你不知道怎么做呢?

>有些数据的转换是无法向量化的，因此无法从上述向量化提供的显著改进中获益(但您仍然不应该使用)。这种类型转换的一个例子是对字符串的regex操作，比如从url中提取顶级域。在这些情况下，有几种不同的方法，比如使用Cython或运行时编译器(如Numba)编译函数，或者使用并行处理。我更喜欢并行处理，因为它只需要对现有代码进行最少的重写。您只需添加几行包装器代码就可以使它工作。下面的例子说明了如何做到这一点  iterrows

```
import concurrent.futures
import numpy as np
import pandas as pd
from functools import partial
import time
df = pd.DataFrame(np.random.binomial(n=1000, p=0.2, size=(10000,2)))
# a no op function that simulates a transformation that takes some time to complete
def do_work(row, duration):
    time.sleep(duration)
    return True
FUNCTION_RUN_TIME = 0.0001
# simple use of apply to execute the function against the pd dataframe
def serial_calc(df, duration):
    apply_partial = partial(do_work, duration=duration)
    df['result'] = df.apply(apply_partial, axis=1)
    return df
%timeit serial_calc(df, FUNCTION_RUN_TIME)
# simple wrapper code around serial_calc to parallelize the workdef parallel_calc(df, func, n_core, duration):
    futs = []
    df_split = np.array_split(df, n_core)
#     pool = concurrent.futures.ThreadPoolExecutor(max_workers = n_core)
    pool = concurrent.futures.ProcessPoolExecutor(max_workers = n_core)
    apply_partial = partial(func, duration=duration)
    return pd.concat(pool.map(apply_partial, df_split))
%timeit parallel_calc(df, serial_calc, 32, FUNCTION_RUN_TIME)
```

```
1.42 s ± 9.58 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
184 ms ± 776 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
function_time = []non_parallel_time = []parallel_time = []for d in range(11):#     print(d)
    d_ms = d/20000
    function_time.append(d_ms)
    start = time.time()
    serial_calc(df, d_ms)
    end = time.time()
    non_parallel_time.append(end-start)
    
    start = time.time()
    parallel_calc(df, serial_calc, 32, d_ms)
    end = time.time()
    parallel_time.append(end-start)
    import matplotlib.pyplot as plt
    font = {'family' : 'Arial',
        'weight': 'normal',
        'size'   : 20
    }plt.rc('font', **font)
    fig, ax = plt.subplots(figsize=(14,8))
    plt.xlabel('Function run time (ms)')
    plt.ylabel('Overall processing time (s)')
    x = [t*1000 for t in function_time]
    ax.plot(x, non_parallel_time, '-', linewidth=1)
    ax.plot(x, parallel_time, '-', linewidth=1)
    ax.legend(['Serial Apply', 'Parallelized Apply'])
    plt.show()
```

As you can see with relatively little effort and no modification to the existing code, we were able to achieve a 7x speed up to the code by leveraging parallel processing. It is worth noting that this improvement increases with data scale and function run time, as can be seen in the graph below, where I varied simulated function run time from 0 to 0.5ms. The serial implementation processing time scales at a much higher rate than parallel implementation.

>正如您所看到的，只需相对较少的工作，并且无需修改现有代码，我们就能够通过利用并行处理来实现7倍的代码速度。值得注意的是，这种改进随着数据规模和函数运行时间的增加而增加，如下图所示，我将模拟函数运行时间从0更改为0.5ms。串行实现处理时间比并行实现速度快得多。

![](https://static.leiphone.com/uploads/new/sns/article/202012/1608878819673692.png)

A few important consideration to keep in mind when using parallel processing

Overhead — parallel processing incurs quite a bit of overhead, and as a result it is not guaranteed to be faster than serial processing if you function is relatively fast and your data is not super large. For the example above (10k rows), we can see that parallel processing overtakes serial processing, at around 0.01ms function run time.

CPU bound vs I/O bound — If your function is CPU bound, meaning that the transformation logic just takes a long time and it is not waiting for data, then you should use multiprocess (or process pool). If your function is I/O bound, meaning that it is spending a lot of time waiting for data (e.g. making api requests over the internet), then multithreading (or thread pool) will be the better and faster option. [This article](https://realpython.com/python-concurrency/ "This article") [3] provides a really good discussion on differences between the two.

>在使用并行处理时，需要记住几个重要的注意事项

>开销——并行处理会带来相当大的开销，因此，如果您的功能相对较快，且数据不是特别大，那么并行处理不能保证比串行处理更快。对于上面的示例(10k行)，我们可以看到并行处理超过了串行处理，大约在0.01ms函数运行时。

>CPU绑定vs I/O绑定——如果你的函数是CPU绑定的，这意味着转换逻辑需要很长时间，它不需要等待数据，那么你应该使用多进程(或进程池)。如果你的函数是I/O绑定的，这意味着它要花费大量的时间来等待数据(例如，通过internet发出api请求)，那么多线程(或线程池)将是更好更快的选择。这篇[3]文章很好地讨论了两者之间的区别。

Takeaways
Whenever possible write your transformation functions using native pre-compiled SIMD methods (most Numpy and many Pandas native methods are)

If you are not sure how to vectorize yourself, try but your mileage may varynp.vectorize,

Use or for looping, never use unless you know exactly why you must use itdf.applydf.itertuplesdf.iterrows

If your data is large and transformation is slow and cannot be vectorized easily, parallel processing is an easy way to boost performance

>需要掌握的主要词语

>只要可能，使用本机预编译SIMD方法(大多数Numpy和许多Pandas本机方法)编写转换函数

>如果你不确定如何向量化自己，尝试一下，但效果可能会有所不同  np.vectorize,

>对于循环，除非你确切知道为什么你必须使用它，否则不要使用它

>df.apply df.itertuples  df.iterrows
如果数据很大，转换很慢，而且不容易向量化，那么并行处理是提高性能的一种简单方法

###参考文献：

[1]https://stackoverflow.com/questions/52673285/performance-of-pandas-apply-vs-np-vectorize-to-create-new-column-from-existing-c

[2] https://towardsdatascience.com/decoding-the-performance-secret-of-worlds-most-popular-data-science-library-numpy-7a7da54b7d72

[3] https://realpython.com/python-concurrency/

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可，转载请附上原文出处链接和本声明。
本文链接地址：https://www.flyai.com/article/799

讨论

500字

表情

每日优质讨论奖励 20FAI

发送

每日优质讨论奖励 20FAI

删除确认

是否删除该条评论？

取消删除