为什么Python中的串联速度越来越慢？

为什么在某些情况下,Python 3中的连接似乎比Python 2中的连接慢？

影响最大的串联方法似乎是字节对象的连续串联,从O(n)到O(n²)操作.

我的分析代码大部分在这里：

#!/usr/bin/env python

from operator import concat
from sys import version, version_info
from timeit import timeit  # Compatibility: ver >= 2.6

# ver = version.partition('\n')[0].rstrip()
ver = '.'.join(str(v) for v in version_info[:3])
print(ver)

if version_info[0] == 2:
    from StringIO import StringIO
else:
    from io import StringIO
    from functools import reduce
    xrange = range


def build_plus():
    output = ''
    for _ in xrange(input_len):
        output += 'a'
    return output


def build_join():
    return ''.join('a' for _ in xrange(input_len))


def build_bytes_plus():
    output = b''
    for _ in xrange(input_len):
        output += b'a'
    return output


def build_stringio():
    output = StringIO()
    for _ in xrange(input_len):
        output.write('a')
    return output.getvalue()


def build_reduce():
    return reduce(concat, ('a' for _ in xrange(input_len)))


builds = {'str+': build_plus,
          'join': build_join,
          'reduce': build_reduce,
          'bytes+': build_bytes_plus,
          'StringIO': build_stringio}

if version_info[0] == 2:
    import cStringIO

    def build_cstringio():
        output = cStringIO.StringIO()
        for _ in xrange(input_len):
            output.write('a')
        return output.getvalue()

    builds['cStringIO'] = build_cstringio
else:
    from io import BytesIO

    def build_bytesio():
        output = BytesIO()
        for _ in xrange(input_len):
            output.write(b'a')
        return output.getvalue()

    builds['BytesIO'] = build_bytesio

resfile = open('times.csv', 'a')
size_range = 50  # Number of points over the size axis
min_order = 1.0  # 10^x byte input min
max_order = 5.0  # 10^x byte input max

for allow_gc in (False, True):
    setup = 'gc.enable()' if allow_gc else 'pass'

    for build_name, build_fun in builds.items():
        # For a roughly constant confidence interval, aim for uniform sample density across the
        # (logarithmic) input size axis.
        for size_index in range(size_range+1):
            input_len = int(10**((max_order-min_order)*size_index/size_range + min_order))

            # Rather than repeating many measurements at one input size, perform one measurement
            # per input size for a continuous range of input sizes and apply smoothing later.
            dur = timeit(build_fun, setup, number=1)

            resfile.write('"%s",%s,"%s",%d,%.6g\n' % (ver, str(allow_gc).upper(), build_name,
                                                      input_len, dur))

我的R脚本中的一些图形如下所示：

解决方法:

在循环中用or =连接字符串从来都不是一个好主意.这似乎很有效,因为字节码解释器循环中有一个weird, controversial special case,如果它能证明没有其他人引用正在混乱的字符串,则它将尝试以可变方式连接字符串.没有有效的调整大小政策；它只是called realloc并希望达到最佳状态,因此如果需要复制realloc,它仍然可能以O(n ^ 2)结尾.

在Python 3中,该weird special case现在可以处理unicode字符串而不是字节字符串.字节串连接每次都返回建立新的字符串对象,因此循环返回到O(n ^ 2).

标签：concatenation,profiling,python
来源： https://codeday.me/bug/20191025/1926097.html