提供maxsplit时string.split对短字符串的行为

Question

I recently ran across some intriguing behavior of the string.split method in python2.7, particularly with respect to short stings (less than around 25 chars, see below), that involves contrasting the behavior of:

# Without maxplsit
my_string.split('A')

to

# With maxsplit=1
my_string.split('A', 1)

The second method is actually slower for short strings, and I'm quite curious as to why.

The Test

This first came about from a small call to timeit that my co-worker discovered:

# Without maxsplit
$ python -m timeit -s "json_line='a|b|c'" "part_one='|'.split(json_line)[0]"
1000000 loops, best of 3: 0.274 usec per loop
# With maxsplit
$ python -m timeit -s "json_line='a|b|c'" "part_one='|'.split(json_line,1)[0]"
1000000 loops, best of 3: 0.461 usec per loop

I thought this was certainly curious, so I put together a more detailed test. First I wrote the following small function that generates random strings of a specified length consisting of the first ten capital letters:

from random import choice

# 'A' through 'J'
choices = map(chr, range(65, 75))

def make_random_string(length):
    return ''.join(choice(choices) for i in xrange(length))

Then I wrote a couple tester functions to repeatedly split and time randomly generated strings of a specified length:

from timeit import timeit

def time_split_of_size(str_length, n_strs_to_split):
    times = []
    data = [make_random_string(str_length) for i in xrange(n_strs_to_split)]
    for s in data:
        t = timeit("'{s}'.split('A')".format(s=s),
                   setup="from __main__ import make_random_string",
                   number=1000)
        times.append(t)
    return times

def time_split_of_size_with_maxcount(str_length, n_strs_to_split):
    times = []
    data = [make_random_string(str_length) for i in xrange(n_strs_to_split)]
    for s in data:
        t = timeit("'{s}'.split('A', 1)".format(s=s),
                   setup="from __main__ import make_random_string",
                   number=1000)
        times.append(t)
    return times

I then ran these testing methods over strings of varying sizes:

from collections import OrderedDict
d = OrderedDict({})
for str_length in xrange(10, 10*1000, 25):
    no_maxcount = mean(time_split_of_size(str_length, 20))
    with_maxcount = mean(time_split_of_size_with_maxcount(str_length, 20))
    d[str_length] = [no_maxcount, with_maxcount]

This gives you the behavior you would expect, O(1) for the method with maxsplit=1 and O(n) for splitting all the way. Here's a plot of the time by the length of the string, the barely visible green curve is with maxsplit=1 and the blue curve is without:

None the less, the behavior my co-worker discovered for small stings is real. Here's some code that times many splits of short stings:

from collections import OrderedDict
d = OrderedDict({})
for str_length in xrange(1, 50, 2):
    no_maxcount = mean(time_split_of_size(str_length, 500))
    with_maxcount = mean(time_split_of_size_with_maxcount(str_length, 500))
    d[str_length] = [no_maxcount, with_maxcount]

With the following results:

It seems like there is some overhead for strings less than 25 or so characters in length. The shape of the green curve is also quite curious, how it increases parallel to the blue before leveling off permenantly.

I took a look at the source code, which you may find here:

stringobject.c (line 1449) stringlib/split.h (line 105)

but nothing obvious jumped out at me.

Any idea what is causing the overhead when maxsplit is passed for the short strings?

Answer 1

The difference actually has nothing to do with what's going on inside string_split . In fact, the time spent inside that function is always slightly longer for default split than for maxsplit=1 , even if there are no splits to be done. And it's not the PyArg_ParseTuple difference (the best report I can get without instrumenting the interpreter says it takes 0ns either way, so whatever difference there is, it's not going to matter).

The difference is that it takes an extra bytecode to pass an extra parameter.

As Stefan Pochmann suggested, you can tell this by testing with an explicit maxsplit=-1 :

In [212]: %timeit ''.split('|')
1000000 loops, best of 3: 267 ns per loop
In [213]: %timeit ''.split('|', 1)
1000000 loops, best of 3: 303 ns per loop
In [214]: %timeit ''.split('|', -1)
1000000 loops, best of 3: 307 ns per loop

So even in this minimal example, the -1 is slightly slower than the 1 . But we're talking about 4ns of extra work. (I'm pretty sure this 4ns is because of preallocating a list of size 12 instead of size 2 , but I don't want to run through a profiler just to make sure.)

Meanwhile, an NOP bytecode takes 32ns to evaluate on my system (from another answer I'm still trying to find…). I can't imagine that LOAD_CONST is faster than NOP .

So, until you're doing enough work to overwhelm that 32ns+, not passing a maxsplit argument will save you time.

In case it isn't obvious, here's the disassembly for the two cases:

  1           0 LOAD_CONST               0 ('')
              3 LOAD_ATTR                0 (split)
              6 LOAD_CONST               1 ('|')
              9 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             12 RETURN_VALUE

  1           0 LOAD_CONST               0 ('')
              3 LOAD_ATTR                0 (split)
              6 LOAD_CONST               1 ('|')
              9 LOAD_CONST               3 (-1)
             12 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             15 RETURN_VALUE

---

For similar examples:

In [217]: %timeit int()
10000000 loops, best of 3: 94 ns per loop
In [218]: %timeit int(0)
10000000 loops, best of 3: 134 ns per loop
In [235]: %timeit [0].pop()
1000000 loops, best of 3: 229 ns per loop
In [236]: %timeit [0].pop(0)
1000000 loops, best of 3: 270 ns per loop

So the LOAD_CONST takes about 40ns in both these cases, just like passing -1 instead of no argument for split .

Python 3.4 is a little harder to test, because it caches some things that 2.7 doesn't, but it looks like it's about 33ns to pass an extra argument—or 533ns if it's a keyword argument. So, if you need to split tiny strings a billion times in Python 3, use s.split('|', 10) , not s.split('|', maxsplit=10) .

Answer 2

the proper initial test (original test had json_line and '|' mixed up)

python -m timeit -s "json_line='a|b|c'" "part_one=json_line.split('|')[0]"
1000000 loops, best of 3: 0.239 usec per loop
python -m timeit -s "json_line='a|b|c'" "part_one=json_line.split('|',1)[0]"
1000000 loops, best of 3: 0.267 usec per loop

The time difference is smaller.