如何使 str.splitlines 方法不按十六进制字符拆分行？

Question

I'm trying to parse output from GNU Strings utility with str.splitlines() Here is the raw output from GNU Strings:

279304 9k=pN\n 279340 9k=PN\n 279376 9k<LN\n 279412 9k=\x0cN\n 279448 9k<4N\n

When I parse the output with the following code:

process = subprocess.run(['strings', '-o', main_exe], check=True, \
                        stdout=subprocess.PIPE, universal_newlines=True)
output = process.stdout
print(output)
lines = output.splitlines()
for line in lines:
    print(line)

I get a result that I don't expect and it breaks my further parsing:

279304 9k=pN
279340 9k=PN
279376 9k<LN
279412 9k=
          N
279448 9k<4N
279592 9k<hN
279628 9k;TN
279664 9k<$N

Can I somehow tell the splitlines() method not trigger on \x0 characters?

The desired result should have lines which starts with an offset (that 6 digits at the start of each line):

279304 9k=pN
279340 9k=PN
279376 9k<LN
279412 9k=N
279448 9k<4N
279592 9k<hN
279628 9k;TN
279664 9k<$N

Answer 1

I think that you actually get the expected result. But assuming ASCII or any of its derevatives (Latin-x, UTF8, etc.) '\\x0c' is the control character FormFeed which happens to be represented here as a vertical one line jump.

Said differently I would bet a coin that the resulting file contains the expected bytes, but that your further processing chokes on the control character.

Answer 2

The documentation for str.splitlines() says it will split lines based on a number of line boundary types including \\x0c . If you only want to explicitly split by \\n then you could user str.split('\\n') instead. However note that if your line ends with a `\\n then you will end up with an empty group that you might want to drop the last index if its empty string.

data = '279304 9k=pN\n 279340 9k=PN\n 279376 9k<LN\n 279412 9k=\x0cN\n 279448 9k<4N\n'
lines = data.split('\n')
if lines[-1] == '':
    lines.pop()
print(lines)
for line in lines:
    print(line)

OUTPUT

['279304 9k=pN', ' 279340 9k=PN', ' 279376 9k<LN', ' 279412 9k=\x0cN', ' 279448 9k<4N']
279304 9k=pN
 279340 9k=PN
 279376 9k<LN
 279412 9k=N
 279448 9k<4N

Answer 3

process = subprocess.run(['strings', '-o', main_exe], check=True, \
                        stdout=subprocess.PIPE, universal_newlines=True)
lines = [line.strip() for line in process.stdout.split('\n') if len(line) > 0]

Remove the call to strip() if you do want to keep that leading whitespace on every line

Answer 4

Your problem arises from using the splitlines method of Unicode strings, which produces different results than the splitlines method of byte strings.

There is an issue for cpython for this problem, open since 2014: . str.splitlines splitting on non-\r\n characters - Issue #66428 - python/cpython .

Below I have added a portable splitlines function that uses the traditional ASCII line break characters for both Unicode and byte strings and works both under Python2 and Python3. A poor man's version for efficiency enthusiasts is also provided.

• In Python 2, type str is an 8-bit string and Unicode strings have type unicode .
• In Python 3, type str is a Unicode string and 8-bit strings have type bytes .

Although there is no actual difference in line splitting between Python 2 and Python 3 Unicode and 8-bit strings, when running vanilla code under Python 3, it is more likely to run into trouble with the extended universal newlines approach for Unicode strings.

The following table shows which Python data type employs which splitting method.

Split Method	Python 2	Python 3
ASCII	str.splitlines	bytes.splitlines
Unicode	unicode.splitlines	str.splitlines

str_is_unicode = len('a\fa'.splitlines()) > 1

def splitlines(string): # ||:fnc:||
    r"""Portable definitive ASCII splitlines function.

    In Python 2, type :class:`str` is an 8-bit string and Unicode strings
    have type :class:`unicode`.

    In Python 3, type :class:`str` is a Unicode string and 8-bit strings
    have type :class:`bytes`.

    Although there is no actual difference in line splitting between
    Python 2 and Python 3 Unicode and 8-bit strings, when running
    vanilla code under Python 3, it is more likely to run into trouble
    with the extended `universal newlines`_ approach for Unicode
    strings.

    The following table shows which Python data type employs which
    splitting method.

    +--------------+---------------------------+---------------------------+
    | Split Method | Python 2                  | Python 3                  |
    +==============+===========================+===========================+
    | ASCII        | `str.splitlines <ssl2_>`_ | `bytes.splitlines`_       |
    +--------------+---------------------------+---------------------------+
    | Unicode      | `unicode.splitlines`_     | `str.splitlines <ssl3_>`_ |
    +--------------+---------------------------+---------------------------+
    
    This function provides a portable and definitive method to apply
    ASCII `universal newlines`_ for line splitting. The reencoding is
    performed to take advantage of splitlines' `universal newlines`_
    aproach for Unix, DOS and Macintosh line endings.

    While the poor man's version of simply splitting on \\n might seem
    more performant, it falls short, when a mixture of Unix, DOS and
    Macintosh line endings are encountered. Just for reference, a
    general implementation is presented, which avoids some common
    pitfalls.

    >>> test_strings = (
    ...     "##\ftrail\n##\n\ndone\n\n\n",
    ...     "##\ftrail\n##\n\ndone\n\n\nxx",
    ...     "##\ftrail\n##\n\ndone\n\nx\n",
    ...     "##\ftrail\r##\r\rdone\r\r\r",
    ...     "##\ftrail\r\n##\r\n\r\ndone\r\n\r\n\r\n")

    The global variable :data:`str_is_unicode` determines portably,
    whether a :class:`str` object is a Unicode string.

    .. code-block:: sh

       str_is_unicode = len('a\fa'.splitlines()) > 1

    This allows to define some generic conversion functions:

    >>> if str_is_unicode:
    ...     make_native_str = lambda s, e=None: getattr(s, 'decode', lambda _e: s)(e or 'utf8')
    ...     make_uc_string = make_native_str
    ...     make_u8_string = lambda s, e=None: ((isinstance(s, str) and (s.encode(e or 'utf8'), 1)) or (s, 1))[0]
    ... else:
    ...     make_native_str = lambda s, e=None: ((isinstance(s, unicode) and (s.encode(e or 'utf8'), 1)) or (s, 1))[0]
    ...     make_u8_string =  make_native_str
    ...     make_uc_string = lambda s, e=None: ((not isinstance(s, unicode) and (s.decode('utf8'), 1)) or (s, 1))[0]

    for a protable doctest:

    >>> for test_string in test_strings:
    ...     print('--------------------')
    ...     print(repr(test_string))
    ...     print(repr([make_native_str(_l) for _l in splitlines(make_u8_string(test_string))]))
    ...     print(repr([make_native_str(_l) for _l in poor_mans_splitlines(make_u8_string(test_string))]))
    ...     print([make_native_str(_l) for _l in splitlines(make_uc_string(test_string))])
    ...     print([make_native_str(_l) for _l in poor_mans_splitlines(make_uc_string(test_string))])
    --------------------
    '##\x0ctrail\n##\n\ndone\n\n\n'
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    --------------------
    '##\x0ctrail\n##\n\ndone\n\n\nxx'
    ['##\x0ctrail', '##', '', 'done', '', '', 'xx']
    ['##\x0ctrail', '##', '', 'done', '', '', 'xx']
    ['##\x0ctrail', '##', '', 'done', '', '', 'xx']
    ['##\x0ctrail', '##', '', 'done', '', '', 'xx']
    --------------------
    '##\x0ctrail\n##\n\ndone\n\nx\n'
    ['##\x0ctrail', '##', '', 'done', '', 'x']
    ['##\x0ctrail', '##', '', 'done', '', 'x']
    ['##\x0ctrail', '##', '', 'done', '', 'x']
    ['##\x0ctrail', '##', '', 'done', '', 'x']
    --------------------
    '##\x0ctrail\r##\r\rdone\r\r\r'
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    --------------------
    '##\x0ctrail\r\n##\r\n\r\ndone\r\n\r\n\r\n'
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']
    ['##\x0ctrail', '##', '', 'done', '', '']

    For further details see

    - Python 2: `5. Built-in Types - Python 2.7.18 documentation
      <https://docs.python.org/2.7/library/stdtypes.html>`_
    - Python 3: `Built-in Types - Python 3.10.4 documentation
      <https://docs.python.org/3/library/stdtypes.html>`_

    .. _`universal newlines`: https://docs.python.org/3/glossary.html
    .. _`ssl2`: https://docs.python.org/2.7/library/stdtypes.html#str.splitlines
    .. _`unicode.splitlines`: https://docs.python.org/2.7/library/stdtypes.html#unicode.splitlines
    .. _`ssl3`: https://docs.python.org/3/library/stdtypes.html#str.splitlines      -
    .. _`bytes.splitlines`: https://docs.python.org/3/library/stdtypes.html#bytes.splitlines
    """
    if ((str_is_unicode and isinstance(string, str))
        or (not str_is_unicode and not isinstance(string, str))):
        # unicode string
        u8 = string.encode('utf8')
        lines = u8.splitlines()
        return [l.decode('utf8') for l in lines]
    # byte string
    return string.splitlines()

def poor_mans_splitlines(string):
    r"""
    """
    if str_is_unicode:
        native_uc_type = str
    else:
        native_uc_type = unicode
    if ((str_is_unicode and isinstance(string, str))
        or (not str_is_unicode and isinstance(string, native_uc_type))):
        # unicode string
        sep = '\r\n|\n'
        if not re.search(sep, string):
            sep = '\r'
        else:
            # |:info:|
            # if there is a single newline at the end, `$` matches that newline
            # if there are multiple newlines at the end, `$` matches before the last newline
            string += '\n'
        sep_end = '(' + sep + ')$'
        # prevent additional blank line at end
        string = re.sub(sep_end, '', string)
        return re.split(sep, string)
    # byte string
    return string.splitlines()