分割ascii / unicode字符串

Question

I am trying to decode ID3v2 (MP3 header) protocol using python. The format of the data to be decoded is as follows.

s1 , s2 , ... sn-1 are unicode (utf-16/utf-8) strings, and last string 'sn' may be unicode or binary string.

data = s1+delimiters+s2+delimiters+...+sn

Where, delimiter for utf-16 is '\\x00'+'\\x00' and delimiter for utf-8 is '\\x00'

I get data along with unicode-type. Now I have to extract all the strings ( s1 , s2 , ... sn ) from data . For this I am using split() as follows,

#!/usr/bin/python

def extractStrings(encoding_type, data):
    if(encoding_type == "utf-8"): delimitors = '\x00'
    else: delimitors = '\x00'+'\x00'
    return data.split(delimitors)

def main():        
    # Set-1
    encoding_type = "utf-8"
    delimitors = '\x00'
    s1="Hello".encode(encoding_type)
    s2="world".encode(encoding_type)
    data = s1+delimitors+s2
    print extractStrings(encoding_type, data)

    # Set-2
    encoding_type = "utf-16"
    delimitors = '\x00'+'\x00'
    s1="Hello".encode(encoding_type)
    s2="world".encode(encoding_type)
    data = s1+delimitors+s2
    print extractStrings(encoding_type, data)

if __name__ == "__main__":
    main()

output:

['Hello', 'world']

['\xff\xfeH\x00e\x00l\x00l\x00o', '\x00\xff\xfew\x00o\x00r\x00l\x00d\x00']

It works for set-1 data but doesn't work for set-2. Since, 'data' in set-2

'\xff\xfeH\x00e\x00l\x00l\x00o\x00\x00\x00\xff\xfew\x00o\x00r\x00l\x00d\x00'
                             ^               ^

has an extra '\\x00' precedes delimiter, due to letter '0', its unable to do a proper job.

Can anyone help me to decode 'data' properly for both the cases?

Update2: Solution

The following code works for my requirement,

 def splitNullTerminatedEncStrings(self, data, encoding_type, no_of_splits): data_dec = data.decode(encoding_type, 'ignore') chunks = data_dec.split('\\x00', no_of_splits) enc_str_lst = [] for data_dec_seg in chunks[:-1]: enc_str_lst.append(data_dec_seg.encode(encoding_type)) data_dec_chunks = '\\x00'.join(chunks[:-1]) if(data_dec_chunks): data_dec_chunks += '\\x00' data_chunks = data_dec_chunks.encode(encoding_type) data_chunks_len = len(data_chunks) enc_str_lst.append(data[data_chunks_len:]) # last segment return enc_str_lst 

Answer 1

Where, delimiter for utf-16 is '\\x00'+'\\x00' and delimiter for utf-8 is '\\x00'

Not exactly. The delimiter for UTF-16 is \\0\\0 only at a code unit boundary. One \\0 at the end of one code unit followed by \\0 at the start of another code unit does not constitute a delimiter. The ID3 standard, talking about byte 'synchronisation' implies that this isn't the case, but it's wrong.

[Aside: unfortunately many tag-reading tools do take it literally that way, with the result that any sequence with a double-zero-byte in (eg U+0100,U+0061 Āa in UTF-16BE, or, as you discovered, any ASCII at the end of a string in UTF-16LE) will break the frame. As a result, UTF-16 text formats (UTF-16+BOM 0x01 and UTF-16BE 0x02) are completely unreliable and should be avoided by all tag writers. And text format 0x00 is unreliable for anything but pure-ASCII. UTF-8 is the winner!]

If you have a list-of-encoded-terminated-strings structure like those specified for the T frames (other than TXXX ), then the simple approach is to just decode them before splitting on the U+0000 terminator:

def extractStrings(encoding_type, data):
    chars = data.decode(encoding_type)
    # chars is now a Unicode string, delimiter is always character U+0000
    return chars.split(u'\0')

If data is a whole ID3 frame I'm afraid you can't process it with a single split() . Frames other than the T family contain a mixture of encoded-terminated-strings, ASCII-only-terminated-strings, binary objects (which have no termination) and integer byte/word values. APIC is one such, but for the general case you'd have to know the structure of every frame you want to parse in advance, and consume each field one-by-one, finding each terminator manually as you go.

To find the code-unit-aligned terminator in UTF-16-encoded data without misinterpreting Āa et al, you could use a regex, eg:

ix= re.match('((?!\0\0)..)*', data, re.DOTALL).end()
s, remainder= data[:ix], data[ix+2:]

This isn't a lot of fun really - ID3v2 is not a very clean format. Of the top of my head and untested, this sort of thing is how I might approach it:

p= FrameParser(data)
if frametype=='APIC':
    encoding= p.encoding()
    mimetype= p.string()
    pictype= p.number(1)
    desc= p.encodedstring()
    img= p.binary()

class FrameParser(object):
    def __init__(self, data):
        self._data= data
        self._ix= 0
        self._encoding= 0

    def encoding(self): # encoding byte - remember for later call to unicode()
        self._encoding= self.number(1)
        if not 0<=self._encoding<4:
            raise ValueError('Unknown ID3 text encoding %r' % self._encoding)
        return self._encoding

    def number(self, nbytes= 1):
        n= 0
        for i in nbytes:
            n*= 256
            n+= ord(self._data[self._ix])
            self._ix+= 1
        return n

    def binary(self): # the whole of the rest of the data, uninterpreted
        s= self._data[self._ix:]
        self._ix= len(self._data)
        return s

    def string(self): # non-encoded, maybe-terminated string
        return self._string(0)

    def encodedstring(self): # encoded, maybe-terminated string
        return self._string(self._encoding)

    def _string(self, encoding):
        if encoding in (1, 2): # UTF-16 - look for double zero byte on code unit boundary
            ix= re.match('((?!\0\0)..)*', self._data[self._ix:], re.DOTALL).end()
            s= self._data[self._ix:self._ix+ix]
            self._ix+= ix+2
        else: # single-byte encoding - look for first zero byte
            ix= self._data.find('\0', self._ix)
            s= self._data[self._ix:self._ix+ix] if ix!=-1 else self._data[self._ix:]
            self._ix= ix if ix!=-1 else len(self._data)
        return s.decode(['windows-1252', 'utf-16', 'utf-16be', 'utf-8][encoding])

Answer 2

Why don't you decode the strings first?

Python 2:

decoded = unicode(data, 'utf-8')
# or
decoded = unicode(data, 'utf-16')

Python 3:

decoded = str(data, 'utf-8')
# or
decoded = str(data, 'utf-16')

Then you work directly with encoding-agnostic data and the delimiters are always a single null.

Answer 3

The following code works for my requirement,

def splitNullTerminatedEncStrings(self, data, encoding_type, no_of_splits):
data_dec = data.decode(encoding_type, 'ignore')
chunks = data_dec.split('\x00', no_of_splits) 
enc_str_lst = []
for data_dec_seg in chunks[:-1]: 
    enc_str_lst.append(data_dec_seg.encode(encoding_type)) 
data_dec_chunks = '\x00'.join(chunks[:-1])   
if(data_dec_chunks): data_dec_chunks += '\x00'
data_chunks = data_dec_chunks.encode(encoding_type) 
data_chunks_len = len(data_chunks)
enc_str_lst.append(data[data_chunks_len:]) # last segment
return enc_str_lst