sizeof(string) not equal to string length

Question

I used to think that each character is one byte (at least that is the case in c/c++) so the size of string should be equal to len(string) bytes. However, a simple experiment tells me that it is not the case in python:

import string, sys, random
# abstracted code, removed unnecessary parts
def loadKeyLength(db,key,N):
    val = key[:5] + ''.join(random.choice(string.ascii_letters + string.digits) for _ in xrange(N-5))
    print sys.getsizeof(val), len(val),val

def loadKeysSize(s): 
    r=0
    key=str(s).zfill(5)+str(r).zfill(5)
    loadKeyLength(None,key,s)

for i in range(80,120,3):
    loadKeysSize(i)

output on Ubuntu 14.04:

117 80 000802qdxV2TY3qjGpe6F35hLczQNE2h7bWRWpHEMxqcyrb01sI2A6gcTLKLxdQSFjGMFtWPZJDOtKe4
120 83 00083oX6FouwlAyUkmZZCLWuIWnDMAKZDlNvO4ElHTK4x6vjka42APnwOcEMFHDLXbTZg9CUpd5ALqveowX
123 86 000860z1yhl3i1mKYFMhY4D2kWKA6Bvpfw91VeI7gXyP52PrVbLoP95ykgkz47k3KhCgmgrHq3CBCEdV14aiOa
126 89 00089xfcmZyf8RrftFbxvx9qvJUd8bvG5FKH2Ydz7aN5EsnaBpQkvrTLIsAKNRADeF1M74Ghvk1opzRs28IokPVhS
129 92 00092COlhIGMXrQ4Zl7e6GPlVz43BVWLbnvC3ymtoZ6Itus8KWuM1I31xGPU5Y4vggpcq2g4c6uSvnmUjsAgpYkNoX1u
132 95 00095IrjvnSVC8ECKf2cNUsBkzrSfuTNobIUmAD9BktiMfQSoCBLkwPOa2QmovhnUEpYyAsCKdM2haVqb53PggDviQHseex
135 98 00098DsLvbvWmqgyuWsnQd0DillNmd3LyTSJ98XjKUDhbqBSxhVRoXyv0IkOjWAbZIEb5lmrnISWS28WS4OpisoJYPCIfnB4bw
138 101 00101JNfUNutpjBFhFlhyNhFae2gulYTIfBpfoBbnLl881LPeZNGQkwYF49pbDEvnqYkPSleFUrZ1tEfO4AokI7ka3Gcn8KkTmWWg
141 104 00104lbmN2zeZeUMS6xGQfjtImCkwQwmewbXxsxj0NGETdBGwAfnhBmXOSew8LMdULQYCEA3Nz8ny6OlGfOUP3zjf5lZXlNC8Cn89Il4
144 107 00107UwQCaa4szAYj9if1oIPleauAvyWVkyDzbtZSt0SiKfJgNG7avZLe4TSTWXuEZBOUICfTAjIzVlShwXJ54Oz14rZlBrQL0w05FJsckY
147 110 00110lNZA1HsGmdFZke0Up1PwxPtpt2RFDM9EdOljQ6K3oao44Q6CNsBZHxo56n63Lny5l6k5ny7rhgWtkEGoJS7JbeNBg9ACXApfz4seWiZrX
150 113 00113KJ1sSGNZfZx2xl0MBXY0yf6ybPNjpmYBYiHi0ZsBb9GFuE9hIQgR0TssgbdE9sqq1m90YlS1ZWHSwwElaCkNOT05GbbIt3AfZAzzlpul5jEJ
153 116 001167jYYE6oyKM7qKQdzjpV1xUVUb85hpHpliNZRyiX7r6vJJ4n2FSe9tLUJ4W0ecUEALEemAZ0mUSkSROPl3AdQJ9AFdUAWvT5v4WTbNUZlFk2x0JX
156 119 00119ehpukL2CAOfCDbdtvuEvROVZJUvg044u8YS3d81SQ1FQqZDoVe55F8zCi7ikH1rEk2MWGUQLrmdJkMDKCXrtoeuZBDpo7pJOcmLRYZMLcLiC37iWXx

On Windows10:

101 80 00080Ra86ljAznn9AM17OtGUuFmxdYd7lU1hkInjZoPQJ4C2g3itkqn7wV0thhxPgpxrDimJwUElXzL2
104 83 00083nxTNohavc5sQfvlmnPnGOQNpzn2PKQJTYeDq4I9lMkuMKAxOhdOm1l3KAxmyCNOhlCKla9KMp8XYit
107 86 00086NBNeIqBFwWPbGzvtEihHvFnO3XyfPUEL0izlzF45P0NdfNTCyCDHvO6xa6BX4TyybChCEllhPOFXWpWd4
110 89 00089PmSiJYRGmI2AlXkbFUwcO1Ipr0bFvCmA2Se1A9JGMRTcg7617mXmG7fNCmDZWWFwI5DgTNHtqRDvTzrrmf08
113 92 00092DbsiuxTEZJT8DznuF3mtpdRP4LP4Nboj8tpCbgZkfeeP925U8N7v34qQpT69bw26Lfwp1jJXhkcb1o0wGUsgSIt
116 95 000958GexIZILU3le53WUGzTC6sRLK3vQVCsNI1yOuFt1HdW6QHZm05n5XGGMsluSamrKINAoBPxuQ5yrYSQHE7BlrWI6Jl
119 98 00098suYJVfpHKDkjHmnXwevRUOskhnCfF5Zp2jcN4avlg7ZN9g98G3vFeMpoXrulM7g5VfOQKI7UudzNfqkGDSaNfSuDvfyEE
122 101 00101doDJHZQEJre8aDWDGIPeKzN2aFXKZxYH9w6o9ZxgAXXozc75KMMwQ23YN0N27IMdAY5Oe9WLQSUgIf6AkfSNjWTFOODBvXeg
125 104 00104lhRSLYEXD1waMMkSVKct8jnb8M97rRQl582dlnzRr8hGM00jJLxrhHVvq1Kbu59dtcCSb4vm0KzbXKGiIdarDakNVLuCWYA9Mrh
128 107 00107y11AyDhwM8BZzT73VhkYu9U4ogvbw10ZPmnA824MzAznGbhLbHPDJjnl2NfquwH9XEOTx4vJjz74HC7I8GceZsCTlIQE4tQtWEtmig
131 110 00110GD8f97kpuRShkyrXYI40UvWlWOvskqRrDbUNjR2x6cZcg4NywVe1UAecrehoTJU5WUqZvvxseD16fYFvzaTKv7Jdwn1yQOazXSKheHORZ
134 113 001130coEBOEdkY1raJ65VK77UPU7eRraN2dz9mibcbu4khQwFQWf9WVBPUwTjlddveJKGLKS4gtNLWNeN4U720DW8XmHdpqhkXxqGBouZ2ARYfU6
137 116 001164x8ALzFvQKijeOIcDz6DCBnqzcMPQiR7rLaMBNuNFBYULSJ2xWIcGdyHHZw2lqW817fYo56Yg5hibAO7NzOyehOyxxUA865lQUjiP8LwmffCdnO
140 119 00119Sak3ByDRCFDMYpzpNEIKU5yNEWbWdL0popfhspb8cjE9sEBpMNxyGj5wjofhdois8DYQUTumJ3Xy7nzR04xGCG3mNQkVzKw1d97XP5RwN99Yac6I5F

So what does this mean? What can I do if I want to generate a random string of specific size in bytes? Why is there a difference in size on two different platforms (some python internals I am sure) ?

Note: My main purpose is to generate random strings whose size follows a specific distribution for a memory research problem but that is irrelevant here.

Answer 1

Python string objects contain more information than just the characters. They contain a reference count, a reference to the type definition, the length of the string, the cached hash and the interning state. See the PyStringObject struct , as well as the PyObject_VAR_HEAD struct referenced.

As a result, an empty string has a memory size too:

>>> import sys
>>> sys.getsizeof('')
37

This size is platform dependent, because pointers and C integers have different sizes on different platforms. 37 is the size of a Python 2 str object on Mac OS X.

For unicode objects the picture is even more distorted; Python 2 can use either 2 or 4 bytes per codepoint , depending on a compilation-time choice. The most recent Python 3 versions use a variable number of bytes for Unicode text, between 1 and 4 bytes per codepoint depending on the highest codepoint requirements in the text.

As such, it is normal for sys.getsizeof() to return a different, higher value. sys.getsizeof() is not a function to get a string length. Use len() for that.

If you are want to know how much memory other software uses for a string, you definitely can't use the sys.sizeof() value; other software will make different choices about how to store text, and will have different overheads. The len() value of the encoded text may be a starting point, but you'll have to check with the documentation or developers for that other piece of software to see what they can tell you how much memory is required for a given piece of text.