Django small bytestrings on model, what type should be used for modelling?

Question

I have a model where I want to store a handful of small (2-4 byte) byte strings (ex: b'foo' or b'\\x02' ). I'm wondering about the best way to model them on my Django model. I thought I could use CharField , but that doesn't seem to work as I'd expect. BinaryField does seem to work, but I'm not sure if it's appropriate for fields that are short length (again, typically 2-4 bytes)

Given the model:

class Foobar(models.Model):
    charfield = models.CharField(max_length=10)
    binaryfield = models.BinaryField()

When I do:

>>> fb1 = Foobar()
>>> fb1.charfield = b'\0000'
>>> fb1.binaryfield = b'\0000'
>>> fb1.save()

And then read the record back:

>>> read = Foobar.objects.get(id=fb1.id)
>>> read.charfield == b'\0000'
False
>>> read.binaryfield == b'\0000'
True

I'd expect both equality checks to be True. Furthermore, the docs seem to indicate that filtering a queryset on a Binaryfield is not allowed (and is something I'll need to be able to do). Having said that, it seems to work for me:

>>> Foobar.objects.filter(binaryfield__in=[b'\0000', b'blarg'])
<QuerySet [<Foobar: Foobar object>]>

Am I missing something about CharField's? Is BinaryField the appropriate choice here? Or is there a better alternative?

In case it matters, I'm using Django 1.11 (most recent LTS version currently), and this is a project running on Python 3.6.

Answer 1

BinaryField is the right choice for this—as of Django 2.1. You're unfortunately right that before that the documentation contains a caveat about "it is not possible to filter a queryset on a BinaryField value". Given that you were able to do just that, you might want to investigate and see exactly what the limitations are here.

Passing a plain bytestring to a CharField is definitely wrong. Django will implicitly convert your byte string to Unicode before encoding it for the database, and that will create errors. For example, there are byte sequences that are not valid utf-8 representations: try Foobar.objects.create(charfield=b'\\xf8') .

Another option is to explicitly encode the bytefield yourself (into hex characters, say), probably by making a custom field. But then you have to do the same thing whenever you filter() . Ugly.

So try to make BinaryField work.