How to perform DB bitwise queries in Django?

问题

How can I perform bitwise queries on the DB with Django?
I haven't found anything about it in the docs.
Should I retrieve a queryset and then filter programmically?

If you're interested, I use bitwise ops as an alternative to IN() statements in very large and complex queries, in order to improve performance.
I have a DB containing millions of items (records). Some fields use binary representation of an item property.
For example: the Color field can have multiple values, so it is structured like so:

0001 - Red
0010 - Green
0100 - Blue
1000 - White

(these are binary values)
So if an item has Red and Blue colors, the Color field will contain 0101.
When a user queries the DB, I use bitwise-OR to find matches (instead of IN() which is very slow).

回答1:

For postgres db you might use .extra() params with django orm.

For example:

SomeModel.objects.extra(where=['brand_label & 3 = 3'])

回答2:

You can perform database-level bitwise operations with F objects.

If field is non-negative, it means that condition field & mask > 0 can be re-written as (field > 0) AND (field >= (field & mask)). If you want to check if all bits of mask apply ((field & mask) == mask), you can build previous expression for each bit and then merge conditions via sql AND. Please see example how it can be done. (Custom QuerySet is just for convenience. If you use old django versions you can implement has_one_of and has_all as separate functions or classmethods, or better PathThroughManager). Note 1 * F is a workaround to force parenthesis over bitwise operation, otherwise django (as for version 1.5) will produce bad sql (colors >= colors & mask, comparison has higher priority, so it will mean TRUE & mask)

import operator
from django.db import models
from django.db.models import Q, F

_bit = lambda x: 2**(x-1)
RED = _bit(1)
GREEN = _bit(2)
BLUE = _bit(3)
WHITE = _bit(4)


class ItemColorsQuerySet(models.QuerySet):

    def has_one_of(self, colors):
        """
            Only those that has at least one of provided colors
        """
        return self.filter(
            colors__gt=0,
            # field value contains one of supplied color bits
            colors__lt=F('colors') + (1 * F('colors').bitand(reduce(operator.or_, colors, 0)))
        )

    def has_all(self, colors):
        """
            Has all provided colors (and probably others)
        """
        # filter conditions for all supplied colors: 
        # each one is "field value has bit that represents color"
        colors_q = map(lambda c: Q(colors__gte=1 * F('colors').bitand(c)), colors)
        # one complex Q object merged via sql AND:
        # colors>0 and all color-bit conditions
        filter_q = reduce(operator.and_, colors_q, Q(colors__gt=0))
        return self.filter(filter_q)


class Item(models.Model):

    name = models.CharField(max_length=100, unique=True)
    # can handle many colors using bitwise logic. Zero means no color is set.
    colors = models.PositiveIntegerField(default=0)

    objects = ItemColorsQuerySet.as_manager()

回答3:

Check django-bitfield , it works well w/ PostgreSQL (probably also well MySQL)

回答4:

same like @Ivan Klass,

You can use bitand or bitor from django orm F,

# filter Red and Blue(0101)
Color.objects.annotate(
  color_filter=F('color').bitand(0101)
).filter(color_filter__gte=0101)

 # filter Red or Blue(0101)
Color.objects.annotate(
  color_filter=F('color').bitand(0101)
).filter(color_filter__gt=0)

来源：https://stackoverflow.com/questions/9921221/how-to-perform-db-bitwise-queries-in-django

标签

django

bit-manipulation