Custom lookup is not being registered in Django

Question

I've created a custom lookup. I'm using it for a query, however, when I do so, the error Related Field got invalid lookup: lcb is thrown.

I'm assuming that this is because this custom lookup isn't being registered properly. As you'll see below, I've tried several things and I'm lost as to what the issue is.

Here's my code:

tenants/views.py

from main.lookups import *

def find_tenants(request, house_id):
    house = HouseListing.objects.get(pk=house_id)
    applications = HousingApplication.objects.filter(date_from__gte=house.available_from)
    applications = applications.filter(pets__lcb=house.allowed_pets.values_list('id', flat=True))

    context = {'house': house, 'applications': applications}
    return render(request, 'landlords/find-tenants.html', context)

main/lookups.py

from django.db.models import Lookup, ManyToManyField


# Custom lookups

@ManyToManyField.register_lookup
class ListContainedBy(Lookup):
    lookup_name = 'lcb'

    def as_sql(self, compiler, connection):
        lhs, lhs_params = self.process_lhs(compiler, connection)
        rhs, rhs_params = self.process_rhs(compiler, connection)
        params = lhs_params + rhs_params
        return '%s <@ %s' % (lhs, rhs), params

I find this very strange, as the docs suggest registering the lookup in the AppConfig, or in models.py. I've tried both of these things and neither worked.

Traceback

Environment:


Request Method: GET
Request URL: http://127.0.0.1:8000/landlords/find-tenants/5/

Django Version: 1.10.2
Python Version: 2.7.12
Installed Applications:
['main.apps.MainConfig',
 'tenants.apps.TenantsConfig',
 'landlords.apps.LandlordsConfig',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.contrib.postgres',
 'imagekit']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  39.             response = get_response(request)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)

File "/Users/mightyspaj/Development/Projects/housingfinder/housingfinder/landlords/views.py" in find_tenants
  132.     applications = applications.filter(pets__lcb=house.allowed_pets.values_list('id', flat=True))

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/db/models/query.py" in filter
  796.         return self._filter_or_exclude(False, *args, **kwargs)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/db/models/query.py" in _filter_or_exclude
  814.             clone.query.add_q(Q(*args, **kwargs))

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/db/models/sql/query.py" in add_q
  1227.         clause, _ = self._add_q(q_object, self.used_aliases)

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/db/models/sql/query.py" in _add_q
  1253.                     allow_joins=allow_joins, split_subq=split_subq,

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/db/models/sql/query.py" in build_filter
  1178.             lookup_class = field.get_lookup(lookups[0])

File "/Users/mightyspaj/Development/Projects/housingfinder/lib/python2.7/site-packages/django/db/models/fields/related.py" in get_lookup
  694.             raise TypeError('Related Field got invalid lookup: %s' % lookup_name)

Exception Type: TypeError at /landlords/find-tenants/5/
Exception Value: Related Field got invalid lookup: lcb

All the things I've tried so far

1. Registering the lookup in the MainConfig.ready() function
2. Registering the lookup in the find_tenants() view
3. Registering the lookup in main/models.py
4. Registering the lookup as a Field instead of a ManyToManyField
5. Registering the lookup using ManyToManyField.register_lookup() instead of a decorator
6. Changing %s <@ %s to %s = %s. I thought the issue might be that it thinks my SQL is invalid

Answer 1

Edit 2

It looks like some of this confusion might be sorted out in later versions of django... Edit 1 below is for 1.10, but the code is different in tip so perhaps this has now been fixed. You could try upgrading to 1.11 and see if that fixes it. But the options I listed below are still worth a try if that doesn't work.

Edit 1

I realized that I misread the exception in my initial answer. Actually, the exception being hit is the one in django.db.models.fields.related.ForeignObject.get_lookup, and the code there is much clearer that django does not support custom lookups on Relational fields:

class ForeignObject(RelatedField):

    ...

    def get_lookup(self, lookup_name):
        if lookup_name == 'in':
            return RelatedIn
        elif lookup_name == 'exact':
            return RelatedExact
        elif lookup_name == 'gt':
            return RelatedGreaterThan
        elif lookup_name == 'gte':
            return RelatedGreaterThanOrEqual
        elif lookup_name == 'lt':
            return RelatedLessThan
        elif lookup_name == 'lte':
            return RelatedLessThanOrEqual
        elif lookup_name == 'isnull':
            return RelatedIsNull
        else:
            raise TypeError('Related Field got invalid lookup: %s' % lookup_name)

    ...

There are a couple of things is one thing you could try to get it to work though:

1. Implement a custom lookup on one of the fields through the many to many relationship, so you'd have a query like

   applications = applications.filter(
       pets__id__lcb=house.allowed_pets.values_list('id', flat=True))
   

   I was able to get something along these lines to work using the IN operator.

2. Alternatively, you may be able to subclass the ForeignObject you are using and override get_lookup so that it doesn't throw an exception for your new operator... no doubt there would be a bit of a fiddle you'd have to do to get that to work though. Edit: I tried this, and it didn't work because it's more complicated than that, and for a join operation it's the implicit field from the right-hand table that is the Field used, so subclassing the Field for the left hand table was not enough. Option 1 definitely seems the right way to go about it.

Original answer

I believe the exception is telling you that django won't attempt to use custom lookups for relational fields. Given that you mention it I'm guessing that pets is a ManyToManyField, i.e. a relational field, and so I guess that your lookup has been registered, django is just refusing to use it.

The bit of code in django you are hitting (in django.db.models.sql.query.Query.build_filter()) is:

....
if field.is_relation:
    # No support for transforms for relational fields
    num_lookups = len(lookups)
    if num_lookups > 1:
        raise FieldError('Related Field got invalid lookup: {}'.format(lookups[0]))
....

I can't say I fully understand the rationale, but that certainly would explain why whatever mechanism you tried to use to register the lookup you got the same result.

It seems to be a not-very-well documented issue. The only thing online I could find out about it (with 10 minutes searching) was this, where someone else came to the same conclusion.