GeoDjango order_by('distance') not working as expected?

Using GeoDjango, I am trying to sort a list of users based on the distance from a point. It does generate a list, but its not ordered by distance, though I specify that in my query.

This is the query I am using.

origin = GEOSGeometry('SRID=4326;'+ 'POINT('+ str(buyer.buyer_current_location.x)+' '+str(buyer.buyer_current_location.y)+')')
close_sellers = Seller.objects.exclude(profile__user = user).filter(seller_current_location__distance_lte=(origin, D(mi=distance_mi)),profile__user__is_active = True).distance(origin, field_name='seller_current_location').order_by('distance')[:20]

and these are the models that I have

class Buyer(models.Model):
    # This field is required.

    profile = models.ForeignKey(UserProfile,unique=True)
    # Other fields here
    screen_name = models.CharField(max_length = 200,default='buyer')
    buyer_current_location = models.PointField(null = True, srid=4326)
    objects = models.GeoManager()

class Seller(models.Model):
    # This field is required.
    profile = models.ForeignKey(UserProfile,unique=True)
    # Other fields here

    seller_current_location = models.PointField(null = True, srid=4326)
    objects = models.GeoManager()

Does anyone have any clue regarding what might be going wrong?

The list seems sorted, but only messes up on for sellers in a certain part of town (its totally crazy!)