Keeping annotated rank when filtering Django

Question

I have a filtered Queryset where I have annotated the rank of each object according to a specific order.

When I then perform other filters like changing the order by