We are currently developing a project management software. And we are having trouble deciding on the correct approach to implement security. We have looked at both ACL and RBAC
Well, I use Yii framework with its nice RBAC layer. I'm not too familiar with ACLs, nor did I need to be, lately.
In Yii RBAC terms, your key to the solution is using 'business rules'. Bizrules are small snippets of code that are attached to a 'permission' or a 'role' (an 'auth item' in Yii's terms). This code is run dynamically when the need to determine access to a certain 'permission' (lets say, but it could also be attached to a 'role'), and it recieves the 'item in question' (task in your example) and determine actual access to the specific task or not. Here's a more detailed example:
That's in a nutshell. If you're interested in more, see this section of the official guide. There are also a bunch of other resources, should you need.
You need to look beyond ACL and RBAC and consider attribute-based access control (ABAC - see NIST's guide here). Gartner calls this space "externalized authorization management".
With ABAC, you can easily express any rules that take into account not just who the user is but also what the user wants to do, where, when, why, and how. Using attributes to define authorization, you can use XACML to implement policies. XACML is an OASIS standard (just like SAML).
With XACML, you get an API where you can ask questions e.g.: can Alice view this record? But in your case, it's not enough because you want to filter out records from the database. And, as you describe, you want the query to be the right from the start rather than going back and forth to the database until you have the right number of authorized records. This is where XACML becomes particularly interesting because it's technology-neutral. You can apply XACML to Java, C#, and other languages e.g. Python but also apply XACML to different layers (presentation, APIs, and... databases). XACML can be queried in a reverse query way to produce a SQL statement which you could then use to query your backend database for the relevant records:
HTH