RESTful API and bulk operations

Question

I have a middle tier which performs CRUD operations on a shared database. When I converted the product to .NET Core I thought I\'d also look at using REST for the API as CRUD i

Answer 1

If I understand correctly, you want optimistic concurrency for each record individually. That is, each record is to be deleted only if its state matches the client’s expectation. (If you only want to assert the entire collection’s state, then If-Match and 412 are sufficient.)

Roman Vottner’s answer does an excellent job of explaining the HTTP methods involved, but I’ll try to fill in some details.

Caveat emptor

When we talk about “how would REST handle” this or that, you understand that technically you can use HTTP as a transport for any operation in any way that suits you.

So when you’re asking about REST, I’m assuming you’re interested in a uniform interface — an approach that could theoretically be used by a range of various clients and servers.

But the key word there is “theoretically”. For example, once you define your own media type (your own JSON structure), a lot of the uniformity goes down the drain, because a client would have to be coded against your specific API anyway, at which point you can ask it to jump through any hoops you want.

But if you’re still interested in salvaging as much of the uniformity as possible, then read on.

All or nothing

If you want an all-or-nothing operation, which fails entirely if any of the individual preconditions fails, then, as Roman suggests, you can use PATCH with the JSON Patch format. For this, you need a conceptual representation of your collection as a single JSON object, to which the patch is to be applied.

For example, suppose you have resources like /my/collection/1, /my/collection/4, and so on. You could represent /my/collection/ as:

{
    "resources": {
        "1": {
            "href": "1",
            "etag": "\"BRkDVtYw\"",
            "name": "Foo Bar",
            "price": 1234.5,
            ...
        },
        "4": {
            "href": "4",
            "etag": "\"RCi8knuN\"",
            "name": "Baz Qux",
            "price": 2345.6,
            ...
        },
        ...
    }
}

Here, "1" and "4" are URLs relative to /my/collection/. You could use domain-specific IDs instead, but proper REST operates in terms of opaque URLs.

The standards don’t require you to actually serve this representation on GET /my/collection/, but if you do support such a request, then you should use that representation. Anyway, to this structure you can apply the following JSON patch:

PATCH /my/collection/ HTTP/1.1
Content-Type: application/json-patch+json

[
    {"op": "test", "path": "/resources/1/etag", "value": "\"BRkDVtYw\""},
    {"op": "remove", "path": "/resources/1"},
    {"op": "test", "path": "/resources/4/etag", "value": "\"RCi8knuN\""},
    {"op": "remove", "path": "/resources/4"},
    ...
]

Here, path is not a URL path, it’s a JSON pointer into the above representation.

If all patch operations succeed, then you respond with a successful status code like 204 (No Content) or 200 (OK).

If any of the ETag test operations fails, you respond with 409 (Conflict). You should not respond with 412 (Precondition Failed) in this case, because there is no precondition (like If-Match) on the request itself.

If anything else goes wrong, you respond with other appropriate status codes: see RFC 5789 § 2.2 and RFC 7231 § 6.6.

Mixed result

If you don’t want “all-or-nothing” semantics, then I’m not aware of any standardized solution. As Roman notes, you cannot use the PATCH method in this case, but you can use POST with a custom media type (RFC 6838 § 3.4). It could look like this:

POST /my/collection/ HTTP/1.1
Content-Type: application/x.my-patch+json
Accept: application/x.my-patch-results+json

{
    "delete": [
        {"href": "1", "if-match": "\"BRkDVtYw\""},
        {"href": "4", "if-match": "\"RCi8knuN\""},
        ...
    ]
}

You can respond to such a request with 200 (OK), regardless of whether any of the individual deletes succeeded. Another option would be 207 (Multi-Status), but I don’t see any benefits to it in this case, and it’s not widely used outside of WebDAV, so Postel’s law would suggest not going there.

HTTP/1.1 200 OK
Content-Type: application/x.my-patch-results+json

{
    "delete": [
        {"href": "1", "success": true},
        {"href": "4", "success": false, "error": {...}},
        ...
    ]
}

Of course, if the patch was invalid in the first place, you should instead respond with 415 (Unsupported Media Type) or 422 (Unprocessable Entity) as appropriate.

Another angle

The back-and-forth time for 1,000 individual calls is considerable

It is in HTTP/1.1. But if you can use HTTP/2 — which has much better support for concurrent requests, as well as much smaller network overhead per request — then 1000 individual requests might work out just fine for you.