How to combine websockets and http to create a REST API that keeps data up to date? [closed]

Answer 1

I don't know Java, but I worked with both Ruby and C on these designs...

Funny enough, I think the easiest solution is to use JSON, where the REST API simply adds the method data (i.e. method: "POST") to the JSON and forwards the request to the same handler the Websocket uses.

The underlying API's response (the response from the API handling JSON requests) can be translated to any format you need, such as HTML rendering... though I would consider simply returning JSON for most use cases.

This helps encapsulate the code and keep it DRY while accessing the same API using both REST and Websockets.

As you might infer, this design makes testing easier, since the underlying API that handles the JSON can be tested locally without the need to emulate a server.

Good Luck!

P.S. (Pub/Sub)

As for the Pub/Sub, I find it best to have a "hook" for any update API calls (a callback) and a separate Pub/Sub module that handles these things.

I also find it more resource friendly to write the whole data to the Pub/Sub service (option B) instead of just a reference number (option C) or an "update available" message (options A and D).

In general, I also believe that sending the whole user list isn't effective for larger systems. Unless you have 10-15 users, the database call might be a bust. Consider the Amazon admin calling for a list of all users... Brrr....

Instead, I would consider dividing this to pages, say 10-50 users a page. These tables can be filled using multiple requests (Websocket / REST, doesn't matter) and easily updated using live Pub/Sub messages or reloaded if a connection was lost and reestablished.

EDIT (REST vs. Websockets)

As For REST vs. Websockets... I find the question of need is mostly a subset of the question "who's the client?"...

However, once the logic is separated from the transport layer, than supporting both is very easy and often it makes more sense to support both.

I should note that Websockets often have a slight edge when it comes to authentication (credentials are exchanged once per connection instead of once per request). I don't know if this is a concern.

For the same reason (as well as others), Websockets usually have an edge with regards to performance... how big an edge over REST depends on the REST transport layer (HTTP/1.1, HTTP/2, etc').

Usually these things are negligible when it comes time to offer a public API access point and I believe implementing both is probably the way to go for now.

Answer 2

To summarize your ideas:

A: Send a message to all clients when a user edits data on the server. All users then request an update of all data.
-This system may make a lot of unnecessary server calls on behalf of clients who are not using the data. I don't recommend producing all of that extra traffic as processing and sending those updates could become costly.

B: After a user pulls data from the server, they then subscribe to updates from the server which sends them information about what has changed.
-This saves a lot of server traffic, but if you ever get out of sync, you're going to be posting incorrect data to your users.

C: Users who subscribe to data updates are sent information about which data has been updated, then fetch it again themselves.
-This is the worst of A and B in that you'll have extra round trips between your users and servers just to notify them that they need to make a request for information which may be out of sync.

D: Users who subscribe to updates are notified when any changes are made and then request the last change made to the server.
-This presents all of the problems with C, but includes the possibility that, once out of sync, you may send data that will be nonsense to your users which might just crash the client side app for all we know.

I think that this option E would be best:
Every time data changes on the server, send the contents of all the data to the clients who have subscribed to it. This limits the traffic between your users and the server while also giving them the least chance of having out of sync data. They might get stale data if their connection drops, but at least you wouldn't be sending them something like Delete entry 4 when you aren't sure whether or not they got the message that entry 5 just moved into slot 4.

Some Considerations:

• How often does the data get updated?
• How many users need to be updated each time an update occurs?
• What are your transmission costs? If you have users on mobile devices with slow connections, that will affect how often and how much you can afford to send to them.
• How much data gets updated in a given update?
• What happens if a user sees stale data?
• What happens if a user gets data out of sync?

Your worst case scenario would be something like this: Lots of users, with slow connections who are frequently updating large amounts of data that should never be stale and, if it gets out of sync, becomes misleading.

Answer 3

With all great information all the great people added before me.

I found that eventually there is no right or wrong, its simply goes down to what suits your needs:

lets take CRUD in this scenario:

WS Only Approach:

Create/Read/Update/Deleted information goes all through the websocket.    
--> e.g If you have critical performance considerations ,that is not 
acceptable that the web client will do successive REST request to fetch 
information,or if you know that you want the whole data to be seen in 
the client no matter what was the event ,  so just send the CRUD events 
AND DATA inside the websocket.

WS TO SEND EVENT INFO + REST TO CONSUME THE DATA ITSELF

Create/Read/Update/Deleted , Event information is sent in the Websocket,
giving the web client information that is necessary to send the proper 
REST request to fetch exactly the thing the CRUD that happend in server.

e.g. WS sends UsersListChangedEvent {"ListChangedTrigger: "ItemModified" , "IdOfItem":"XXXX#3232" , "UserExtrainformation":" Enough info to let the client decide if it relevant for it to fetch the changed data"}

I found that using WS [Only for using Event Data] and REST [To consume the data ]is better because:

[1] Separation between reading and writing model, Imagine you want to add some runtime information when your data is retrieved when its read from REST , that is now achieved because you are not mixing Write & Read models like in 1.

[2] Lets say other platform , not necessarily web client will consume this data. so you just change the Event trigger from WS to the new way, and use REST to consume the data.

[3] Client do not need to write 2 ways to read the new/modified data. usually there is also code that reads the data when the page loads , and not
through the websocket , this code now can be used twice , once when page loads , and second when WS triggered the specific event.

[4] Maybe the client do not want to fetch the new User because its showing currently only a view of old Data[E.g. users] , and new data changes is not in its interest to fetch ?