Pivoting EAV data from the client to a Relational Model on the server

Question

I am building a software platform for mobile electronic data collection. It should support any type of data. For example, the government might use it for a population survey; a manufacturing company might use it to evaluate plant condition at their factories; a research organizations might use it for clinical trials, e.t.c

As such, the software is powered by a database, with standard relational design for the metadata and entity attribute value for the actual data. Client software then reads the metadata and renders the appropriate user interface, complete with rules, validations, skip logic and so on. I believe the choice of EAV is a good one owing to the diversity of data that might be collected, but ...

Once the data is submitted from the mobile clients to the customer's server, the EAV model is no longer useful because the customer expects just his set of (usually very few) tables, for visualization and processing.

I have considered two options for pivoting the data.

1) Pivot the data immediately it is submitted to the server (via a JSON web service) and save it straightaway into a relational model.

2) Save the data in a similar schema on the server but have a background process that pivots it periodically and saves it in a relational model.

The first alternative seems more efficient as pivoting one record at a time is obviously quicker and less CPU intensive. The disadvantage is that if the metadata is changed, this process needs to adapt immediately by changing the relational model for the data accordingly. Depending on the extent of the changes, this can take some time. Worse, if it fails for any reason, upload requests might start being declined. If using the second approach, such failure would not "break" anything as urgent.

Are there other potential pitfalls I might be missing or design considerations I should make? What are some good reasons to do it one way or the other? Are there other alternatives I should be exploring to solve this problem?

Answer 1

Just define a straightforward relational schema of tables for their data using DDL. EAV is just an encoding of a proper schema & its metadata. Which, of course, the DBMS can't understand so you lose practically all the benefits of a DBMS. The only possible reason to use EAV is when tables are not known at compile time and DDL isn't fast enough or able to hold enough tables.

The EAV requests are just textual rearrangements of the DDL requests. (EAV configuration is typically a table for multiple entity-attribute-value requests given a table and key column(s) of the entities having virtual tables.) Moreover one only has to write a single interface easily implemented to map EAV configuration-then-updates to whichever of the two implementations one chooses. (It is better to use a pure relational interface and hide the chosen implementation but the nature of interfaces to SQL DBMSes, namely SQL, makes that difficult. Ie it would be easy if one is using a relational API rather than SQL.)

The EAV configuration without such an interface is only simpler if you don't declare the appropriate constraints or transactions on the virtual per-entity tables. Also every EAV version update or query must reconstruct the virtual tables then embed those expressions in the DDL version's update or query. (Only in the case of simply inserting or deleting or retrieving a single triple is the EAV DML as simple.)

Only if you showed that creating & deleting new tables was infeasible and the corresponding horrible integrity-&-concurrency-challenged mega-joining table-and-metadata-encoded-in-table EAV information-equivalent design was feasible should you even think of using EAV.