My subscriber database lost connection to the publisher and expired. Can my data be saved?

Question

I have a publisher database A and I have two subscriber databases B and C that subscribe to A. My application resides locally at sites B and C and through replication, changes at B and/or C are replicated to each other.

The problem is since 31 January 2019 C stopped subscribing to A and the IT guys at site C didn't know about it (no alerts).

The bigger problem is that during this time, people using the application at B have been entering data which is replicated back to A. At the same time, people at site C have been adding data to database C which was not replicating back.

If I reinstate a subscription, it will take the data at A and overwrite C which is a major problem because I will lose the data added during this time at C. Since this is health data, it's all encrypted and stored in xml format and it's not as simple as just updating the data which was missing because some files are shared between sites B and C which means they would add data to the saved xml tree.

As an example, if someone sees a patient at site B and enters a note, it updates one file. But then if same patient goes to site C the next week, the nurse there will update the same file that was updated at B.

I have no clue how do I sync back the updates made at C first before reinstating a new subscription.

If anyone has any advice that would be greatly appreciated thanks!

Answer 1

We have been using merge replication for years and sometimes (after DB updates for example) it does break with weird errors which are impossible to fix no matter what we try. In all those cases the ONLY solution that worked was to reinitialize subscriptions or even remove and re-create publications with all subscriptions.

But since it is the merge replication case and new data is being added at all ends (i.e. publisher and all subscribers) when syncs are not happening we cannot simply re-init subscriptions because like you have pointed out not synchronized data generated at failed subscriber's side will be lost. Your case is even more difficult because you are using encryption and XML files to store data. Do you store encrypted XML as BLOBs/CLOBs in DB?

However, I will provide the solution which has worked for us many times. So, if one or more endpoints of merge replication fails the following steps must be taken:

1. Manually add non-synchronized data to all endpoints using tools like Red Gate SQL Data Compare (commercial with free 14-day trial) or free SQL Server Data Tools (SSDT) from Microsoft. The Red Gate's tool is very powerful. You can define custom mappings of tables/columns between databases for comparison. While SSDT from MS at this time can only compare tables having equal names and same schema name. Both tools can generate insert/update/delete scripts to get your data in sync. They even automatically disable and then restore foreign key constraints checks.

2. Change your publication and set Action if name is in use property to Keep existing object unchanged for all articles.

3. Reinitialize subscriptions.

After you perform step #2 and re-init subscription replication agent will still take time to process existing records (and depending on data set size it may take quite some time so think about adding Date filters to articles) but it will not change any data, no deletes or inserts will happen if both publisher and subscriber databases are fully synchronized after step #1. The agent will only mark those existing records as processed to skip them during future runs, so you will only see Update commands counts increased at Replication Monitor. Once subscriptions are reinitialized you should be fine and new data generated at any side will get synchronized.

Your case is difficult because you can't easily perform manual data sync from step #1. But this is a pre-requisite and you will need to think how to do that. You didn't provide details of how you actually store the encrypted data so I can't suggest the detailed plan but somehow you need to manually decrypt existing XML records, insert missing parts then encrypt again and make sure the encrypted columns are equal at all sides.

Hope this helps you to resolve your problem.

PS. I'm not affiliated anyhow neither to Red Gate nor to Microsoft.