Data synchronization is hard, but it doesn’t have to be!

I’m your host and author of this site, Jason Whitehorn, and learned these lessons the hard way – through years of experience working on various data synchronization projects. Sadly, most of this knowledge is not well documented. To my knowledge, no books or dedicated resources exist that document this to any great extent, until now.

This site explains various patterns, tools, and techniques for reliably implementing data synchronization without pulling your hair out. If you’re new to this site I recommend starting with an introduction to data synchronization and going from there. Returning visitors will likely find the list of content updates a helpful resource.

Large portions of this website are available completely free, and I hope that you read and enjoy those pages. If you find them useful, I recommend registering as a paid member. By doing so you’ll not only support this website, but you’ll also get a lifetime of ad-free access to 100% of the content on this website, including many sections reserved exclusively for paid members.

Table of Contents

• Introduction
  • Technologies
  • Code Samples

Part I: Server to Client Updates

1. Whole Database Synchronization
   • Pros and Cons
   • Algorithm Overview
   • Implementation
   • Server Side Components
   • Mobile Updates
   • Concurrency
   • Summary
2. Delta-based Synchronization
   • Algorithm Overview
   • Bootstrapping the Database
   • Mobile Synchronization
   • Asynchronous Logic
   • Fetch Updates
   • Timestamps, System Clocks, and Missing Data
3. Handling Deletions
   • Deletes are Difficult
   • Solutions
   • Avoiding Deletions
   • Logical Deletes
   • Tracking Deletions
   • Propagating Deletions
   • Processing Deletions
   • Summary
4. Access Rights
   • Potential Complexities
   • Excluding Records
   • Access Chains
   • Granting & Revoking Access
   • Object Hierarchies
   • Summary

Part II: Client to Server Updates

5. Foundations of Up-Sync
   • A New Application
   • Unique Identifiers
   • Eventual Consistency, Accuracy, and Repeatability
6. Conflict Resolution
   • Time Series Events
   • The Rectifier Pattern
7. Audit Logging

Part III: Performance & Scaling

8. Identifying Changed Records
9. Rapid Updates

Part IV: Legacy Integrations

10. Unreliable Timestamps
11. Replaying Transaction Logs

– denotes content which is only available to paid members.