Architecture to publish offline data from android app to backend

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. Emmanuel Mtali is looking for an . 。

I am working on app that primarily used in offline condition. User generate data offline and later I schedule a periodic worker that sends the data (Publish) to backend server when device is online

My publishes looks like the following (This is for user registration)

{ "event":"dbase_mob", "data":"1,0,Emmanuel,Mtali,1930,Male,0719XX,", "device_id":"34e01af81782", "publish_at":1637311191, "app_version":"2.4.1" }

• event - describes the type of event database, transaction etc
• data - carry the actual payload the first field is pub_id, other carry additional info
• device_id - unique identifier given to a device
• publish_at - when this was publish was created
• app_version -:)

The publishes gets incremental value as identifier 'pub_id'

Problems

• I need to sync the incremental value (pub_id) between server and app. Why? The app can be removed from the device! Installing the new app require publishes to continue from the last point
• Any failure to process a publish on the backend service makes hard to debug issues on the client app. (No chance for mistake)
• Edge case can happen where publish can move from say 100.. 102 skipping the order creating a gap that the backends interprets as a missing publish!

What I did? Concerning the issue of missing publish I have implemented another worker that requests for missing publishes from the server and publish them. This is still chaotic:- Example, if a publish for creating user failed for any reason and the publish for transaction for that specific user arrive to the server it create a data dependency conflict

Questions How can I get rid of additional sync of pub_id How can I ensure no missing publishes

Curious

How some apps like Instagram handle publishing offline data efficiently

Answer 1

The data generate in offline can be in different table or set a flag to determine if it synced or not and update when it successfully push to server.

But still it a really tough topic, there are many things need to handle like will your app install in multiple devices? if so then the continuous incremental id might not work as if 2 local devices have same id and push to service how will you merge them?