Hi people, I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this? Regards, Allan
The organisation I work for has done something similar. In our case, we roughly followed these principles: 1. The client app has is own database, at the beginning a copy of the main server database 2. The server app also has its own database 3. The client app can exports its data into a json file, which can be imported on the server app 4. The server app can export its data into a json file which can be imported on the client app 5. The end users are the ones who initiate the sync process, so that control rests with them, but its easy to automate this sync. 6. The json import logic on the client and server is then able to deal with duplicate data Be on the lookout for autogenerated primary keys, those can be a nasty little demon to deal with when syncing data. Infact, I would go out on a limb and suggest you dont use autogenerated primary keys but instead use something else. Also, MySQL supports Master-Master replication, where data flows both ways. But of course it assumes that the two computers can communicate directly, i.e. public IP addresses on both (unless networking gurus have another way). On Fri, May 20, 2016 at 11:34 AM, Allan O. via skunkworks < skunkworks@lists.my.co.ke> wrote:
Hi people,
I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this?
Regards,
Allan
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
This may not be directly relevant to the question, but Microsoft has an *Amazing* offline data synchronization framework: http://www.codeproject.com/Articles/18027/Take-Data-Offline-Using-Microsoft-... I am yet to see a solution from other vendors that gets even half-way close. You could probably borrow ideas from there. Martin. On Fri, May 20, 2016 at 2:57 AM, Peter Karunyu via skunkworks < skunkworks@lists.my.co.ke> wrote:
The organisation I work for has done something similar.
In our case, we roughly followed these principles:
1. The client app has is own database, at the beginning a copy of the main server database 2. The server app also has its own database 3. The client app can exports its data into a json file, which can be imported on the server app 4. The server app can export its data into a json file which can be imported on the client app 5. The end users are the ones who initiate the sync process, so that control rests with them, but its easy to automate this sync. 6. The json import logic on the client and server is then able to deal with duplicate data
Be on the lookout for autogenerated primary keys, those can be a nasty little demon to deal with when syncing data. Infact, I would go out on a limb and suggest you dont use autogenerated primary keys but instead use something else.
Also, MySQL supports Master-Master replication, where data flows both ways. But of course it assumes that the two computers can communicate directly, i.e. public IP addresses on both (unless networking gurus have another way).
On Fri, May 20, 2016 at 11:34 AM, Allan O. via skunkworks < skunkworks@lists.my.co.ke> wrote:
Hi people,
I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this?
Regards,
Allan
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
Hi Allan, I remember when HTML 5 was being introduced there was a buzz about offline mode. I have personally not tried it but I think it would be fun to explore it (http://www.html5rocks.com/en/features/offline). You can also do the same with persistent storage with javascript, I saw this on stackoverflow, check the first answer ( http://stackoverflow.com/questions/2786303/offline-mode-app-in-a-html5-brows... ). If you decide to go with either of the two, kindly share your experience. It would be nice to know the results regards, Derrick On Fri, May 20, 2016 at 8:57 PM, Martin Akolo Chiteri via skunkworks < skunkworks@lists.my.co.ke> wrote:
This may not be directly relevant to the question, but Microsoft has an *Amazing* offline data synchronization framework: http://www.codeproject.com/Articles/18027/Take-Data-Offline-Using-Microsoft-...
I am yet to see a solution from other vendors that gets even half-way close. You could probably borrow ideas from there.
Martin.
On Fri, May 20, 2016 at 2:57 AM, Peter Karunyu via skunkworks < skunkworks@lists.my.co.ke> wrote:
The organisation I work for has done something similar.
In our case, we roughly followed these principles:
1. The client app has is own database, at the beginning a copy of the main server database 2. The server app also has its own database 3. The client app can exports its data into a json file, which can be imported on the server app 4. The server app can export its data into a json file which can be imported on the client app 5. The end users are the ones who initiate the sync process, so that control rests with them, but its easy to automate this sync. 6. The json import logic on the client and server is then able to deal with duplicate data
Be on the lookout for autogenerated primary keys, those can be a nasty little demon to deal with when syncing data. Infact, I would go out on a limb and suggest you dont use autogenerated primary keys but instead use something else.
Also, MySQL supports Master-Master replication, where data flows both ways. But of course it assumes that the two computers can communicate directly, i.e. public IP addresses on both (unless networking gurus have another way).
On Fri, May 20, 2016 at 11:34 AM, Allan O. via skunkworks < skunkworks@lists.my.co.ke> wrote:
Hi people,
I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this?
Regards,
Allan
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
Thanks guys. I am still structuring the solution. Will share the experience after successful execution. Regards, Allan On Wed, May 25, 2016 at 9:43 PM, Derrick Wesonga via skunkworks < skunkworks@lists.my.co.ke> wrote:
Hi Allan, I remember when HTML 5 was being introduced there was a buzz about offline mode. I have personally not tried it but I think it would be fun to explore it (http://www.html5rocks.com/en/features/offline). You can also do the same with persistent storage with javascript, I saw this on stackoverflow, check the first answer ( http://stackoverflow.com/questions/2786303/offline-mode-app-in-a-html5-brows... ). If you decide to go with either of the two, kindly share your experience. It would be nice to know the results regards, Derrick
On Fri, May 20, 2016 at 8:57 PM, Martin Akolo Chiteri via skunkworks < skunkworks@lists.my.co.ke> wrote:
This may not be directly relevant to the question, but Microsoft has an *Amazing* offline data synchronization framework: http://www.codeproject.com/Articles/18027/Take-Data-Offline-Using-Microsoft-...
I am yet to see a solution from other vendors that gets even half-way close. You could probably borrow ideas from there.
Martin.
On Fri, May 20, 2016 at 2:57 AM, Peter Karunyu via skunkworks < skunkworks@lists.my.co.ke> wrote:
The organisation I work for has done something similar.
In our case, we roughly followed these principles:
1. The client app has is own database, at the beginning a copy of the main server database 2. The server app also has its own database 3. The client app can exports its data into a json file, which can be imported on the server app 4. The server app can export its data into a json file which can be imported on the client app 5. The end users are the ones who initiate the sync process, so that control rests with them, but its easy to automate this sync. 6. The json import logic on the client and server is then able to deal with duplicate data
Be on the lookout for autogenerated primary keys, those can be a nasty little demon to deal with when syncing data. Infact, I would go out on a limb and suggest you dont use autogenerated primary keys but instead use something else.
Also, MySQL supports Master-Master replication, where data flows both ways. But of course it assumes that the two computers can communicate directly, i.e. public IP addresses on both (unless networking gurus have another way).
On Fri, May 20, 2016 at 11:34 AM, Allan O. via skunkworks < skunkworks@lists.my.co.ke> wrote:
Hi people,
I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this?
Regards,
Allan
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
You can use an offline first dbms like NoSQL The front end can be done in JavaScript. Have a look at https://www.youtube.com/watch?v=1sLjWlWvCsc <https://www.youtube.com/watch?v=1sLjWlWvCsc&feature=youtu.be> &feature=youtu.be how such systems were deployed in west Africa during the Ebola response Regards, Alex From: Allan O. via skunkworks [mailto:skunkworks@lists.my.co.ke] Sent: Thursday, May 26, 2016 5:04 PM To: Skunkworks Mailing List <skunkworks@lists.my.co.ke> Subject: Re: [Skunkworks] Offline web app Thanks guys. I am still structuring the solution. Will share the experience after successful execution. Regards, Allan On Wed, May 25, 2016 at 9:43 PM, Derrick Wesonga via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> > wrote: Hi Allan, I remember when HTML 5 was being introduced there was a buzz about offline mode. I have personally not tried it but I think it would be fun to explore it (http://www.html5rocks.com/en/features/offline). You can also do the same with persistent storage with javascript, I saw this on stackoverflow, check the first answer (http://stackoverflow.com/questions/2786303/offline-mode-app-in-a-html5-brows...). If you decide to go with either of the two, kindly share your experience. It would be nice to know the results regards, Derrick On Fri, May 20, 2016 at 8:57 PM, Martin Akolo Chiteri via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> > wrote: This may not be directly relevant to the question, but Microsoft has an *Amazing* offline data synchronization framework: http://www.codeproject.com/Articles/18027/Take-Data-Offline-Using-Microsoft-... I am yet to see a solution from other vendors that gets even half-way close. You could probably borrow ideas from there. Martin. On Fri, May 20, 2016 at 2:57 AM, Peter Karunyu via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> > wrote: The organisation I work for has done something similar. In our case, we roughly followed these principles: 1. The client app has is own database, at the beginning a copy of the main server database 2. The server app also has its own database 3. The client app can exports its data into a json file, which can be imported on the server app 4. The server app can export its data into a json file which can be imported on the client app 5. The end users are the ones who initiate the sync process, so that control rests with them, but its easy to automate this sync. 6. The json import logic on the client and server is then able to deal with duplicate data Be on the lookout for autogenerated primary keys, those can be a nasty little demon to deal with when syncing data. Infact, I would go out on a limb and suggest you dont use autogenerated primary keys but instead use something else. Also, MySQL supports Master-Master replication, where data flows both ways. But of course it assumes that the two computers can communicate directly, i.e. public IP addresses on both (unless networking gurus have another way). On Fri, May 20, 2016 at 11:34 AM, Allan O. via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> > wrote: Hi people, I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this? Regards, Allan _______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24 <http://my.co.ke/phpbb/viewtopic.php?f=24&t=94> &t=94 ------------ Other services @ http://my.co.ke _______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24 <http://my.co.ke/phpbb/viewtopic.php?f=24&t=94> &t=94 ------------ Other services @ http://my.co.ke _______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24 <http://my.co.ke/phpbb/viewtopic.php?f=24&t=94> &t=94 ------------ Other services @ http://my.co.ke _______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24 <http://my.co.ke/phpbb/viewtopic.php?f=24&t=94> &t=94 ------------ Other services @ http://my.co.ke
We use HTML5 client-cache in the implementation of http://MiniERP.co.ke to create an system that continues working when internet i down, and then automatically synchronizes when the net is back up. Briefly explained our experiences are: - Offline-enabled web app is not a "feature" you add on after you build your online app, it has to be thought into the architecture from the start. - The minimum common denominator of whats possible across browsers buts a strong limit on how "big" you can go (manage your cached bytes efficiently). - You quickly end up with building the same application twice, once in JS and once on the server. - GUID's can be very useful to handle situations where the internet goes down right after the transaction is sent to the server, but you still await the response to come back (remember to test this oddity). - For robustness, ensure that there are no issues with the same transactions being repeated multiple times. - Getting all the HTML, JS, and other required files into the app-cache is tricky to manage. - Testing is challenging, at a minimum you need to test everything twice (online and offline). - Rule of Thumb/estimate is that building an offline enabled web app takes 3x development time compared to the same app just online. .. Mike On 05/25/2016 08:43 PM, Derrick Wesonga via skunkworks wrote:
Hi Allan, I remember when HTML 5 was being introduced there was a buzz about offline mode. I have personally not tried it but I think it would be fun to explore it (http://www.html5rocks.com/en/features/offline). You can also do the same with persistent storage with javascript, I saw this on stackoverflow, check the first answer (http://stackoverflow.com/questions/2786303/offline-mode-app-in-a-html5-brows...).
If you decide to go with either of the two, kindly share your experience. It would be nice to know the results regards, Derrick
On Fri, May 20, 2016 at 8:57 PM, Martin Akolo Chiteri via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke>> wrote:
This may not be directly relevant to the question, but Microsoft has an *Amazing* offline data synchronization framework: http://www.codeproject.com/Articles/18027/Take-Data-Offline-Using-Microsoft-...
I am yet to see a solution from other vendors that gets even half-way close. You could probably borrow ideas from there.
Martin.
On Fri, May 20, 2016 at 2:57 AM, Peter Karunyu via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke>> wrote:
The organisation I work for has done something similar.
In our case, we roughly followed these principles:
1. The client app has is own database, at the beginning a copy of the main server database 2. The server app also has its own database 3. The client app can exports its data into a json file, which can be imported on the server app 4. The server app can export its data into a json file which can be imported on the client app 5. The end users are the ones who initiate the sync process, so that control rests with them, but its easy to automate this sync. 6. The json import logic on the client and server is then able to deal with duplicate data
Be on the lookout for autogenerated primary keys, those can be a nasty little demon to deal with when syncing data. Infact, I would go out on a limb and suggest you dont use autogenerated primary keys but instead use something else.
Also, MySQL supports Master-Master replication, where data flows both ways. But of course it assumes that the two computers can communicate directly, i.e. public IP addresses on both (unless networking gurus have another way).
On Fri, May 20, 2016 at 11:34 AM, Allan O. via skunkworks <skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke>> wrote:
Hi people,
I have a requirement to include offline functionality for an already existing web application. The web app is built on the LAMP stack. I am thinking of having two similar MySQL databases. One local on the clients' machines and another in the cloud, then have a way of syncing records. MySQL offers replication but that assumes a master - slave configuration which won't work for this scenario (because data flows two way). Anyone done this before or know the best way to implement this?
Regards,
Allan
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke <mailto:skunkworks@lists.my.co.ke> ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
_______________________________________________ skunkworks mailing list skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------
Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
participants (6)
-
Alex Watila -
Allan O. -
Derrick Wesonga -
Martin Akolo Chiteri -
Michael Pedersen -
Peter Karunyu