Why did the servers crash? because the admin didn't sleep!

Interesting how sysadmins/coders get production systems, especially on LAMP and HA cluster setups online without a Performance/Scalability Test - case in point, you have a very good application (lots of $$ expected once its launched) and mounting pressure from the senior management to 'get things working!!' A couple of minutes, hours, days, weeks or months after launching the system(depending on the amount of load you're dealing with) and you realize: 1. Did I need a 3-way failover plan? DRBD only works with 2 nodes. Should I have slaves replicating off the replicated databases? 2. Mysql Replication or DRBD / DRBD+Heartbeat.or DRBD+Mysql Cluster - how often Im I making schema changes on my application(s)? 3. For my Apache setup maybe FastCGI was better to use than mod_php? 4. Ok.. its a network issue, do I have a technical solution to improve resilience and network load? 5. IM I LEARNING FROM A PRODUCTION SYSTEM? *Something I observed as I took backstage while constantly giggling and later left in my capacity as volunteer-until-the-system-runs-then-we-can-discuss-employment-terms in an upcoming IT firm. Until I get a serious employer: $cd /pub $more beer $find /share/.trash/ -type f -atime +30 -exec rm -fr {} \;

OMG, nice one. On 9/24/10, Patrick Kariuki <patrick.kariuki@gmail.com> wrote:
Interesting how sysadmins/coders get production systems, especially on LAMP and HA cluster setups online without a Performance/Scalability Test - case in point, you have a very good application (lots of $$ expected once its launched) and mounting pressure from the senior management to 'get things working!!'
A couple of minutes, hours, days, weeks or months after launching the system(depending on the amount of load you're dealing with) and you realize:
1. Did I need a 3-way failover plan? DRBD only works with 2 nodes. Should I have slaves replicating off the replicated databases? 2. Mysql Replication or DRBD / DRBD+Heartbeat.or DRBD+Mysql Cluster - how often Im I making schema changes on my application(s)? 3. For my Apache setup maybe FastCGI was better to use than mod_php? 4. Ok.. its a network issue, do I have a technical solution to improve resilience and network load? 5. IM I LEARNING FROM A PRODUCTION SYSTEM?
*Something I observed as I took backstage while constantly giggling and later left in my capacity as volunteer-until-the-system-runs-then-we-can-discuss-employment-terms in an upcoming IT firm.
Until I get a serious employer: $cd /pub $more beer $find /share/.trash/ -type f -atime +30 -exec rm -fr {} \; _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
-- -- Gichuki John Ndirangu, C.E.H , C.P.T.P, O.S.C.P I.T Security Analyst and Penetration Tester infosigmer@inbox.com {FORUM}http://lists.my.co.ke/pipermail/security/ http://nspkenya.blogspot.com/ http://chuksjonia.blogspot.com/

Feel u! You should have landed those clients who ask for a demo of an application thats custom to them then We'll pay once we make money from it. $ su password *********** # rm -R /application Then you thank them for the opportunity to develop a complex app for 30 days and the sodas and FREE internet they gave you and promise to return once they make the millions for your 40k. *_______________________________________________________________ its ok child, He's heard you * On Fri, Sep 24, 2010 at 1:02 PM, Gichuki John Chuksjonia < chuksjonia@gmail.com> wrote:
OMG, nice one.
On 9/24/10, Patrick Kariuki <patrick.kariuki@gmail.com> wrote:
Interesting how sysadmins/coders get production systems, especially on LAMP and HA cluster setups online without a Performance/Scalability Test - case in point, you have a very good application (lots of $$ expected once its launched) and mounting pressure from the senior management to 'get things working!!'
A couple of minutes, hours, days, weeks or months after launching the system(depending on the amount of load you're dealing with) and you realize:
1. Did I need a 3-way failover plan? DRBD only works with 2 nodes. Should I have slaves replicating off the replicated databases? 2. Mysql Replication or DRBD / DRBD+Heartbeat.or DRBD+Mysql Cluster - how often Im I making schema changes on my application(s)? 3. For my Apache setup maybe FastCGI was better to use than mod_php? 4. Ok.. its a network issue, do I have a technical solution to improve resilience and network load? 5. IM I LEARNING FROM A PRODUCTION SYSTEM?
*Something I observed as I took backstage while constantly giggling and later left in my capacity as volunteer-until-the-system-runs-then-we-can-discuss-employment-terms in an upcoming IT firm.
Until I get a serious employer: $cd /pub $more beer $find /share/.trash/ -type f -atime +30 -exec rm -fr {} \; _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f...
------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
-- -- Gichuki John Ndirangu, C.E.H , C.P.T.P, O.S.C.P I.T Security Analyst and Penetration Tester infosigmer@inbox.com
{FORUM}http://lists.my.co.ke/pipermail/security/ http://nspkenya.blogspot.com/ http://chuksjonia.blogspot.com/ _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke

How about when someone wants to start an IT firm or Project; they allocate funds towards that achievement? A typical case in Kenya. Are we running IT firms or IT farms? Cos in the later, cows eat grass for years until such a time they produce milk.

Steve, We're talking days here, keeping Holstein-Friesian grade 1's on the milking machines with no food until they develop mastitis and then culling them and getting a new bunch in. A PHP developer today, probably the best I've met so far got 'fired' for asking payment/salary after almost two months developing a system for a lead regulatory agency in agriculture (sub-contracted job), the guy's code is on a webserver waiting for another unknowing php programmer to finish off the job and then get kicked out for 'sneezing on the job' or get paid peanuts under the banner 'we started it, you merely touched it'. This talent exploitation cycle goes on ....

This talent exploitation cycle goes on ....
<sigh> Will it ever end </sigh> and on a more important note, how can it be stopped? On Thu, Sep 30, 2010 at 10:30 PM, Patrick Kariuki <patrick.kariuki@gmail.com> wrote:
Steve,
We're talking days here, keeping Holstein-Friesian grade 1's on the milking machines with no food until they develop mastitis and then culling them and getting a new bunch in.
A PHP developer today, probably the best I've met so far got 'fired' for asking payment/salary after almost two months developing a system for a lead regulatory agency in agriculture (sub-contracted job), the guy's code is on a webserver waiting for another unknowing php programmer to finish off the job and then get kicked out for 'sneezing on the job' or get paid peanuts under the banner 'we started it, you merely touched it'.
This talent exploitation cycle goes on .... _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke
participants (5)
-
Gichuki John Chuksjonia
-
joe mwirigi
-
Kiti Chigiri
-
Patrick Kariuki
-
Stephen Ndungu