I'm very happy that skunks are actually thinking of smart solutions to this "problem". But what really is the problem? "Database Error: Unable to connect to the database:Could not connect to MySQL" Looks like a single tweak on mysql's config file - my.ini - would fix this. Increase MySQL's max_connections variable. This might require more GBs of RAM, but that's all. However, the situation may stem from each query taking too long. This could be a result of either too many joins, or each query returning too much data. Or just a very slow server. With less than 5KB of data / student, the entire dataset snugly fits inside 2.5GB of RAM. MySQL is clever enough to cache this for you if you set it up correctly. Otherwise, the arrays in servlets is lovely. If I was the sysadmin, for this 1-2 day phenomenon, I'd replace KNEC's $5,000, 1GB, 1.66Ghz dual core servers with my $1,000 4GB 2.4 GHz quad core laptop. Heck, it's a few hours phenomenon, so I borrow my friends laptops, load up the same data in their MySQL servers and set up load balancing. Bernard --- On Thu, 3/1/12, Moses Muya <mouzmuyer@gmail.com> wrote: From: Moses Muya <mouzmuyer@gmail.com> Subject: Re: [Skunkworks] KNEC WEBSITE To: "Shadrack Mwaniki" <shadrack_mwaniki@yahoo.com>, "Skunkworks Mailing List" <skunkworks@lists.my.co.ke> Date: Thursday, March 1, 2012, 5:27 AM @Solomon You have stated the solution very clearly.As for the webserver,I would use both Apache and Nginx.Apache will be fronted with Nginx because it has a VERY SMALL memory footprint and is quite scalable compared to Apache.By default it handles around 1024 requests per second and with a reasonable amount of RAM,we could push that to about 12,000 Requests/Second(720,000 Requests/Minute).Theoretically,using this calculation,it would take about 2 minutes to service the requests. Apache is good with processing dynamic files but it also happens to be a resource hog.Nginx will run on port 80 and will forward any dynamic requests to Apache which will be on port 8080. The underlying O.S determines a lot.Since we want to keep the costs as low as possible without sacrificing performance,we will be using either FreeBSD or Ubuntu.Besides,Windows is no good with Nginx as it has a different way of handling event polling.The installation of either O.S will be a bare installation comprising only of a web-server,database server,application server(php or java) and SSH(will explain later why we need this) to save on the resources we have. With Ubuntu,we will be able to provision new server instances using 'Juju' based on the performance(FreeBSD geeks will let us know whether there is an equivalent of this on FreeBSD). The database set up is well explained,I won't go into that. Two things that are rather unconventional will have to be done here : 1.0 Have parents/guardians/students send their email addresses to the ministry.These will be sorted and saved ready for d-day.Immediately the results are released,everyone who gave their email address will have them safely in their inbox,hence no need to go to the website!We never know,this could cut about 100,000 requests from the web server if everything goes well. 2.0 The second option is rather unique and calls for true patriotism.There are many people on this list who can donate their bandwidth towards helping this cause.What if we all set up a sub-domain on our respective websites and mirrored the results there?It's simple,anyone who has the resources to do it will create something like knec.example.com or knec.example.co.ke.You will give the necessary access details to this sub domain and on d-day again,after the announcements have been made,the results will be Rsynced to that sub-directory and will appear on the sub-domain!The ministry will then display the mirrored sites on their website for anyone having trouble loading the knec website.This will only be done for a maximum of 3 days after the announcements then everyone can remove the sub domain and wait until the following year! Viola!Ladies and gentlemen,we might have the solution at hand...By Kenya For Kenyans! Corrections and improvements will be highly appreciated. On 1 March 2012 13:33, Shadrack Mwaniki <shadrack_mwaniki@yahoo.com> wrote: @Solomon,Very good analysis. The problem is many of the people offering solutions are simply programmers and not developers. This is why theory is very important. Anyone who has not studied the theory and analyiss of algorithms cannot understand the importance of achieving O(logn) in the worst case scenario for such a system Regards From: solomon kariri <solomonkariri@gmail.com> To: Skunkworks Mailing List <skunkworks@lists.my.co.ke> Sent: Thursday, March 1, 2012 10:24 AM Subject: Re: [Skunkworks] KNEC WEBSITE Ok in my opinion,All this data is read only, Its so little it can fit into RAMI believe the limit should be bandwidth, ok lets assume this implementation,First of all they get rid of that php file and replace it with a simple index.html, that way it will just be served, nothing processed to generate html, plus it will be cached by the browser. They will then add a javascript that simply does an ajax query, receives a JSON response and generates the relevant html to display the JSON. That will move quite a lot of processing to the client side.On the server, they can simply load all the records on an array and sort on index number. That index number can actually be treated as a long, so no complex comparison. The sorting will be done just once, when the server starts since the data doesn't change. This will take O(nlogn) time. that will be like 5 seconds on the maximum. For any requests, a binary search is done on the sorted data and response is offered immediately. Since the data doesn't change, they can have a pool of threads servicing the requests and performing the binary searches concurrently. All searches will take O(logn) time, that's like negligible for the amount of data involved. If they want to keep access logs as well, well, that's pretty simple, they will create a simple in memory queue and add an entry to the queue and leave the process of writing that to disk/database to a separate thread or a number of threads, that way, the slow disk access speeds don't affect response time. With that, the only limit left will be the bandwidth. Actually with a 5mbps up and down link, they will be sorted, all people are looking for is text, most of the time. So I just wonder, is this so hard to implement or I'm I missing something? On Thu, Mar 1, 2012 at 9:51 AM, James Kagwe <kagwejg@gmail.com> wrote: Surprising they don't want to fix a problem that occurs only once a year yet the system is only relevant once a year. Its better not to offer a service than to offer a substandard service. They must build the required capacity or just kill the service altogether, otherwise its just a waste of resources. They probably an learn from electoral commission tallying system. On 3/1/2012 8:52 AM, Peter Karunyu wrote: A member of this list who knows someone in KNEC said here that they know what the problem is, they know how to fix it, they just don't see the logic in fixing a problem which occurs once a year. So, in addition to lamenting here, why don't we think a lil bit outside the box; We propose a solution which not only works for this annual occurrence, but also works for other problems they have which we don't know. For example, how about coming up with a solution which they can use to disseminate ALL exam results, not just KCSE, online? That should save then quite a bit in paper and printing costs. But I think the real cause of this problem is lack of accountability; the CIRT team @ CCK focuses solely on security, the Ministry of Info. focuses on policies, KICTB focuses on implementing some of those policies and a few other things, but not including quality of software. The directorate of e-government provides oversight on these systems. So if my opinions here are correct, someone @ Dr. Kate Getao's office is sleeping on the job. On Thu, Mar 1, 2012 at 8:11 AM, Bernard Owuor <b_owuor@yahoo.com> wrote: True. Fact that you can see "Failed connection to mysql DB" means that there's more than enough infrastructure. (1) You get a response from the server - this means there is sufficient bandwidth, and the webserver that hosts the app has sufficient CPU cycles (2) they're using mysql Apart from potential limitations in the the number of connections in windows, you can easily do 500 - 1000 simultaneous connections. Only one connection is needed, though, so this should not be an issue Obviously, the architecture is poor and the app is not tested. The developer really skimped on their computer science classes, or didn't have any at all. --- On Wed, 2/29/12, Rad! <conradakunga@gmail.com> wrote: From: Rad! <conradakunga@gmail.com> Subject: Re: [Skunkworks] KNEC WEBSITE To: "Skunkworks Mailing List" <skunkworks@lists.my.co.ke> Date: Wednesday, February 29, 2012, 1:57 PM Why are we assuming the problem is the infrastructure? On Wednesday, February 29, 2012, Solomon Mbũrũ Kamau wrote: Can we do a harambee, like the one we did, the other day, for the purchase of a server(s) for KNEC and give it to them as a gift? On 29 February 2012 17:38, ndungu stephen <ndungustephen@gmail.com> wrote: But of course.. _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke -----Inline Attachment Follows----- _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke -- Regards, Peter Karunyu ------------------- _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke -- Solomon Kariri, Software Developer, Cell: +254736 729 450 Skype: solomonkariri _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke -- Kind Regards, Moses Muya. -----Inline Attachment Follows----- _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke ------------ List info, subscribe/unsubscribe http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke