@peter just a little information about the author, this is the format of the "from" field; 'steve at sobbayi.com (Steve Obbayi)' notice tht you can easily extract the Author from that field as it appears in the brackets Steve Obbayi -----Original Message----- From: skunkworks-bounces@lists.my.co.ke [mailto:skunkworks-bounces@lists.my.co.ke] On Behalf Of Steve Obbayi Sent: Tuesday, May 26, 2009 11:24 AM To: 'Peter Karunyu'; 'Skunkworks forum' Subject: [Skunkworks] FW: Skunkworks List and available resources @peter I forgot to add the query you will use to import the data, there is a slight difference from the mysql defaults. load data local infile '/path/to/datafile' into table `archives` fields escaped by '\\' terminated by ';' enclosed by '\'' lines terminated by '|^^|' (`from`, `date`, `timezone`, `subject`, `body`); @peter, Here is the SQL code you can use to create your table for this list then you can get the actual data files from Alex. This is just for storage purposes so there are no indices so you will have to create them as you like depending on how you plan to make the script. It also creates the table using InnoDB Engine so you might prefer to use MyISAM or any other you prefer. Now before I post the source code for the Parser, I want to make some few changes on the compilation optimization options coz with the wrong option even though it will still compile there is a bug in three inner loops (I havent bothered fixing) that could cause the loops to corrupt the heap when run at a very high optimized runtime state thereby crushing the progrm as it attempts to escape the string data. Maybe I should just sort out the bug once and for all to make it easier to use it fully optimized for speed and send it in a day or two. /*!40101 SET NAMES utf8 */; /*!40101 SET SQL_MODE=''*/; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; CREATE TABLE `archives` ( `from` varchar(255) NOT NULL, `date` datetime NOT NULL default '0000-00-00 00:00:00', `timezone` varchar(5) NOT NULL default '', `subject` varchar(255) NOT NULL, `body` longtext NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1; /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; Steve Obbayi, ________________________________ From: Peter Karunyu [mailto:pkarunyu@gmail.com] Sent: Monday, May 25, 2009 3:08 PM To: steve@sobbayi.com; Skunkworks forum Subject: Re: [Skunkworks] Skunkworks List and available resources Wow! that's fantastic. If you send me the schema of the DB, I can whip up some PHP to search the DB. Maybe you could share the source code as well? Thanks. On Mon, May 25, 2009 at 2:44 PM, Steve Obbayi <steve@sobbayi.com> wrote: @peter. I decided to take up and do what you had been trying to do. i have mailed alex gakuru the parsed monthly archives. Now some months were missing from the archives and all i had was up to March 2009. So there are about four archive files which still need to be parsed. To do this I developed a tool which has done this job very well. Am sending Alex the source code for the tool which can be compiled on any OS as long as it has a C++ compiler . Now all i developed was the tool and used it to parse all the files, i have also given the schema for the data file and the query to load it into a MYSQL database. So as far as code to make the data searchable i dont think i can squeez in time for that so maybe you or someone from this list can take it up and develop one for our sakes. Steve Obbayi, _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks Other services @ http://my.co.ke Other lists ------------- Skunkworks announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science - http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi - http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general