hey ppl,<br><br>I have a twitter sub-project I&#39;m working on and wondering if anyone is interesting in collaborating. Here&#39;s a brief description:<br><br>1) collect data from twitter&#39;s public stream<br>2) extract basic meta-data from the JSON dump from 1)<br>

2) categorize tweets according to language (not necessarily the same as location)<br>3) categorize tweets according to content (this is the hardest part, and most interesting to me)<br><br>I need help with 1) and 2). I have a ruby script (go it online) that is collecting data from a bunch of people&#39;s tweets. The data I&#39;m collecting right now isn&#39;t much, just over 1MB/minute. The script is based on the excellent Ruby twitter library (tweetstream) but I&#39;d like the script to be made much more robust.<br>

<br>The output of 1) is a json dump that I also need help with (this can be in any language that has good json parsers). The idea is to extract date/time info, the sender of the tweet, whether it&#39;s a retweet, what links are in the tweet, which other twitter username is in the tweet, what hashtags are referenced etc etc. Just very basic meta-data.<br>

<br>Presently I have some code that can do 3). But the code gets confused sometimes since each tweet is really small and sometimes includes multiple languages in it. For instance, I&#39;d like to be able to identify a Kenyan (regardless of where they&#39;re physically located) solely by their tweets. It&#39;s a lot harder than it appears, but I think it&#39;s doable.<br>

<br>Ultimately I&#39;d like to be able to automatically cluster tweets according to content, but that is a long-term project that I may not be able to achieve. But it&#39;s worth a try I think.<br><br>This will all be open-source, so there is no direct monetary compensation. I&#39;m doing it to learn concepts that I think will be extremely useful. I&#39;d love to answer any other questions you have about this.<br>

<br>Interested? Holla.<br><br>saidi<br><br>PS Don&#39;t sell yourself short and assume you can&#39;t do it. You&#39;re probably more skilled than you allow yourself to believe, or at the very least you can quickly learn where that is necessary.<br>

<br>PPS As you may have guessed, this isn&#39;t really about twitter. Twitter just happens to be a great source of huge amounts of freely available data. And it also happens to contain a wide spectrum of people/topics/places, making the info contained within extremely valuable.<br>