Discussion on Python performance, Google, and Unladen Swallow
Hey people, For those interested in the present and future of Python, here's an interesting discussion about Python's use at Google, it's performance problems, the Google-sponsored Unladen Swallow (an optimization-focused branch of CPython), and other possible solutions: < http://groups.google.com/group/unladen-swallow/browse_thread/thread/4edbc406...
.
Joseph.
Thanks
Nice discussion. Python seems to come out not looking so good :( I guess the problem always comes down to a choice between runtime-efficiency and coding-efficiency (easier to code almost always means not-the-most-efficient runtimes). On the issue of Python and multi-threading on > 1 CPU/machine: I think most major problems can be split into > 1 process that don't even need to run on the same machine (thereby avoiding Python's dreaded "global interpreter lock" problem). So one can simply add more machines when more processing power in needed, without touching the code at all. I wonder if the proposed iHub will have cluster facilities? Perhaps a couple of Hadoop nodes for people to experiment and learn with? A handful, modest desktops can be trivially re-jigged into a modest Hadoop cluster. Saidi On Sat, Jan 30, 2010 at 11:34 AM, Dennis Kioko <dmbuvi@gmail.com> wrote:
Thanks _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
Just goes to show the ubiquity of the 'no free lunch' principle. When balancing performance, cross platform, usability & parallelism improving on one will compromise one (or more) of the others! As for facilities at the iHub we are indeed going to have a couple of servers for guys to experiment off. On Sat, Jan 30, 2010 at 8:31 PM, saidimu apale <saidimu@gmail.com> wrote:
Nice discussion. Python seems to come out not looking so good :(
I guess the problem always comes down to a choice between runtime-efficiency and coding-efficiency (easier to code almost always means not-the-most-efficient runtimes).
On the issue of Python and multi-threading on > 1 CPU/machine: I think most major problems can be split into > 1 process that don't even need to run on the same machine (thereby avoiding Python's dreaded "global interpreter lock" problem). So one can simply add more machines when more processing power in needed, without touching the code at all.
I wonder if the proposed iHub will have cluster facilities? Perhaps a couple of Hadoop nodes for people to experiment and learn with? A handful, modest desktops can be trivially re-jigged into a modest Hadoop cluster.
Saidi
On Sat, Jan 30, 2010 at 11:34 AM, Dennis Kioko <dmbuvi@gmail.com> wrote:
Thanks _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
_______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
Python3, which is itself quite a major rewrite, still leaves the "global interpreter lock" (GIL) problem intact (I may be wrong on this one, I'm not quite certain). Part of the problem is that it's incredibly difficult to solve, and maybe it isn't such a problem if one can work around it. The GIL issue is that on multi-processor machines, multi-threaded Python programs cannot effectively use the other CPUs/cores because of the lock. The alternative to creating threads is then to create processes instead, but with more overhead costs. In practice, apparently this isn't much of a problem (or so I've read, I don't have first-hand experience with it). Unless one operates at Google/Yahoo/Microsoft data-center levels, then such performance issues are probably mostly academic. Very informative to think about them nonetheless. Saidi On Sat, Jan 30, 2010 at 1:05 PM, Dennis Kioko <dmbuvi@gmail.com> wrote:
From the discussion, the biggest problem facing python seems to be lack of developers to improve its shortfalls. The discussion seems to suggest that a code rewrite for python would help a lot _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
On Sat, Jan 30, 2010 at 9:34 PM, saidimu apale <saidimu@gmail.com> wrote:
Python3, which is itself quite a major rewrite, still leaves the "global interpreter lock" (GIL) problem intact (I may be wrong on this one, I'm not quite certain). Part of the problem is that it's incredibly difficult to solve, and maybe it isn't such a problem if one can work around it.
The GIL issue is that on multi-processor machines, multi-threaded Python programs cannot effectively use the other CPUs/cores because of the lock. The alternative to creating threads is then to create processes instead, but with more overhead costs. In practice, apparently this isn't much of a problem (or so I've read, I don't have first-hand experience with it).
Unless one operates at Google/Yahoo/Microsoft data-center levels, then such performance issues are probably mostly academic. Very informative to think about them nonetheless.
Saidi
It should also be noted that these performance issues aren't issues with the language itself, but with Python implementations. And I think many Python users will agree that it's simply a joy to work in. As for the global interpreter lock (GIL), yeah, it isn't a real issue, as you can use the new "multiprocessing" module, and other workarounds, to achieve multi-core/multi-processor concurrency. And I understand Ruby (YARV) also has a GIL. Let's wait and see the effect of the Unladen Swallow optimizations, once (hopefully) they're merged back into CPython. We should also watch out for PyPy: <http://morepypy.blogspot.com/>. Joseph.
And I think many Python users will agree that it's simply a joy to work in.
Absolutely. Python and Java are my primary languages, I use the rest only if I have to and there are some I will never use no matter what. We should also watch out for PyPy Check out Stackless Python <http://www.stackless.com/> as well, it has a very easy-to-use multi-threading model. Saidi On Sun, Jan 31, 2010 at 12:07 PM, Joseph Wayodi <jwayodi@gmail.com> wrote:
On Sat, Jan 30, 2010 at 9:34 PM, saidimu apale <saidimu@gmail.com> wrote:
Python3, which is itself quite a major rewrite, still leaves the "global interpreter lock" (GIL) problem intact (I may be wrong on this one, I'm not quite certain). Part of the problem is that it's incredibly difficult to solve, and maybe it isn't such a problem if one can work around it.
The GIL issue is that on multi-processor machines, multi-threaded Python programs cannot effectively use the other CPUs/cores because of the lock. The alternative to creating threads is then to create processes instead, but with more overhead costs. In practice, apparently this isn't much of a problem (or so I've read, I don't have first-hand experience with it).
Unless one operates at Google/Yahoo/Microsoft data-center levels, then such performance issues are probably mostly academic. Very informative to think about them nonetheless.
Saidi
It should also be noted that these performance issues aren't issues with the language itself, but with Python implementations. And I think many Python users will agree that it's simply a joy to work in.
As for the global interpreter lock (GIL), yeah, it isn't a real issue, as you can use the new "multiprocessing" module, and other workarounds, to achieve multi-core/multi-processor concurrency. And I understand Ruby (YARV) also has a GIL.
Let's wait and see the effect of the Unladen Swallow optimizations, once (hopefully) they're merged back into CPython. We should also watch out for PyPy: <http://morepypy.blogspot.com/>.
Joseph.
_______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
On Sun, Jan 31, 2010 at 9:00 PM, saidimu apale <saidimu@gmail.com> wrote:
...
Python and Java are my primary languages, I use the rest only if I have to and there are some I will never use no matter what.
:) I'm a Java refugee myself, just a few months old in Python. And I love most of what I've seen so far.
Check out Stackless Python <http://www.stackless.com/> as well, it has a very easy-to-use multi-threading model.
Stackless Python looks interesting. I haven't needed to do any real concurrency work yet, but I'll definitely check it out as soon as I get the opportunity. Joseph.
As for facilities at the iHub we are indeed going to have a couple of servers for guys to experiment off.
Very nice! I hope folks will fully utilize these resources and think outside, around and in-between the proverbial boxes. Saidi On Sat, Jan 30, 2010 at 12:35 PM, Rad! <conradakunga@gmail.com> wrote:
Just goes to show the ubiquity of the 'no free lunch' principle. When balancing performance, cross platform, usability & parallelism improving on one will compromise one (or more) of the others!
As for facilities at the iHub we are indeed going to have a couple of servers for guys to experiment off.
On Sat, Jan 30, 2010 at 8:31 PM, saidimu apale <saidimu@gmail.com> wrote:
Nice discussion. Python seems to come out not looking so good :(
I guess the problem always comes down to a choice between runtime-efficiency and coding-efficiency (easier to code almost always means not-the-most-efficient runtimes).
On the issue of Python and multi-threading on > 1 CPU/machine: I think most major problems can be split into > 1 process that don't even need to run on the same machine (thereby avoiding Python's dreaded "global interpreter lock" problem). So one can simply add more machines when more processing power in needed, without touching the code at all.
I wonder if the proposed iHub will have cluster facilities? Perhaps a couple of Hadoop nodes for people to experiment and learn with? A handful, modest desktops can be trivially re-jigged into a modest Hadoop cluster.
Saidi
On Sat, Jan 30, 2010 at 11:34 AM, Dennis Kioko <dmbuvi@gmail.com> wrote:
Thanks _______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
_______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
_______________________________________________ Skunkworks mailing list Skunkworks@lists.my.co.ke http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks ------------ Skunkworks Server donations spreadsheet
http://spreadsheets.google.com/ccc?key=0AopdHkqSqKL-dHlQVTMxU1VBdU1BSWJxdy1f... ------------ Skunkworks Rules http://my.co.ke/phpbb/viewtopic.php?f=24&t=94 ------------ Other services @ http://my.co.ke Other lists ------------- Announce: http://lists.my.co.ke/cgi-bin/mailman/listinfo/skunkworks-announce Science: http://lists.my.co.ke/cgi-bin/mailman/listinfo/science kazi: http://lists.my.co.ke/cgi-bin/mailman/admin/kazi/general
participants (4)
-
Dennis Kioko -
Joseph Wayodi -
Rad! -
saidimu apale