Prototyping Go’s Select for Stackless Python with Stackless.py

Dear Folks:

On April 26th, 2010, I did a dry run of the talk Prototyping Go’s Select for Stackless Python with Stackless.py. I thank everyone who attended my talk despite the hockey game being on at the same time. I also deeply appreciate the feedback! Hopefully this talk will be accepted for EuroPython 2010. The motivation for the talk was simple. Stackless Python and Google’s new language, Go, borrow their concurrency model from Bell Lab’s Limbo. However Stackless Python does not have the select statement. Select allows a coroutine to monitor multiple channels without resorting to polling. I find the idea of Stackless Python written Python to be a powerful idea. How hard would it be to implement select functionality in Stackless Python? How easy would it be to use PyPy’s stackless.py package to prototype the feature? Does stackless.py deviate so much from Stackless Python as not be a good estimator of required time and effort?

As Stackless Pythonistas pointed out, select like functionality could be implemented with additional tasklets. I understand this approach fits in with the tao of Stackless Python to customise through subclassing and using the API. Keep Stackless Python small. For the most part I agree. However in the case of select, how efficient would this be? It has also been argued that no one wants select. Well we all know the adage about hackers wanting to write software they will use. Based on my work with WS-BPEL and business process orchestration, I have implemented workarounds. So I see uses for select like abilities in the areas of complex event processing and workflow patterns. Perhaps these features could make Stackless Python attractive to a new audience?

We will soon be in a position to experiment and benchmark….

Incidentally I forgot to tell the audience that I tended to submit the talk to EuroPython 2010 and I wanted to iron out the bugs. My apologies. Based on feedback, I will restructure the talk. However I will post the original slides and code soon. Really. For better or worse, the video will be out there.

Prototyping is still very much a work in progress. I am still learning about many things about the Go language and Stackless Python internals. For instance, Go does expose more of the concurrency system than I thought, through its runtime and reflect packages. Although it doesn’t alter the big picture, this new information will be reflected in the updated talk and paper.

As of April 26th, I have implemented a few versions of a select like feature in stackless.py. For now select (called eventHandler) is a class rather than a language feature. These implementations have been very conservative. I will write more versions in the days and weeks to follow. Meanwhile, my friend Kevin BuluĊĦek has bravely taken the plunge and is implementing the changes in the Stackless Python implementation to support a select. From working with Kevin, I was reminded of a line uttered by Ben Affleck’s character Morgan from the film Good Will Hunting comes to mind: “My boy’s wicked smart.”

To date, Prototying is the most esoteric of the three talks I have given. It is the most fun to research and do. A lot is happening in the background. Some the action won’t get into a thirty or forty-five minute talk. For instance, I think the Agile technique of two man programming is really effective. I am also excited about Christian’s work with integrating Stackless Python with Pscyo. yet another reason to avoid programming Python extensions in C. And my C is really rusty. It was a blast to take the algorithms out of Rob Pike’s paper The Implementation of Newsqueak and implement them in Python. Although stackless.py is a low priority concern for the PyPy team (and not maintained), I think there is much potential in using stackless.py. And each time I look at the PyPy blog, the team seems to be making significant gains.

Most important for the current project, Stephan Diehl and Carl Bolz pointed out an important trick-of-the-trade: use stackless.py with the greenlets package running on top of Standard Python. Previously I used the PyPy interpreter that was so slow. And forget compiling to pypy-c. It is all about high clockspeed (a book I am currently reading, recommended by a former professor). Thanks guys! Christian Tismer also gave helpful advice and was correct in saying that the C code base would have to altered to support select. I originally thought we could get away with no changes….. My first prototype gave me the wrong impression.

During the course of Prototyping, I attended a McGill Management school talk by John Seeley Brown called “Power of Pull: How Small Moves, Smartly Made, Can Set Big Things in Motion.” The aforementioned is the name of Brown’s new book in collaboration with Jon Hagel. I had heard of Jon Hagel before because I read “Out of the Box” that dealt with web services in business. This topic was right up my alley. A central theme of Brown’s talk was about how to create high performance organisations. Drawing from cases ranging from elite surfers, to Amazon, to WoW guiles, Brown listed some simple principles:

  • Deep collaboration.
  • Fail fast
  • Using frame-by-frame analysis to improve
  • Look at adjacencies.
  • Look for spikes of capabilities.

A lot of the principles applied to my current adventure. For instance, prototyping in with stackless.py allowed me to fail fast. . Believe me, I made many mistakes. The revision control system has the bones of many versions nipped in the bud. It helped that Kevin acted as a springboard and a sanity check. Occasionally Kevin would say, remember the KISS principle and all I could do is sheepishly grin (funny I just read a New York Times article called “It’s Complicated: Making Sense out of Complexity” that talked about American’s love affair with the complicated). As for adjacencies, I closely looked at Rob Pike “Newsqueak” paper and reviewed the various Google videos on Newsqueak and Go. And I have dabbled with Erlang. Perhaps the principle that didn’t dawn on me earlier was “look for spikes of capabilities.” I decided to be audacious and subscribe to the Go Language Nuts newsgroup and post questions. I was pleasantly surprised when ‘Commander’ Rob Pike himself, Russ Cox and Ian Taylor, kindly indulged me and answered my questions. Looking at the Plan 9 code was really really helpful although there are parts like the lock acquisition that I still haven’t wrapped my head around it. At the end of the day, Stackless Python and Go are *different* languages built for different purposes despite having a common ancestor. Nevertheless. I feel Newsqueak, Plan 9, and Go will show where the dragons may be for a C Stackless Python implementation. Perhaps once I have enough of a framework in place, I can find optimisations: one can hope to dream :-)

Here is a copy of the original talk. Warts and all.

Cheers,
Andrew

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: