Archive for the ‘Uncategorized’ Category

The Montreal Python 26 Talk

December 21, 2011

Hi Folks

I recently gave a talk for Montréal Python 26 (Jolly Justifications!) appropriately called “How Do You Solve A Problem Like Santa: Prototyping Join Patterns with stackless.py for Stackless Python.” Here are the slides for How To Solve a Problem Like Santa. Once again, I would like to thank L’equip de Montréal Python and the wonderful audience for providing me with an opportunity to share.

Seasons Greetings,
Andrew

The Initial Version of Stackless.py with Select

June 9, 2010

Hi Folks:

Introduction:

Enclosed is a link to the initial copy of Stackless.py with select. This is different from the EventHandler class I presented at the dry run at Montreal Python 13 on April 26th. That version was completely clean roomed from Robert Pike’s Implementation of Newsqueak paper. Although I wrote the implementation, most of the credit has to be given to my friend Kevin Bulušeck for doing a clever Stackless Python mockup that I referenced. I also have to thank Russ Cox for pointing me to the Plan9 implementation at http://swtch.com/usr/local/plan9/src/libthread/channel.c and answering a few questions.

Usage

Select is now a tasklet method.


tasklet.select(list of chanops)
"""returns (channel, operation, value).

channel.receiveCase()
"""returns a receive chanop

channel.sendCase()
"""returns a send chanop

An example:

import stackless

def sender(ch, val):
    ch.send(val)

def receiver(ch):
    val = ch.receive()

def selector(a, b, c):

    print "selector: begin"
    count = 0
    while count < 3:
        ch, operation, value =  stackless.getcurrent().select([a.sendCase("A"), \
                                                       b.receiveCase(), \
                                                       c.sendCase("C")])
        if ch == a:
           print "sender A completed"
        elif ch == b:
           print "received ", value, "from receiver B"
        elif ch == c:
           print "sender C completed"
        else:
           print "should not get here"

        count += 1

    print "selector: end"

if __name__ == '__main__':
    a = stackless.channel("CHANNEL-A")
    b = stackless.channel("CHANNEL-B")
    c = stackless.channel("CHANNEL-C")

    stackless.tasklet(selector, "SELECTOR")(a, b, c)
    stackless.tasklet(receiver, "RECEIVER-A")(a)
    stackless.tasklet(sender,"SENDER-B")(b, "B")
    stackless.tasklet(receiver,"RECEIVER-C")(c)
    stackless.run()

Of course, a hash table can be used in lieu of if statements. Also unlike the Go/Newsqueak version, cases can be dynamically added and removed. I believe the only real limitation is that a channel cannot do both a sendCase and receiveCase at the same time.

Under the hood, the main changes are: 1) there is a new object, chanop (channel operation) 2)channel has two new methods: receiveCase() and sendCase() that return chanops. Chanop is meant to be manipulated by stackless and not the programmer. 3) channel now has a queue of operations rather than tasklets.

I would appreciate it if folks could download the code, play with it and make comments. Remember the code is still rather rough. It would be nice to see how easy/hard it is to implement select in other stackless.py based implementations 🙂 In the meanwhile, I will write tests and clean up the code. I also want to further experiment with implementing join objects a la JoCAML and Polyphonic C#.

Cheers,
Andrew

Prototyping Go’s Select for Stackless Python with Stackless.py

May 2, 2010

Dear Folks:

On April 26th, 2010, I did a dry run of the talk Prototyping Go’s Select for Stackless Python with Stackless.py. I thank everyone who attended my talk despite the hockey game being on at the same time. I also deeply appreciate the feedback! Hopefully this talk will be accepted for EuroPython 2010. The motivation for the talk was simple. Stackless Python and Google’s new language, Go, borrow their concurrency model from Bell Lab’s Limbo. However Stackless Python does not have the select statement. Select allows a coroutine to monitor multiple channels without resorting to polling. I find the idea of Stackless Python written Python to be a powerful idea. How hard would it be to implement select functionality in Stackless Python? How easy would it be to use PyPy’s stackless.py package to prototype the feature? Does stackless.py deviate so much from Stackless Python as not be a good estimator of required time and effort?

As Stackless Pythonistas pointed out, select like functionality could be implemented with additional tasklets. I understand this approach fits in with the tao of Stackless Python to customise through subclassing and using the API. Keep Stackless Python small. For the most part I agree. However in the case of select, how efficient would this be? It has also been argued that no one wants select. Well we all know the adage about hackers wanting to write software they will use. Based on my work with WS-BPEL and business process orchestration, I have implemented workarounds. So I see uses for select like abilities in the areas of complex event processing and workflow patterns. Perhaps these features could make Stackless Python attractive to a new audience?

We will soon be in a position to experiment and benchmark….

Incidentally I forgot to tell the audience that I tended to submit the talk to EuroPython 2010 and I wanted to iron out the bugs. My apologies. Based on feedback, I will restructure the talk. However I will post the original slides and code soon. Really. For better or worse, the video will be out there.

Prototyping is still very much a work in progress. I am still learning about many things about the Go language and Stackless Python internals. For instance, Go does expose more of the concurrency system than I thought, through its runtime and reflect packages. Although it doesn’t alter the big picture, this new information will be reflected in the updated talk and paper.

As of April 26th, I have implemented a few versions of a select like feature in stackless.py. For now select (called eventHandler) is a class rather than a language feature. These implementations have been very conservative. I will write more versions in the days and weeks to follow. Meanwhile, my friend Kevin Bulušek has bravely taken the plunge and is implementing the changes in the Stackless Python implementation to support a select. From working with Kevin, I was reminded of a line uttered by Ben Affleck’s character Morgan from the film Good Will Hunting comes to mind: “My boy’s wicked smart.”

To date, Prototying is the most esoteric of the three talks I have given. It is the most fun to research and do. A lot is happening in the background. Some the action won’t get into a thirty or forty-five minute talk. For instance, I think the Agile technique of two man programming is really effective. I am also excited about Christian’s work with integrating Stackless Python with Pscyo. yet another reason to avoid programming Python extensions in C. And my C is really rusty. It was a blast to take the algorithms out of Rob Pike’s paper The Implementation of Newsqueak and implement them in Python. Although stackless.py is a low priority concern for the PyPy team (and not maintained), I think there is much potential in using stackless.py. And each time I look at the PyPy blog, the team seems to be making significant gains.

Most important for the current project, Stephan Diehl and Carl Bolz pointed out an important trick-of-the-trade: use stackless.py with the greenlets package running on top of Standard Python. Previously I used the PyPy interpreter that was so slow. And forget compiling to pypy-c. It is all about high clockspeed (a book I am currently reading, recommended by a former professor). Thanks guys! Christian Tismer also gave helpful advice and was correct in saying that the C code base would have to altered to support select. I originally thought we could get away with no changes….. My first prototype gave me the wrong impression.

During the course of Prototyping, I attended a McGill Management school talk by John Seeley Brown called “Power of Pull: How Small Moves, Smartly Made, Can Set Big Things in Motion.” The aforementioned is the name of Brown’s new book in collaboration with Jon Hagel. I had heard of Jon Hagel before because I read “Out of the Box” that dealt with web services in business. This topic was right up my alley. A central theme of Brown’s talk was about how to create high performance organisations. Drawing from cases ranging from elite surfers, to Amazon, to WoW guiles, Brown listed some simple principles:

  • Deep collaboration.
  • Fail fast
  • Using frame-by-frame analysis to improve
  • Look at adjacencies.
  • Look for spikes of capabilities.

A lot of the principles applied to my current adventure. For instance, prototyping in with stackless.py allowed me to fail fast. . Believe me, I made many mistakes. The revision control system has the bones of many versions nipped in the bud. It helped that Kevin acted as a springboard and a sanity check. Occasionally Kevin would say, remember the KISS principle and all I could do is sheepishly grin (funny I just read a New York Times article called “It’s Complicated: Making Sense out of Complexity” that talked about American’s love affair with the complicated). As for adjacencies, I closely looked at Rob Pike “Newsqueak” paper and reviewed the various Google videos on Newsqueak and Go. And I have dabbled with Erlang. Perhaps the principle that didn’t dawn on me earlier was “look for spikes of capabilities.” I decided to be audacious and subscribe to the Go Language Nuts newsgroup and post questions. I was pleasantly surprised when ‘Commander’ Rob Pike himself, Russ Cox and Ian Taylor, kindly indulged me and answered my questions. Looking at the Plan 9 code was really really helpful although there are parts like the lock acquisition that I still haven’t wrapped my head around it. At the end of the day, Stackless Python and Go are *different* languages built for different purposes despite having a common ancestor. Nevertheless. I feel Newsqueak, Plan 9, and Go will show where the dragons may be for a C Stackless Python implementation. Perhaps once I have enough of a framework in place, I can find optimisations: one can hope to dream 🙂

Here is a copy of the original talk. Warts and all.

Cheers,
Andrew

A Survey of Stackless Python First Edition

October 14, 2009

Hi Folks:

On September 30th, 2009 I gave a talk entitled A Survey of Stackless Python to the Montreal Python Group. I thank the Montreal Python Group for having me. And it is always a pleasure to talk about Stackless Python. The talk was well received despite hiccups. I am sure I will cringe when I watch the video. As a teenager, I cringed watching my performance on a TV game show (even though my team won). I believe I fumbled the part on channel preferences execution and response times. As well as pronouncing Gregorio’s name. I kept on thinking ‘Grogono’ (A computer science professor at Concordia University). I am sure there are more gaffs.

I am tempted to say that the slides are better than the video.

A number of cool talks inspired “Survey”: Joe Gregorio’s (Lack of) Design Patterns in Python, David Beazley’s A Curious Course on Coroutines and Robert Pike’s Advanced Topics in Programming Languages: Concurrency and Message Passing in Newsqueak. Also at the time, I had design patterns on my mind as a part of reading Bruce Eckel’s work in progress Python 3 Idioms and Design Patterns. One of my favourite computer science courses was Software Design Patterns with Dwight Deugo.

What I liked about the three talks is the way they interconnect. Towards the end of his really informative talk, David Beazley raises some legitimate questions and comments: “Is it really worth it to write your own multi-tasker?” and “Threads are a well understood model.” With Gregorio, the answer is “well the language should incorporate the mult-tasker design pattern as a feature.” And Pike: “threads are low level” and “(channels as a part of) CSP, is a well understood model.”

I feel the cumulation of the ideas in these talks provide the framework for discussing Stackless Python and other concurrency frameworks in a more systematic fashion. Essentially Stackless Python provides a powerful built-in design pattern : The Active Object. And the relevantly easy to implement but lesser known Half-Sync Half-Async pattern gives the Stackless Programmer (and his or her choice of asynchronous networking package) a best of both worlds between synchronous and asynchronous concurrency models. This is a major theme of the talk.

I know I am bad on following up on blogs and talks. However in the weeks to come, I will be posting a new version of the talk. I want to clarify some facts – i.e., memory usage and a tasklet’s state machine (there is a little more to it). I want to provide a benchmarks between say Stackless Python and a generator based solution. I want to include a section on pickling. I want to tie all the networking techniques together through the use of a small web server (I would like to see what a Newsqueak web server looks like). To provide context, I will discuss about what got me into the wonderful world of Stackless Python (Service Oriented Architectures). Finally, I want to do some learning (then again, I am always learning something new with Stackless) in the form of seeing how easy (or difficult) it is to add new language features via the future of Stackless, PyPy.

Cheers,

Andrew

The Adventures in Stackless Python and Twisted Continue…

March 18, 2008

Hello Folks:

I recently gave two talks at PyCon 2008. The first was “Stackless 101” where I was a substitute teacher for Christian Tismer. If it weren’t for Christian Tismer and the wonderful work he has done, I wouldn’t have had any Python adventures worth telling. My own talk was entitled “Adventures in Stackless Python/Twisted Integration.” It was difficult to cover all the issues in a twenty five minute talk. Unfortunately there are problems with the original slides.

However this is the Web. There is no reason why I can’t use the feedback to correct mistakes and strengthen arguments. It would be a bigger mistake to leave the slides as is and let important things go unsaid.

In the days and weeks to follow, I will post a new edition of “Adventures in Stackless Python / Twisted Integration” I will also post coding examples.

That said, I cannot stress enough the fearless programming aspects of Python, Stackless Python and Twisted. I first learnt about the notion of “Fearless Programming” in the May 1987 IEEE Software article “Experimental Prototyping in Smalltalk.” I have long since forgotten Smalltalk but I still remember the sense of fearlessness. A fearlessness that I now experience with Python. Moreover prototyping WS-BPEL constructs with Stackless Python and Twisted allowed me to get a far better grasp of the issues than sitting on committee meetings!

I guess the best way to learn a specification is to implement it….

Yes there is some investment involved in learning Stackless Python and Twisted. However one will be amazed at the type of programming projects one can tackle with even a little Stackless Python and Twisted. One of the goals of this blog is to encourage the newbie to hang in there.

Cheers,
Andrew