Python and HTTP Pipelining

Attention, this method is NOT pipelining as described in comments, and even might break if the http connection is too fast with httplib.ResponseNotReady. I’ll update this post when i’ll found a real and simple way to achieve pipelining, because one possible way to do it with httplib is really ugly

I wanted to do HTTP Pipelining using urllib2. But, first of all, what is pipelining ?
HTTP pipelining is a technique in which multiple HTTP requests are written out to a single socket without waiting for the corresponding responses.

What is the benefit of pipelining ? Less network load, speedup processing !
I was searching a way to do it with urllib2… But solution are complicated, and not fit well to my needs.
But way, why stay on urllib2 ? Use httplib !

Reusing the connection :

First, reusing the same connection

import httplib
server = httplib.HTTPConnection('')
server.request('GET', '/index.html')
print 'RESPONSE1:', server.getresponse().read()

server.request('GET', '/index2.html')
print 'RESPONSE2:', server.getresponse().read()

server.request('GET', '/index3.html')
print 'RESPONSE3:', server.getresponse().read()

Second, try pipelining !

import httplib
server = httplib.HTTPConnection('')
server.request('GET', '/index.html')
res1 = server.getresponse()
server.request('GET', '/index2.html')
res2 = server.getresponse()
server.request('GET', '/index3.html')
res3 = server.getresponse()

print 'RESPONSE1:',
print 'RESPONSE2:',
print 'RESPONSE3:',

Color swapping and Python

For PyMT, Sharath need BGR support into PyMT, while his graphic card don’t support GL_BGR.
Well, after adapting swap code from Pyglet sourcecode, he say: “< patali> it works but very slow”

The goal is to swap a string from ‘bgrbgrbgr’ to ‘rgbrgbrgb’. Just let’s do a rapid benchmark.

import sys
import re
import time

swap1_pattern = re.compile('(.)(.)(.)', re.DOTALL)
swap1_repl = r'\3\2\1'
def swap1(bytes):
    return swap1_pattern.sub(swap1_repl, bytes)

def swap2(bytes):
    out = ''
    for i in xrange(0, len(bytes), 3):
        b, g, r = bytes[i:i+3]
        out += r
        out += g
        out += b
    return out

def swap3(bytes):
    bytes = list(bytes)
    for i in xrange(0, len(bytes), 3):
        b, g, r = bytes[i:i+3]
        bytes[i:i+3] = r, g, b
    return ''.join(bytes)

def swap4(bytes):
    blues = bytes[0::3]
    greens = bytes[1::3]
    reds = bytes[2::3]
    return ''.join(''.join(x) for x in zip(reds, greens, blues))

def swap5(bytes):
    from array import array
    a = array('b', bytes)
    a[0::3], a[2::3] = a[2::3], a[0::3]
    return a.tostring()

def swap6(bytes):
    import numpy
    a = numpy.array(bytes, 'c')
    b = a[...,::-1]
    return b.tostring()

def swap7(bytes):
    a = list(bytes)
    a[0::3], a[2::3] = a[2::3], a[0::3]
    return ''.join(a)

def bench(func, bytes):
    sys.stderr.write('Bench %s: ' % str(func))

    start = time.time()
    for i in xrange(20):
        ret = func(bytes)
        if ret[:3] != 'rgb' or ret[-3:] != 'rgb':
            sys.stderr.write('INVALID DATA, start=%s, end=%s' %
                            (ret[:3], ret[-3:]))

    end = time.time() - start
    sys.stderr.write('| Finished in %.4f
' % end)
    return ret

bytes = 'bgr' * 256 * 256
bench(swap1, bytes)
bench(swap2, bytes)
bench(swap3, bytes)
bench(swap4, bytes)
bench(swap5, bytes)
bench(swap6, bytes)
bench(swap7, bytes)

Swap1() is actually the way of pyglet. swap2/swap3 is from me, and others is found on some threads in pygame mailing list.

So, who win ?

21:09 tito@ashaka ~ $ python 
Bench <function swap1 at 0x7f1d154632a8>: ....................| Finished in 3.2713
Bench <function swap2 at 0x7f1d15463230>: ....................| Finished in 1.2860
Bench <function swap3 at 0x7f1d15470cf8>: ....................| Finished in 0.6535
Bench <function swap4 at 0x7f1d15470e60>: ....................| Finished in 0.6475
Bench <function swap5 at 0x7f1d15470ed8>: ....................| Finished in 0.0367
Bench <function swap6 at 0x7f1d15470f50>: ....................| Finished in 0.1959
Bench <function swap7 at 0x7f1d15473050>: ....................| Finished in 0.1482

The array solution is a lot faster than any other implementation ! And it’s python standard πŸ™‚ Let’s take this…

Edit: thanks for Milan to come with another awesome solution !

def swap0(bytes):
  return bytes[::-1]

Result is: 0.0059 !

Spreading PyMT toolkit

I’ve been some week without blogging, but lot of things happened ! Finally, i’ve been able to see Thomas in IRL at Google Mentor Summit in San Francisco. It was an awesome week-end, funny, creative and geeky ! Lot of ideas, and goal : making PyMT more stable and well designed : the roadmap of PyMT 0.4 is finally available. We’ve also done a little session too at summit.

Thomas is now 3 days to ITS 2009 in Banff Canada, for a conference about PyMT. The paper will be available for everyone soon πŸ™‚ And this week-end, i’m going to Edinburgh with Lucie to meet Mark, and see the awesome new multitouch room of Napier University. Sharath is also going next week to a FOSS in India to spread PyMT too πŸ™‚

Theses days are really cool πŸ™‚

San Francisco, i’m coming !

Here is a very good news, Christian, Pawel and Nuigroup support me to go on the Google Mentor Summit, in 23/24 October ! I’ll finally meet all of them, and specially Thomas, the creator of awesome PyMT framework πŸ™‚

This is the result to take care of Sharath during his GSOC πŸ™‚

I’ll stay 2 days more with Thomas, to hack the future PyMT :p (well, work isn’t the real word… i want to see California a little :p)

Thanks Guys !

Experimenting widget animation with Fbo

PyMT provide 2 way to make animations :

  • property animation with Animation() class, can be created with start_animation() / stop_animation(),
  • inner animation, that watch changes on a property and start automatically an animation.

They are things we cannot do actually :

  • Fade in/out a widget and their children
  • Scale the widget content

So, i was thinking to use Fbo to make more complex animation on a widget.

Before starting the how-to, here a demo-video.

PyMT – New desktop with menu animation


Comment booster votre linux ? Installer un SSD…

Ancien disque dur SEAGATE ST3750630AS 750Go :

$ sudo hdparm -tT /dev/sda
  Timing cached reads:   7224 MB in  2.00 seconds = 3616.79 MB/sec
  Timing buffered disk reads:  304 MB in  3.00 seconds = 101.17 MB/sec
$ sudo ~/seeker /dev/sda1
Seeker v2.0,
Benchmarking /dev/sda1 (14456MB), wait 30 seconds.......
Results: 114 seeks/second, 8.77 ms random access time

Nouveau disque dur SSD G.SKILL Falcon 64Go + 64Mo cache :

$ sudo hdparm -tT /dev/sdb
 Timing cached reads:   7056 MB in  2.00 seconds = 3532.79 MB/sec
 Timing buffered disk reads:  528 MB in  3.00 seconds = 175.71 MB/sec
$ sudo ~/seeker /dev/sdb2
Seeker v2.0,
Benchmarking /dev/sdb2 (51513MB), wait 30 seconds......
Results: 5452 seeks/second, 0.18 ms random access time

J’ai dΓ©placΓ© / et /home sur le nouveau disque, l’ancien reste lΓ  pour le stockage de donnΓ©e sur /usr/local πŸ™‚

Ca marche du feu de dieu !