Python and HTTP Pipelining

Attention, this method is NOT pipelining as described in comments, and even might break if the http connection is too fast with httplib.ResponseNotReady. I’ll update this post when i’ll found a real and simple way to achieve pipelining, because one possible way to do it with httplib is really ugly

I wanted to do HTTP Pipelining using urllib2. But, first of all, what is pipelining ?
HTTP pipelining is a technique in which multiple HTTP requests are written out to a single socket without waiting for the corresponding responses.

What is the benefit of pipelining ? Less network load, speedup processing !
I was searching a way to do it with urllib2… But solution are complicated, and not fit well to my needs.
But way, why stay on urllib2 ? Use httplib !

Reusing the connection :

First, reusing the same connection

Second, try pipelining !

Facebook Twitter Delicious Reddit Digg

4 thoughts on “Python and HTTP Pipelining

  1. Actually, if you look at httplib.py, getresponse() creates an HTTPResponse object and calls it’s begin() method which will block until a non-100 response is received and the headers are parsed. Therefore, the “pipelining” example above is not really pipelining. Httplib does not support pipelining out of the box; you need to do a little “magic” to get pipelining to work.

  2. This is *not* http piplining. This is keeping alive a connection. If you could do it, this would be http piplining;

    server.request(‘GET’, ‘/index.html’)
    server.request(‘GET’, ‘/index2.html’)
    res1 = server.getresponse()
    res2 = server.getresponse()

    As the previous comment says, httplib does not support pipelining and says so in its docs.
    The fact this is a top google result for “python http pipelining” is a bit annoying. I know it is good to get traffic, but as it is wrong, can you take it down please?

  3. Thanks for pointing this. I double checked the wireshark output, and yes it’s not pipelining :( My bad. Neither httplib2 and requests seems to support pipelining yet :(

  4. I agree with typingduck
    I see that you have put as a warning message, but it would be good to remove it from the title while you haven’t updated it with the real pipelining ?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">