Attention, this method is NOT pipelining as described in comments, and even might break if the http connection is too fast with httplib.ResponseNotReady. I’ll update this post when i’ll found a real and simple way to achieve pipelining, because one possible way to do it with httplib is really ugly
I wanted to do HTTP Pipelining using urllib2. But, first of all, what is pipelining ?
HTTP pipelining is a technique in which multiple HTTP requests are written out to a single socket without waiting for the corresponding responses.
What is the benefit of pipelining ? Less network load, speedup processing !
I was searching a way to do it with urllib2… But solution are complicated, and not fit well to my needs.
But way, why stay on urllib2 ? Use httplib !
Reusing the connection :
First, reusing the same connection
|
1 2 3 4 5 6 7 8 9 10 |
import httplib server = httplib.HTTPConnection('yourserver.com') server.request('GET', '/index.html') print 'RESPONSE1:', server.getresponse().read() server.request('GET', '/index2.html') print 'RESPONSE2:', server.getresponse().read() server.request('GET', '/index3.html') print 'RESPONSE3:', server.getresponse().read() |
Second, try pipelining !
|
1 2 3 4 5 6 7 8 9 10 11 12 |
import httplib server = httplib.HTTPConnection('yourserver.com') server.request('GET', '/index.html') res1 = server.getresponse() server.request('GET', '/index2.html') res2 = server.getresponse() server.request('GET', '/index3.html') res3 = server.getresponse() print 'RESPONSE1:', res1.read() print 'RESPONSE2:', res2.read() print 'RESPONSE3:', res3.read() |
Actually, if you look at httplib.py, getresponse() creates an HTTPResponse object and calls it’s begin() method which will block until a non-100 response is received and the headers are parsed. Therefore, the “pipelining” example above is not really pipelining. Httplib does not support pipelining out of the box; you need to do a little “magic” to get pipelining to work.
This is *not* http piplining. This is keeping alive a connection. If you could do it, this would be http piplining;
server.request(‘GET’, ‘/index.html’)
server.request(‘GET’, ‘/index2.html’)
res1 = server.getresponse()
res2 = server.getresponse()
As the previous comment says, httplib does not support pipelining and says so in its docs.
The fact this is a top google result for “python http pipelining” is a bit annoying. I know it is good to get traffic, but as it is wrong, can you take it down please?
Thanks for pointing this. I double checked the wireshark output, and yes it’s not pipelining 🙁 My bad. Neither httplib2 and requests seems to support pipelining yet 🙁