使用python向服务器POST大文件

python 对http操作有几个库 urllib 、 urllib2 还有httplib

httplib比较偏底层一般情况下使用urllib和urllib2就行了

NOTICE

在python3中urllib与urllib2被分割合并为了 urllib.request, urllib.parse, and urllib.error

httplib重命名为 http.client

分析http协议

python的这几个库中并没有提供直接上传文件的接口我们先看下普通浏览器是怎么上传文件的这里我在本地创建简单php程序若有提交文件，则打印出文件相关的信息。否则显示一个上传表单表单

<?php
header('Content-Type: text/html; charset=utf-8');
if(!empty($_FILES)){
    var_dump($_FILES);
    exit;
}

?>

<!DOCTYPE HTML>
<html lang="en-US">
<head>
    <meta charset="UTF-8">
    <title></title>
</head>
<body style="text-align:center;">
    <form action="" method="POST" enctype="multipart/form-data" >
        <input type="text" name="username" value="ksc"/> <br/>
        <input type="file" name="file" />
        <input type="submit" value="submit"/>
    </form>
</body>
</html>

<?php

header('Content-Type: text/html; charset=utf-8');

if(!empty($_FILES)){

var_dump($_FILES);

exit;

}

<!DOCTYPE HTML>

<head>

</head>

</form>

</body>

</html>

这里推荐一个抓包工具fildder 可以很方便的抓取htttp数据，并且直观的显示

下面就是抓取到的内容

POST http://localhost/test/postfile.php HTTP/1.1
Host: localhost
Connection: keep-alive
Content-Length: 295
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarynHbCm1TeYE1AHFXv
DNT: 1
Referer: http://localhost/test/postfile.php
Accept-Encoding: gzip,deflate,sdch
Accept-Language: zh-CN,zh;q=0.8,en;q=0.6

------WebKitFormBoundarynHbCm1TeYE1AHFXv
Content-Disposition: form-data; name="username"

ksc
------WebKitFormBoundarynHbCm1TeYE1AHFXv
Content-Disposition: form-data; name="file"; filename="b.txt"
Content-Type: text/plain

我是文件内容
------WebKitFormBoundarynHbCm1TeYE1AHFXv--

POST http://localhost/test/postfile.php HTTP/1.1

Host: localhost

Connection: keep-alive

Content-Length: 295

Cache-Control: max-age=0

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

Origin: http://localhost

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36

Content-Type: multipart/form-data; boundary=----WebKitFormBoundarynHbCm1TeYE1AHFXv

DNT: 1

Referer: http://localhost/test/postfile.php

Accept-Encoding: gzip,deflate,sdch

Accept-Language: zh-CN,zh;q=0.8,en;q=0.6

------WebKitFormBoundarynHbCm1TeYE1AHFXv

Content-Disposition: form-data; name="username"

ksc

------WebKitFormBoundarynHbCm1TeYE1AHFXv

Content-Disposition: form-data; name="file"; filename="b.txt"

Content-Type: text/plain

我是文件内容

------WebKitFormBoundarynHbCm1TeYE1AHFXv--

从第1-13行就是请求消息http头部的内容，然后下面有个空行比较重要的几行是

POST http://localhost/test/postfile.php HTTP/1.1
Host: localhost
Content-Length: 295
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarynHbCm1TeYE1AHFXv

内容数据

POST http://localhost/test/postfile.php HTTP/1.1

Host: localhost

Content-Length: 295

Content-Type: multipart/form-data; boundary=----WebKitFormBoundarynHbCm1TeYE1AHFXv

内容数据

请求消息(Request message)有以下几部分组成

请求行(request line), 例如 GET /images/logo.png HTTP/1.1
请求头(Request Headers) 比如上面的 Host: localhost、 Content-Length: 295
空行
消息主体 message body

NOTICE

请求行与请求头必须以结尾，空行只能有不能有空格什么的在HTTP/1.1协议中，除了Host 所有的请求头都是可选的（当然若上传文件的话，就必须设置了Content-Length和Content-Type了，不然服务器收不到数据的,虽然也能成功响应）

这里message body的类型是multipart/form-data;

boundary 是随机的内容每次请求都不一样， Content-Type为 multipart/form-data; 可同时传输多项数据，而这些数据就是通过 boundary分割开来的

每一项的数据都是’–‘+boundary+换行开始，然后是Content-Disposition: form-data;name=”表单项名”

若是文件的话还有个filename 以及Content-Type，接下来一个空行

最后’–‘+boundary+’–‘+换行结束

到这里整个http请求就结束了

模拟提交数据

其实http协议就是字符串按照约定规则拼接到一起然后服务器再来解析得到数据所以我们自己直接使用socket也能发起了一个http请求

但是有了urllib2我们可以省很多事只需“拼接”内容部分就行了

urllib2.urlopen(url[, data][, timeout])

1 2	urllib2.urlopen(url[, data][, timeout])

url 可以是一个链接或者Request对象 data 就是我们要“拼接的内容”

# coding=utf-8

import urllib2
import mimetypes
import os
import uuid

mimetypes.init()

url='http://localhost/test/postfile.php'
fileFieldName='myfile'
file_path='/a.txt'

boundary='--------ksc'+uuid.uuid4().hex;
print('boundary:'+boundary)
req = urllib2.Request(url)
req.add_header("User-Agent", 'ksc')
req.add_header("Content-Type", "multipart/form-data, boundary="+boundary)
def getdata(boundary,file_path):
    global fileFieldName
    file_name=os.path.basename(file_path)
    file_name=file_name.encode('utf-8')
    file_type= mimetypes.guess_type(file_name)[0] #根据文件名匹配文件的mime类型若没有匹配到会返回None
    if file_type==None:
        file_type = "text/plain; charset=utf-8"

    print file_type
    print file_name
    fileData=open(file_path,'rb').read()
    CRLF='\r\n'
    body = '';     
    body += '--' + boundary + CRLF;
    body += 'Content-Disposition: form-data; name="'+fileFieldName+'"; filename="' + file_name + '"'+CRLF;     
    body += "Content-Type: "+file_type+CRLF
    body += CRLF;     
    body += fileData + CRLF;     
    body += "--" + boundary + "--"+CRLF;  
    print 'body size:{0}'.format(len(body))
    return body

#req.add_header('Content-Length',len(body) ) #urllib2会自动添加
res = urllib2.urlopen(req,getdata(boundary,file_path))
print res.read().decode('utf-8');

# coding=utf-8

import urllib2

import mimetypes

import os

import uuid

mimetypes.init()

url='http://localhost/test/postfile.php'

fileFieldName='myfile'

file_path='/a.txt'

boundary='--------ksc'+uuid.uuid4().hex;

print('boundary:'+boundary)

req = urllib2.Request(url)

req.add_header("User-Agent", 'ksc')

req.add_header("Content-Type", "multipart/form-data, boundary="+boundary)

def getdata(boundary,file_path):

global fileFieldName

file_name=os.path.basename(file_path)

file_name=file_name.encode('utf-8')

file_type= mimetypes.guess_type(file_name)[0] #根据文件名匹配文件的mime类型若没有匹配到会返回None

if file_type==None:

file_type = "text/plain; charset=utf-8"

print file_type

print file_name

fileData=open(file_path,'rb').read()

CRLF='\r\n'

body = '';

body += '--' + boundary + CRLF;

body += 'Content-Disposition: form-data; name="'+fileFieldName+'"; filename="' + file_name + '"'+CRLF;

body += "Content-Type: "+file_type+CRLF

body += CRLF;

body += fileData + CRLF;

body += "--" + boundary + "--"+CRLF;

print 'body size:{0}'.format(len(body))

return body

#req.add_header('Content-Length',len(body) ) #urllib2会自动添加

res = urllib2.urlopen(req,getdata(boundary,file_path))

print res.read().decode('utf-8');

下面是运行后的输出结果

boundary:--------ksc114eaa38291f43fdaca15ab3b4265a82
text/plain
a.txt
body size:388
array(1) {
  ["myfile"]=>
  array(5) {
    ["name"]=>
    string(5) "a.txt"
    ["type"]=>
    string(10) "text/plain"
    ["tmp_name"]=>
    string(24) "D:\xampp\tmp\php151B.tmp"
    ["error"]=>
    int(0)
    ["size"]=>
    int(197)
  }
}

boundary:--------ksc114eaa38291f43fdaca15ab3b4265a82

text/plain

a.txt

body size:388

array(1) {

["myfile"]=>

array(5) {

["name"]=>

string(5) "a.txt"

["type"]=>

string(10) "text/plain"

["tmp_name"]=>

string(24) "D:\xampp\tmp\php151B.tmp"

["error"]=>

int(0)

["size"]=>

int(197)

}

php能正常接收传输的文件了但是这里有个问题，我们拼接数据的时候使用的是

fileData=open(file_path,'rb').read()

1 2	fileData=open(file_path,'rb').read()

直接把文件内容读到内存中去了，小文件还好，要是几百M上G的文件还用这种方法就不行了，这样会把你的内存吃干。

所以我们就需要弄个变通的方法，让它一块一块的发送数据，读一点发送一点。这几个库并没有提供类似的方法实现这一个功能，所以就需要自己动手了。

首先介绍一个python的关键字 yield 关于介绍python yield的文章可以查看Python yield 使用浅析可以通过它来实现一个生成器（generator）来不断的读取文件数据，而且是只在我们需要的时候读取

def read_file(fpath): 
    BLOCK_SIZE = 1024 
    with open(fpath, 'rb') as f: 
        while True: 
            block = f.read(BLOCK_SIZE) 
            if block: 
                yield block 
            else: 
                return

def read_file(fpath):

BLOCK_SIZE = 1024

with open(fpath, 'rb') as f:

while True:

block = f.read(BLOCK_SIZE)

if block:

yield block

else:

return

然后我重写了 httplib 中 HTTPConnection.send(data)使它可以接收一个generator，从里面取数据然后发送下面是原版HTTPConnection.send的源码

def send(self, data):
    """Send data to the server."""
    if self.sock is None:
        if self.auto_open:
            self.connect()
        else:
            raise NotConnected()

    if self.debuglevel > 0:
        print "send:", repr(data)
    blocksize = 8192
    if hasattr(data,'read') and not isinstance(data, array):
        if self.debuglevel > 0: print "sendIng a read()able"
        datablock = data.read(blocksize)
        while datablock:
            self.sock.sendall(datablock)
            datablock = data.read(blocksize)
    else:
        self.sock.sendall(data)

def send(self, data):

"""Send data to the server."""

if self.sock is None:

if self.auto_open:

self.connect()

else:

raise NotConnected()

if self.debuglevel > 0:

print "send:", repr(data)

blocksize = 8192

if hasattr(data,'read') and not isinstance(data, array):

if self.debuglevel > 0: print "sendIng a read()able"

datablock = data.read(blocksize)

while datablock:

self.sock.sendall(datablock)

datablock = data.read(blocksize)

else:

self.sock.sendall(data)

它支持文件类型File Object 或者是一个字符串

下面是个重新版本

def send(self, data):
    """Send data to the server."""
    if self.sock is None:
        if self.auto_open:
            self.connect()
        else:
            raise NotConnected()

    if self.debuglevel > 0:
        print "send:", repr(data)
    blocksize = 8192
    if hasattr(data,'next'):#支持迭代
        for datablock in data:
            self.sock.sendall(datablock)
    elif hasattr(data,'read') and not isinstance(data, array):
        if self.debuglevel > 0: print "sendIng a read()able"
        datablock = data.read(blocksize)
        while datablock:
            self.sock.sendall(datablock)
            datablock = data.read(blocksize)
    else:
        self.sock.sendall(data)

def send(self, data):

"""Send data to the server."""

if self.sock is None:

if self.auto_open:

self.connect()

else:

raise NotConnected()

if self.debuglevel > 0:

print "send:", repr(data)

blocksize = 8192

if hasattr(data,'next'):#支持迭代

for datablock in data:

self.sock.sendall(datablock)

elif hasattr(data,'read') and not isinstance(data, array):

if self.debuglevel > 0: print "sendIng a read()able"

datablock = data.read(blocksize)

while datablock:

self.sock.sendall(datablock)

datablock = data.read(blocksize)

else:

self.sock.sendall(data)

当然若想实现完美的文件提交，还需要做个封装,下面是一个简单的实现，还有写不完善的地方，post数据只能传文件，打算做个python lib封装一下

# coding=utf-8

import httplib
import mimetypes
import os
import uuid
import socket

mimetypes.init()

url='http://localhost/test/postfile.php'
fileFieldName='myfile'
file_path='/temp/a.txt'

content_length=0   
class builder:

    def __init__(self):

        self.boundary='--------ksc'+uuid.uuid4().hex;
        self.boundary='--------kscKKJFkfo93jmjfd0ismf'
        self.ItemHeaders=[];
        self.CRLF='\n'

    def getBoundary(self):
        return self.boundary

    def getHeaders(self):    
        '''return Request Headers'''
        headers={"Content-Type":"multipart/form-data, boundary="+self.boundary
                ,"Content-Length":self.getContentLength()
                }
        return headers        
    def putItemHeader(self,fieldName,file_path):

        file_name=os.path.basename(file_path)
        print file_name
        file_name=file_name.encode('utf-8')
        file_type= mimetypes.guess_type(file_name)[0] #guess file's mimetype
        if file_type==None:
            file_type = "text/plain; charset=utf-8"

        print file_type

        CRLF=self.CRLF
        head = '';     
        head += '--' + self.boundary + CRLF;
        head += 'Content-Disposition: form-data; name="'+fieldName+'"; filename="' + file_name + '"'+CRLF;     
        head += "Content-Type: "+file_type+CRLF
        head += CRLF;
        self.ItemHeaders.append({'head':head,'file_path':file_path})

    def getContentLength(self):    
        length=0
        for item in self.ItemHeaders:
            length+=len(item['head'])+os.path.getsize(item['file_path'])+len(self.CRLF)

        return length+len("--" + self.boundary + "--"+self.CRLF)

    def getdata(self):

        blocksize=4096
        for item in self.ItemHeaders:
            yield item['head']

            fileobj=open(item['file_path'],'rb')
            while True:
                block = fileobj.read(blocksize)
                if block:
                    yield block
                else:
                    yield self.CRLF
                    break

        body = "--" +self.boundary + "--"+self.CRLF;  

        yield body

class MyHTTPConnection(httplib.HTTPConnection):
    def send(self, value):
        if self.sock is None:
            if self.auto_open:
                self.connect()
                print '-----------------ksc: reconnect'
            else:
                raise NotConnected()

        try:
            if hasattr(value,'next'):
                print "--sendIng an iterable"
                for data2 in value:
                    self.sock.sendall(data2)    
            else:
                print '\n--send normal str'
                self.sock.sendall(value) 
        except socket.error, v: 
            if v[0] == 32:      # Broken pipe
                print(v)
                self.close()
            raise
        print '--send end' 

bl=builder()    

bl.putItemHeader(fileFieldName, file_path) 
bl.putItemHeader('bb', u'/temp/贪吃蛇.zip') 
#bl.putItemHeader('zipfile', u'/temp/a.zip') 

#注意web端post数据大小限制  
#这里 php.ini post_max_size = 30M

content_length=bl.getContentLength()
print "content_length:"+str(content_length)

url='/test/postfile.php'
#这里使用fiddler捕捉数据包，设置了fiddler为代理
fiddler_monitor=True
fiddler_monitor=False

if fiddler_monitor:
    httpconnect=MyHTTPConnection('127.0.0.1',8888)
    url='http://localhost'+url
else:
    httpconnect=MyHTTPConnection('localhost')

httpconnect.set_debuglevel(1)

headers = bl.getHeaders()
data    = bl.getdata()

httpconnect.request('POST', url, data ,headers)

httpres=httpconnect.getresponse()
print httpres.read().decode('utf-8')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

# coding=utf-8

import httplib

import mimetypes

import os

import uuid

import socket

mimetypes.init()

url='http://localhost/test/postfile.php'

fileFieldName='myfile'

file_path='/temp/a.txt'

content_length=0

class builder:

def __init__(self):

self.boundary='--------ksc'+uuid.uuid4().hex;

self.boundary='--------kscKKJFkfo93jmjfd0ismf'

self.ItemHeaders=[];

self.CRLF='\n'

def getBoundary(self):

return self.boundary

def getHeaders(self):

'''return Request Headers'''

headers={"Content-Type":"multipart/form-data, boundary="+self.boundary

,"Content-Length":self.getContentLength()

}

return headers

def putItemHeader(self,fieldName,file_path):

file_name=os.path.basename(file_path)

print file_name

file_name=file_name.encode('utf-8')

file_type= mimetypes.guess_type(file_name)[0] #guess file's mimetype

if file_type==None:

file_type = "text/plain; charset=utf-8"

print file_type

CRLF=self.CRLF

head = '';

head += '--' + self.boundary + CRLF;

head += 'Content-Disposition: form-data; name="'+fieldName+'"; filename="' + file_name + '"'+CRLF;

head += "Content-Type: "+file_type+CRLF

head += CRLF;

self.ItemHeaders.append({'head':head,'file_path':file_path})

def getContentLength(self):

length=0

for item in self.ItemHeaders:

length+=len(item['head'])+os.path.getsize(item['file_path'])+len(self.CRLF)

return length+len("--" + self.boundary + "--"+self.CRLF)

def getdata(self):

blocksize=4096

for item in self.ItemHeaders:

yield item['head']

fileobj=open(item['file_path'],'rb')

while True:

block = fileobj.read(blocksize)

if block:

yield block

else:

yield self.CRLF

break

body = "--" +self.boundary + "--"+self.CRLF;

yield body

class MyHTTPConnection(httplib.HTTPConnection):

def send(self, value):

if self.sock is None:

if self.auto_open:

self.connect()

print '-----------------ksc: reconnect'

else:

raise NotConnected()

try:

if hasattr(value,'next'):

print "--sendIng an iterable"

for data2 in value:

self.sock.sendall(data2)

else:

print '\n--send normal str'

self.sock.sendall(value)

except socket.error, v:

if v[0] == 32: # Broken pipe

print(v)

self.close()

raise

print '--send end'

bl=builder()

bl.putItemHeader(fileFieldName, file_path)

bl.putItemHeader('bb', u'/temp/贪吃蛇.zip')

#bl.putItemHeader('zipfile', u'/temp/a.zip')

#注意web端post数据大小限制

#这里 php.ini post_max_size = 30M

content_length=bl.getContentLength()

print "content_length:"+str(content_length)

url='/test/postfile.php'

#这里使用fiddler捕捉数据包，设置了fiddler为代理

fiddler_monitor=True

fiddler_monitor=False

if fiddler_monitor:

httpconnect=MyHTTPConnection('127.0.0.1',8888)

url='http://localhost'+url

else:

httpconnect=MyHTTPConnection('localhost')

httpconnect.set_debuglevel(1)

headers = bl.getHeaders()

data = bl.getdata()

httpconnect.request('POST', url, data ,headers)

httpres=httpconnect.getresponse()

print httpres.read().decode('utf-8')

现在有很多现成的第三方库比如 poster Requests 可以实现这些功能了，但是由于是第三方库需要用户手动安装，在一个我需要上传大文件时有个回调功能，这样就能显示进度了。等运行稳定了会添加到kuaipan cli 里面，就可以摆脱poster依赖

参考资料

HTTP

geekli

Keep It Simple, Stupid

标签归档：post

使用python向服务器POST大文件

使用python向服务器POST大文件

分析http协议

模拟提交数据