CWYAlpha

Just another WordPress.com site

Thought this was cool: gevent 动态添加下载任务的研究

leave a comment »


最近在做一个站点,需要在网络上收集一些资料。干这活用python很合适,于是就准备用gevent+requests实现一个爬虫。以前曾用gevent抓取过漫画,但那时的url都是先生成的,并非动态添加,所以需要解决gevent的动态添加任务的问题。

普通的gevent下载方式,gevent.pool控制并发,动态添加下载地址

详细:http://blog.plotcup.com/a/82


在微博上关注:
新浪,
腾讯
 
投稿

最新招聘

更多>>


from python.cn(jobs, news): http://simple-is-better.com/news/908

Written by cwyalpha

六月 26, 2012 在 4:47 下午

发表在 Uncategorized

发表评论

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / 更改 )

Twitter picture

You are commenting using your Twitter account. Log Out / 更改 )

Facebook photo

You are commenting using your Facebook account. Log Out / 更改 )

Google+ photo

You are commenting using your Google+ account. Log Out / 更改 )

Connecting to %s

%d 博主赞过: