<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,Arial,'Hiragino Sans GB','Microsoft YaHei',simsun;vertical-align:baseline;color:rgb(93,93,93);background-color:rgb(255,255,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">只有MD5的超
链接可以点，点进去看看，<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">我写这个的爬虫的目的是下载样本，只关心样本的下载地址，并不关心其他的信息。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">现在我们可以理清下思路：<pre style="margin-bottom:0px;padding-right:0px;padding-left:0px;border:0px;font-size:18px;line-height:inherit;font-family:inherit;vertical-align:baseline;color:rgb(93,255);"><pre class="source-code" style="margin-bottom:0px;padding:15px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;background:rgb(243,244,245);"><code style="margin:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">
获取每一页的网页源码
解析当前页的每一个<span style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;color:rgb(0,134,179);">md5对应的详细信息
链接
在详细信息
页面解析下载地址。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
提取规则<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">先从第一页开始爬起<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">scrapy
支持xss选择器和css选择器，一般是用哪个比较方便就用哪个。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">对应
链接的xpath选择器为 <code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">//td/a/@href ,<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
下一页的
链接的xpath选择器为 <code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;">//a[contains(text(),'Next')]/@href<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">登陆<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">之前写爬虫的时候，是可以直接通过post登陆的，而我开始写这篇
文章的时候开始
加上了google的ReCaptcha验证码。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">只能换种方式使用selenium来登陆Malwr.得到登陆后的cookie后，我们就可以用这个cookie开始下载样本了。<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">FilesPipeline是从file_urls中拿到下载地址然后开始下载
文件，因此我们还需要Item.py定义一个file_urls.<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">解析下载地址，将解析到的下载
链接列表放入item的file_urls,返回item。pipeline会file_urls拿到
链接开始
调用get_media_requests下载
文件<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
<p style="margin-top:1em;margin-bottom:0px;border:0px;font-size:18px;line-height:inherit;font-family:helvetica,255);">
