设计图

沙发

赞！

mac下自带的python是不是要安装什么依赖包

赞

顶

屌。赞一个

赞个

mark

灭绝人性的东西？

感觉这个是守株待兔式的找漏洞啊

深夜福利呀！
我打赌依赖包又都贴心地打包进repo里了

深夜福利呀！

配合之前那几个已经成功一步步打造神器中。但是框架跟api不会搞。猪猪侠来教我下

不能批量啊

赞一个

大恩不言谢，猪猪侠

赞一个

漂亮

赞，猪猪

Mac系统下无法安装python-devel libxml2-devel libxslt-devel  所以会出现：IOError: [Errno 2] No such file or directory: './dict/package_ext.lst'

帅

前排支持！

requirements.txt 内容大概不对，一般来说是这样。
tornado>=4.0
sqlalchemy>=0.9.7
jinja2>=2.7.3

这样可以方便的利用 pip install -r requirements.txt 安装，同时 PyCharm 可以正确的识别。

good job

支持

 猪哥你把requestment 当成make来用了

@Nicky = =不对 其实是目录配置问题

感谢猪猪侠大神~

赞一个

对https是不是支持有问题？

感谢

运行出现
Traceback (most recent call last):
  File "wyspider.py", line 17, in <module>
    print json.dumps(start_wyspider(sys.argv[1]), indent=2)
  File "/root/weakfilescan/controller.py", line 78, in start_wyspider
    possibility = checksite_possibility(httpurl)
  File "/root/weakfilescan/common.py", line 65, in checksite_possibility
    httpres = http_request_get(siteurl.rstrip('/')+tempurl)
  File "/root/weakfilescan/common.py", line 39, in http_request_get
    allow_redirects=allow_redirects)
  File "/root/weakfilescan/libs/requests/sessions.py", line 473, in get
    return self.request('GET', url, **kwargs)
  File "/root/weakfilescan/libs/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/weakfilescan/libs/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/root/weakfilescan/libs/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
libs.requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

有些是探测两三个就这样，有些就直接这样。还有些是直接卡死- -是我没安装yum install python-devel libxml2-devel libxslt-devel 这个的原因么，kali运行提示没有yum这个命令- -百度无果- -我看可以运行就没管了，

@东方不败 @D＆G 才刚写好，确实各种不稳定，在改改改

@猪猪侠 好的，幸苦猪猪侠大神了

@小博  http://stackoverflow.com/questions/26473197/how-to-install-libxml2-dev-libxslt-dev-on-mac-os

加了点代码，就可以从txt读取然后挂着去喝咖啡了~~
if __name__ == "__main__":
  if len(sys.argv) == 2:
    print '2'
    print json.dumps(start_wyspider(sys.argv[1]), indent=2)
    sys.exit(0)
  elif len(sys.argv)==3:
    for i in open(sys.argv[2]):
      print i
      print json.dumps(start_wyspider(str(i).strip()), indent=2)
    sys.exit(0)
  else:
    print ("usage: %s http://wuyun.org" % sys.argv[0])
    print ("usage: %s -u url.txt" % sys.argv[0])
    sys.exit(-1)

猪猪 辛苦了 不错

@东方不败 'kali运行提示没有yum这个命令- -百度无果-'  kali又不是centos,怎么会有yum, 试试apt-get

赞~~

希望加入超时自动停止和结果导出功能~thx~

mk

@猪猪侠 猪哥 kali下如何安装呀  kali运行提示没有yum这个命令

顶猪哥

猪哥也是个程序猿。。。

修复了几个大的BUG，各位可以  git pull 更新一下再使用
1、解决了不支持HTTPS的问题
2、解决了checksite_possibility()函数报错的问题

更新了config.py的配置项，剔除掉403
# 无错爬取
python wyspider.py https://mail.qq.com

@猪猪侠 是不是有些问题啊？


[200] http://zone.wooyun.org/zone/wooyun.bz2 => http://zone.wooyun.org/login


[root@lax-01 weakfilescan]# curl -i http://zone.wooyun.org/zone/wooyun.bz2
HTTP/1.1 302 Moved Temporarily
Date: Fri, 03 Apr 2015 19:42:21 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d4bb4dd27532c34ad259ff375546604b11428090140; expires=Sat, 02-Apr-16 19:42:20 GMT; path=/; domain=.wooyun.org; HttpOnly
Set-Cookie: PHPSESSID=hqsh2s54gc943b1949d35ptuc4; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: http://zone.wooyun.org/login
Server: yunjiasu-nginx
CF-RAY: 1d174193084f0af0-WAW

@猪猪侠 求放出Scrapy+ bloom filter 规则，爬虫循环目录扫描。。大杀器

@Mujj 因为 /zone/目录下有.htaccess来重定向后面的内容，就像 http://www.wooyun.org/whitehats/weakfilescan ，
需要你修改config.py 里面的
page_not_found_reg = r'404|(n|N)ot\ (f|F)ound|不存在|未找到|Error'
，来自定义操蛋的url重定向，呵呵

@猪猪侠 报错了Traceback (most recent call last):
  File "wyspider.py", line 20, in <module>
    print json.dumps(start_wyspider(sys.argv[1]),indent=2)
  File "/root/Desktop/weakfilescan-master/controller.py", line 81, in start_wyspider
    possibility = checksite_possibility(httpurl)
  File "/root/Desktop/weakfilescan-master/common.py", line 76, in checksite_possibility
    is_redirect = True if len(httpres.history) > 0 else False
AttributeError: 'NoneType' object has no attribute 'history'

@Jumbo 是最新版吗？扫的哪个网站，我测试一下加个全局错误过滤

@猪猪侠 恩 对，我测试了其他的，扫出来以后还是会报错，加个错误过滤

@Jumbo 改好更新了，你git pull试试新版本看看

猪猪侠的东西不会错

@猪猪侠 好的。

windows下使用。报错了。。

每次有人骂我是猪我都说自己是猪猪侠

@Mujj 新代码加入了误报清洗功能，某个目录返回的结果超过config.py里面resulst_cnt_val定义的个数，会自动被清除掉。

@猪猪侠
python wyspider.py http://www.youmi.net
--------------------------------------------------
* scan http://www.youmi.net start
--------------------------------------------------
Traceback (most recent call last):
  File "wyspider.py", line 20, in <module>
    print json.dumps(start_wyspider(sys.argv[1]),indent=2)
  File "/root/weakfilescan/controller.py", line 41, in start_wyspider
    link_datas = GetAllLink(siteurl).start()
  File "/root/weakfilescan/libs/GetAllLink.py", line 47, in start
    response_obj = LinksParser(http_request_get(self.siteurl))
  File "/root/weakfilescan/common.py", line 157, in __init__
    self.baseurl = get_baseurl(self.url)
  File "/root/weakfilescan/common.py", line 31, in get_baseurl
    netloc = urlparse.urlparse(link).netloc
  File "/usr/lib/python2.7/urlparse.py", line 140, in urlparse
    tuple = urlsplit(url, scheme, allow_fragments)
  File "/usr/lib/python2.7/urlparse.py", line 179, in urlsplit
    i = url.find(':')

@几何黑店
SSLError: hostname 'www.youmi.net' doesn't match either of 'campus.youmi.net', 'www.campus.youmi.net'

服务器开启强制跳转到https，问题是他服务器的证书不匹配，我更新下代码，忽略证书检测

@几何黑店 好了，你git pull下，SSL的warning可以不用理会

@猪猪侠
python wyspider.py http://www.youmi.net
--------------------------------------------------
* scan http://www.youmi.net start
--------------------------------------------------
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/root/weakfilescan/libs/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html

@几何黑店 正常的警告，不用理会的

@几何黑店 好了，我把requests官方的类修改了，就为了屏蔽这个错误，哎

设计图叼得飞起.收下膝盖

@小博 pip

赞一个！

我是来看4L头像的。

大杀器。。。

遇上安全狗会汪汪汪！没什么用　　　但是，我真心觉得很不错的一个工具！　谢谢分享！

@东方不败 look here http://blog.csdn.net/kezhen/article/details/42967143

@猪猪侠
from requests.packages import urllib3
urllib3.disable_warnings()

@纷纭 好人一生平安！ 我要你给我生猴子

@猪猪侠  Windows下报错了

@东方不败 apt-get install

哈喽

辛苦辛苦福利啊

@猪猪侠

@猪猪侠 不用修改官方的。在你主文件开头加上这个吧。 而且官方正在最新版本中计划加个参数可以ignore这个告警信息。
import warnings
from requests.packages.urllib3.exceptions import InsecureRequestWarning
from requests.packages.urllib3.exceptions import InsecurePlatformWarning

warnings.filterwarnings(action="ignore", message="Unverified HTTPS request.*", category=InsecureRequestWarning)
warnings.filterwarnings(action="ignore", message="A true SSLContext object.*", category=InsecurePlatformWarning)

学习了，拿了东西也不感谢下的

好强大的神器,感谢猪猪侠的分享！

@猪猪侠 好像没有对被安全狗屏蔽访问的判断,

@cf_hb 用的爬虫吧

@国士无双 不是爬虫，这是在Fuzzing目录

@cf_hb 你没发现config.py可以定制规则的嘛? 你把安全狗报错的关键字写到404规则里面去，这些200的内容在结果集中就会自动消除了。

@猪猪侠 嗯，看到page_not_found_reg了

弱弱的问句，扫完之后输出文件在哪了..

root@kali:~/weakfilescan# python wyspider.py http://www.xxxx.cn php
--------------------------------------------------
* scan http://www.qeeyou.cn start
--------------------------------------------------
Traceback (most recent call last):
  File "wyspider.py", line 17, in <module>
    print json.dumps(start_wyspider(sys.argv[1]), indent=2)
  File "/root/weakfilescan/controller.py", line 41, in start_wyspider
    link_datas = GetAllLink(siteurl).start()
  File "/root/weakfilescan/libs/GetAllLink.py", line 47, in start
    response_obj = LinksParser(http_request_get(self.siteurl))
  File "/root/weakfilescan/common.py", line 171, in __init__
    self.soup = BeautifulSoup(self.html_content.text, 'lxml')
  File "/usr/lib/python2.7/dist-packages/bs4/__init__.py", line 172, in __init__
    self._feed()
  File "/usr/lib/python2.7/dist-packages/bs4/__init__.py", line 185, in _feed
    self.builder.feed(self.markup)
  File "/usr/lib/python2.7/dist-packages/bs4/builder/_lxml.py", line 175, in feed
    self.parser.close()
  File "parser.pxi", line 1186, in lxml.etree._FeedParser.close (src/lxml/lxml.etree.c:88907)
  File "parsertarget.pxi", line 142, in lxml.etree._TargetParserContext._handleParseResult (src/lxml/lxml.etree.c:98223)
  File "parsertarget.pxi", line 130, in lxml.etree._TargetParserContext._handleParseResult (src/lxml/lxml.etree.c:98047)
  File "lxml.etree.pyx", line 294, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:9077)
  File "saxparser.pxi", line 289, in lxml.etree._handleSaxDoctype (src/lxml/lxml.etree.c:94611)
  File "parsertarget.pxi", line 86, in lxml.etree._PythonSaxParserTarget._handleSaxDoctype (src/lxml/lxml.etree.c:97582)
  File "/usr/lib/python2.7/dist-packages/bs4/builder/_lxml.py", line 150, in doctype
    doctype = Doctype.for_name_and_ids(name, pubid, system)
  File "/usr/lib/python2.7/dist-packages/bs4/element.py", line 720, in for_name_and_ids
    return Doctype(value)
  File "/usr/lib/python2.7/dist-packages/bs4/element.py", line 653, in __new__
    return unicode.__new__(cls, value, DEFAULT_OUTPUT_ENCODING)
TypeError: coercing to Unicode: need string or buffer, NoneType found

弱弱的问句，扫完之后输出文件在哪了..

@xssx wyspider.py 第 17、21 行

加了点代码
扫描完成后会写入结果在当前目录，生成文件result.txt(代码很搓，有点冗余)
加了个用法：python wyspider.py -u url.txt   (从当前目录url.txt循环取url然后开始扫描)

if __name__ == "__main__":
  f = file("result.txt","a+")
  result = "test"
  if len(sys.argv) == 2:    
    result = json.dumps(start_wyspider(sys.argv[1]), indent=2)
    f.write(result)
    print result
    sys.exit(0)
  elif len(sys.argv)==3:
    for i in open(sys.argv[2]):      
      result = json.dumps(start_wyspider(str(i).strip()), indent=2)
      f.write(result)
      print result
    sys.exit(0)
  else:
    print ("usage: %s http://wuyun.org" % sys.argv[0])
    print ("usage: %s -u url.txt" % sys.argv[0])
    sys.exit(-1)

感觉 速度好慢呀~可以加设置线程数不

感谢

@Superman config.py 里面不是有配置项吗？自定义线程数，慢是因为实实在在的发出了足够数量的扫描测试用例任务。。

某些url会报这个错误
Traceback (most recent call last):
  File "wyspider.py", line 26, in <module>
    result = json.dumps(start_wyspider(str(i).strip()), indent=2)
  File "F:\Self\weakfilescan-master\controller.py", line 41, in start_wyspider
    link_datas = GetAllLink(siteurl).start()
  File "F:\Self\weakfilescan-master\libs\GetAllLink.py", line 47, in start
    response_obj = LinksParser(http_request_get(self.siteurl))
  File "F:\Self\weakfilescan-master\common.py", line 170, in __init__
    self.baseurl = get_baseurl(self.url)
  File "F:\Self\weakfilescan-master\common.py", line 31, in get_baseurl
    netloc = urlparse.urlparse(link).netloc
  File "C:\Python27\lib\urlparse.py", line 134, in urlparse
    tuple = urlsplit(url, scheme, allow_fragments)
  File "C:\Python27\lib\urlparse.py", line 173, in urlsplit
    i = url.find(':')
AttributeError: 'NoneType' object has no attribute 'find'

然后http://www.onlylady.com报了这个错误
* scan http://www.onlylady.com start
--------------------------------------------------
encoding error : input conversion failed due to input error, bytes 0xB5 0x00 0xDA 0x00
encoding error : input conversion failed due to input error, bytes 0xB5 0x00 0xDA 0x00
encoding error : input conversion failed due to input error, bytes 0xD3 0x00 0xD1 0x00
encoding error : input conversion failed due to input error, bytes 0xD3 0x00 0xD1 0x00
encoding error : input conversion failed due to input error, bytes 0xC5 0x00 0xD4 0x00
encoding error : input conversion failed due to input error, bytes 0xC5 0x00 0xD4 0x00
encoding error : input conversion failed due to input error, bytes 0xD5 0x00 0xE2 0x00
encoding error : input conversion failed due to input error, bytes 0xD5 0x00 0xE2 0x00
encoding error : input conversion failed due to input error, bytes 0xD1 0x00 0xEE 0x00
encoding error : input conversion failed due to input error, bytes 0xD1 0x00 0xEE 0x00
encoding error : input conversion failed due to input error, bytes 0xB4 0x00 0xA6 0x00
encoding error : input conversion failed due to input error, bytes 0xB4 0x00 0xA6 0x00
encoding error : input conversion failed due to input error, bytes 0xB1 0x00 0xEA 0x00
encoding error : input conversion failed due to input error, bytes 0xB1 0x00 0xEA 0x00
encoding error : input conversion failed due to input error, bytes 0xB1 0x00 0xEA 0x00
encoding error : input conversion failed due to input error, bytes 0xB1 0x00 0xEA 0x00
encoding error : input conversion failed due to input error, bytes 0xB7 0x00 0xA2 0x00
encoding error : input conversion failed due to input error, bytes 0xB7 0x00 0xA2 0x00
encoding error : input conversion failed due to input error, bytes 0xC6 0x00 0xD8 0x00
encoding error : input conversion failed due to input error, bytes 0xC6 0x00 0xD8 0x00
encoding error : input conversion failed due to input error, bytes 0xC7 0x00 0xA7 0x00
encoding error : input conversion failed due to input error, bytes 0xBF 0x00 0xC6 0x00
encoding error : input conversion failed due to input error, bytes 0xBB 0x00 0xBC 0x00
encoding error : input conversion failed due to input error, bytes 0xBC 0x00 0xAA 0x00
encoding error : input conversion failed due to input error, bytes 0xC8 0x00 0xC8 0x00
encoding error : input conversion failed due to input error, bytes 0xA1 0x00 0xBE 0x00
encoding error : input conversion failed due to input error, bytes 0xC7 0x00 0xEF 0x00
encoding error : input conversion failed due to input error, bytes 0xC8 0x00 0xC8 0x00
encoding error : input conversion failed due to input error, bytes 0xA1 0x00 0xBE 0x00
encoding error : input conversion failed due to input error, bytes 0xBB 0x00 0xBC 0x00
encoding error : input conversion failed due to input error, bytes 0xC8 0x00 0xC8 0x00
encoding error : input conversion failed due to input error, bytes 0xC7 0x00 0xA7 0x00
encoding error : input conversion failed due to input error, bytes 0xC7 0x00 0xEF 0x00
encoding error : input conversion failed due to input error, bytes 0xBF 0x00 0xC6 0x00
encoding error : input conversion failed due to input error, bytes 0xC8 0x00 0xC8 0x00
encoding error : input conversion failed due to input error, bytes 0xBC 0x00 0xAA 0x00
encoding error : input conversion failed due to input error, bytes 0xD3 0x00 0xA2 0x00
encoding error : input conversion failed due to input error, bytes 0xD3 0x00 0xA2 0x00
encoding error : input conversion failed due to input error, bytes 0xD2 0x00 0xBB 0x00
encoding error : input conversion failed due to input error, bytes 0xBB 0x00 0xD8 0x00
encoding error : input conversion failed due to input error, bytes 0xD2 0x00 0xBB 0x00
encoding error : input conversion failed due to input error, bytes 0xBB 0x00 0xD8 0x00

神一般的存在

猪猪侠，你是原来邪八的ring04h？

猪哥好吊

猪哥好叼 来膜拜下 顺便收走工具

有些url一直是未响应状态。。

@猪猪侠
经过测试，发现如果是index.php.bak ，可以通过发现index.php 然后扫描备份文件会发现 。但是xxxxx/config.php.bak 就不能通过 xxxxx/config.php 发现 。 这个是为什么  ？

@猪猪侠  还有一个问题 ，我再filename.lst 字典中 添加xxx.rar  运行中确定他是 200的状态了，但是在最后的json结果中 ，没有这项数据。

默默的走过、、、、

Traceback (most recent call last):
  File "wyspider.py", line 20, in <module>
    print json.dumps(start_wyspider(sys.argv[1]),indent=2)
  File "/home/wwwroot/scan/weakfilescan-master/controller.py", line 58, in start_wyspider
    fuzzfile_request_set.extend(urlgenerator_obj.generator())
  File "/home/wwwroot/scan/weakfilescan-master/libs/FuzzUrlGenerator.py", line 46, in generator
    urls_result.append(parser_obj.baseurl + webdir + depend)
MemoryError

感谢猪猪侠分享，运行良好，期待帮助文档哈~

win下已正常运行，谢谢大神分享~

已关注

@灰色轨迹 apt-get阿  你kali新手么。。。。。

神器。。。。卧槽  猪猪侠带我飞阿

谢谢大神分享

@猪猪侠

求解答是何处出了问题@猪猪侠

@猪猪侠 虚拟机中ubuntu 已经成功运行。
只是不懂mac 如何运行

@Skull lxml 没装好。

@猪猪侠 谢谢！

感谢猪猪侠，不过遇到200的自定义错误页怎么破？

找到设置的地方了，page_not_found_reg = r'404|[nN]ot [fF]ound|不存在|未找到|Error'，不过自定义页面内已经有“404”的内容了但为什么还是被扫到了？

顶了。。

@猪猪侠 请问下 扫描完输出的目录是什么？ 文件名叫什么？

感谢

mac下 没有任何错误 内存被吃光。。已卡死，求原因？@猪猪侠