Compare commits
2 Commits
e406d73334
...
658f87b745
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
658f87b745 | ||
|
|
d37c444929 |
10
.gitignore
vendored
10
.gitignore
vendored
@@ -34,6 +34,16 @@ uploads/
|
||||
exports/
|
||||
tmp/
|
||||
*.sqlite3
|
||||
*.pkl
|
||||
domains.txt
|
||||
credentials.json
|
||||
|
||||
# domainCheck local runtime / bundled tools
|
||||
domainCheck/tools/node-v20.19.4-win-x64/
|
||||
domainCheck/app/credentials.json
|
||||
domainCheck/credentials.json
|
||||
domainCheck/**/*.pkl
|
||||
domainCheck/domains.txt
|
||||
|
||||
# OS / editor
|
||||
.DS_Store
|
||||
|
||||
Submodule domainCheck deleted from 83ab7a79e8
12
domainCheck/.env.example
Normal file
12
domainCheck/.env.example
Normal file
@@ -0,0 +1,12 @@
|
||||
# 数据库配置
|
||||
DB_HOST=localhost
|
||||
DB_PORT=5432
|
||||
DB_DATABASE=domain_scan_db
|
||||
DB_USER=postgres
|
||||
DB_PASSWORD=postgres
|
||||
|
||||
# Redis配置
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_PASSWORD=
|
||||
REDIS_DB=0
|
||||
10
domainCheck/.gitignore
vendored
Normal file
10
domainCheck/.gitignore
vendored
Normal file
@@ -0,0 +1,10 @@
|
||||
# 忽略整个目录
|
||||
/node_modules/
|
||||
/build/
|
||||
/dist/
|
||||
|
||||
# 忽略所有压缩文件
|
||||
*.spec
|
||||
.env
|
||||
.venv
|
||||
.idea
|
||||
221
domainCheck/DEPLOYMENT_GUIDE.md
Normal file
221
domainCheck/DEPLOYMENT_GUIDE.md
Normal file
@@ -0,0 +1,221 @@
|
||||
# 域名工具项目部署与使用文档
|
||||
|
||||
## 1. 项目概述
|
||||
|
||||
域名工具是一个用于域名筛选、检测和管理的综合性工具,主要功能包括:
|
||||
- 域名批量导入
|
||||
- 域名筛选与查询
|
||||
- 域名状态检测(注册状态、使用状态、检测状态等)
|
||||
- 域名详细信息检测(备案历史、快照年份、百度收录等)
|
||||
- 域名数据导出(Excel、CSV)
|
||||
- 批量更新域名状态
|
||||
|
||||
## 2. 环境要求
|
||||
|
||||
### 2.1 系统要求
|
||||
- Windows 10/11 64位操作系统
|
||||
- 至少4GB内存
|
||||
- 至少500MB磁盘空间
|
||||
|
||||
### 2.2 软件依赖
|
||||
- Python 3.9+
|
||||
- PostgreSQL 12+
|
||||
- Redis(用于缓存)+布隆过滤器(用于域名去重)
|
||||
|
||||
## 3. 安装步骤
|
||||
|
||||
### 3.1 克隆项目
|
||||
```bash
|
||||
git clone <项目仓库地址>
|
||||
cd domainScanDemo
|
||||
```
|
||||
|
||||
### 3.2 创建虚拟环境
|
||||
```bash
|
||||
python -m venv .venv
|
||||
```
|
||||
|
||||
### 3.3 激活虚拟环境
|
||||
```bash
|
||||
# Windows
|
||||
.venv\Scripts\activate
|
||||
|
||||
# Linux/Mac
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### 3.4 安装依赖
|
||||
```bash
|
||||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
```
|
||||
|
||||
### 3.5 安装额外依赖(用于Excel导出)
|
||||
```bash
|
||||
pip install openpyxl
|
||||
```
|
||||
|
||||
## 4. 配置说明
|
||||
|
||||
### 4.1 环境变量配置
|
||||
|
||||
复制 `.env.example` 文件为 `.env`,并根据实际情况修改配置:
|
||||
|
||||
```env
|
||||
# 数据库连接信息
|
||||
DATABASE_URL=postgresql://username:password@localhost:5432/domain_db
|
||||
|
||||
# Redis连接信息(可选)
|
||||
REDIS_URL=redis://localhost:6379/0
|
||||
|
||||
# 日志配置
|
||||
LOG_LEVEL=INFO
|
||||
LOG_FILE=logs/app.log
|
||||
|
||||
# 线程配置
|
||||
MAX_WORKERS=10
|
||||
|
||||
# 检测配置
|
||||
DETECT_TIMEOUT=30
|
||||
```
|
||||
|
||||
### 4.2 其他配置文件
|
||||
|
||||
- `detect_options.json` - 检测选项配置
|
||||
- `proxy_config.json` - 代理配置
|
||||
- `domain_suffixes.json` - 域名后缀配置
|
||||
- `thread_count.json` - 线程数量配置
|
||||
|
||||
## 5. 数据库初始化
|
||||
|
||||
### 5.1 创建数据库
|
||||
|
||||
在PostgreSQL中创建数据库:
|
||||
```sql
|
||||
CREATE DATABASE domain_db;
|
||||
```
|
||||
|
||||
### 5.2 初始化表结构
|
||||
|
||||
运行数据库初始化脚本:
|
||||
```bash
|
||||
python init_database.py
|
||||
```
|
||||
|
||||
此脚本会创建以下表:
|
||||
- `domains` - 域名基本信息表
|
||||
- `domain_detections` - 域名检测详情表
|
||||
- `sensitive_words` - 敏感词表
|
||||
|
||||
### 5.3 添加敏感词(可选)
|
||||
|
||||
```bash
|
||||
python add_sensitive_words_table.py
|
||||
```
|
||||
|
||||
## 6. 运行项目
|
||||
|
||||
### 6.1 启动主应用
|
||||
|
||||
```bash
|
||||
python app/main.py
|
||||
```
|
||||
|
||||
### 6.2 启动检测端(可选,用于分布式检测)
|
||||
|
||||
```bash
|
||||
python detect_worker.py
|
||||
```
|
||||
|
||||
## 7. 打包指南
|
||||
|
||||
### 7.1 打包主应用
|
||||
|
||||
```bash
|
||||
pyinstaller --onefile --windowed --add-data="app/sdk_leg.js;detect" --add-data="app/sdk_leg_env.js;detect" --add-data="app/chinaz.js;detect" --add-data="detect/module/crack_geetest2x.js;detect/module" --add-data="favicon.ico;." --add-data="new_logo.svg;." --add-data=".env;." --icon="favicon.ico" --name="域名工具" app/main.py
|
||||
```
|
||||
|
||||
### 7.2 打包检测端
|
||||
|
||||
```bash
|
||||
pyinstaller --onefile --windowed --add-data="detect/sdk_leg.js;detect" --add-data="detect/sdk_leg_env.js;detect" --add-data="detect/chinaz.js;detect" --add-data="detect/module/crack_geetest2x.js;detect/module" --add-data="favicon2.ico;." --add-data="new_logo.svg;." --add-data=".env;." --icon="favicon2.ico" --name="域名检测端" detect_worker.py
|
||||
```
|
||||
|
||||
### 7.3 打包结果
|
||||
|
||||
打包完成后,可执行文件会生成在 `dist` 目录中:
|
||||
- `域名工具.exe` - 主应用
|
||||
- `域名检测端.exe` - 检测端
|
||||
|
||||
|
||||
## 8. 使用说明
|
||||
|
||||
### 8.1 域名导入
|
||||
|
||||
1. 点击"域名导入"标签页
|
||||
2. 选择导入方式(文件导入或聚名网爬取)
|
||||
3. 选择文件或输入聚名网URL
|
||||
4. 点击"开始导入"按钮
|
||||
5. 等待导入完成
|
||||
|
||||
### 8.2 域名筛选
|
||||
|
||||
1. 点击"域名筛选"标签页
|
||||
2. 设置筛选条件(注册状态、使用状态、检测状态等)
|
||||
3. 点击"查询"按钮
|
||||
4. 查看查询结果
|
||||
5. 可选择域名进行批量更新或导出
|
||||
|
||||
### 8.3 域名检测
|
||||
|
||||
1. 确保检测端已启动
|
||||
2. 在主应用中选择需要检测的域名
|
||||
3. 点击"开始检测"按钮
|
||||
4. 等待检测完成
|
||||
5. 查看检测结果
|
||||
|
||||
### 8.4 数据导出
|
||||
|
||||
1. 在域名筛选界面选择需要导出的域名
|
||||
2. 点击"导出"按钮
|
||||
3. 选择导出格式(Excel或CSV)
|
||||
4. 选择保存路径
|
||||
5. 等待导出完成
|
||||
|
||||
## 9. 常见问题
|
||||
|
||||
### 9.1 数据库连接失败
|
||||
|
||||
- 检查 `.env` 文件中的数据库连接信息是否正确
|
||||
- 确保 PostgreSQL 服务已启动
|
||||
- 确保数据库用户有足够的权限
|
||||
|
||||
### 9.2 Redis 连接失败
|
||||
|
||||
- 检查 `.env` 文件中的 Redis 连接信息是否正确
|
||||
- 确保 Redis 服务已启动
|
||||
- 如果不需要 Redis 缓存,可以忽略此警告
|
||||
|
||||
### 9.3 检测失败
|
||||
|
||||
- 检查网络连接是否正常
|
||||
- 检查代理配置是否正确
|
||||
- 检查检测端是否已启动
|
||||
|
||||
### 9.4 导出失败
|
||||
|
||||
- 确保已安装 `openpyxl` 库(用于Excel导出)
|
||||
- 确保导出路径有写入权限
|
||||
- 检查导出文件格式是否正确
|
||||
|
||||
## 10. 技术支持
|
||||
|
||||
如果遇到问题,请查看日志文件 `logs/app.log` 获取详细错误信息,或联系技术支持。
|
||||
|
||||
## 11. 版本信息
|
||||
|
||||
- 项目版本:1.0.0
|
||||
- 最后更新:2026-04-14
|
||||
|
||||
---
|
||||
|
||||
**注意**:本文档仅供参考,具体配置和使用方法可能因环境不同而有所差异。
|
||||
192
domainCheck/README_acw_sc__v2.md
Normal file
192
domainCheck/README_acw_sc__v2.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# acw_sc__v2 生成器
|
||||
|
||||
从混淆的HTML代码中提取的 `acw_sc__v2` cookie 生成函数。
|
||||
|
||||
## 文件说明
|
||||
|
||||
### JavaScript版本
|
||||
- **acw_sc__v2_generator.js** - JavaScript版本的生成函数,可直接在浏览器控制台中使用
|
||||
|
||||
### Python版本
|
||||
- **acw_sc__v2_fixed.py** - 简化版Python生成器,适合快速使用
|
||||
- **acw_sc__v2_complete.py** - 完整版Python生成器,包含更多功能和错误处理
|
||||
- **acw_sc__v2_simple.py** - 最简化版本,直接使用硬编码密钥
|
||||
|
||||
## 使用方法
|
||||
|
||||
### JavaScript版本
|
||||
|
||||
在浏览器控制台中执行:
|
||||
|
||||
```javascript
|
||||
// 加载acw_sc__v2_generator.js文件后
|
||||
var arg1 = '1EEB3321F0286855D50CD9E96C1CDF3D4A913DB0';
|
||||
var acwScV2 = generateAcwScV2(arg1);
|
||||
console.log('acw_sc__v2的值:', acwScV2);
|
||||
|
||||
// 设置cookie(可选)
|
||||
setAcwScV2Cookie(acwScV2);
|
||||
```
|
||||
|
||||
### Python版本
|
||||
|
||||
#### 简化版(acw_sc__v2_fixed.py)
|
||||
|
||||
```python
|
||||
from acw_sc__v2_fixed import generate_acw_sc_v2
|
||||
|
||||
arg1 = '1EEB3321F0286855D50CD9E96C1CDF3D4A913DB0'
|
||||
acw_sc_v2 = generate_acw_sc_v2(arg1)
|
||||
print(f'acw_sc__v2的值: {acw_sc_v2}')
|
||||
```
|
||||
|
||||
#### 完整版(acw_sc__v2_complete.py)
|
||||
|
||||
```python
|
||||
from acw_sc__v2_complete import AcwScV2Generator, generate_acw_sc_v2
|
||||
|
||||
# 方法1:使用便捷函数
|
||||
arg1 = '1EEB3321F0286855D50CD9E96C1CDF3D4A913DB0'
|
||||
result = generate_acw_sc_v2(arg1)
|
||||
print(f'acw_sc__v2的值: {result}')
|
||||
|
||||
# 方法2:使用生成器类
|
||||
generator = AcwScV2Generator()
|
||||
result = generator.generate(arg1)
|
||||
print(f'acw_sc__v2的值: {result}')
|
||||
|
||||
# 方法3:批量生成
|
||||
test_args = [
|
||||
'1EEB3321F0286855D50CD9E96C1CDF3D4A913DB0',
|
||||
'A13D8B4174A10C57667C1CD92E54D01F449CA76E',
|
||||
'F0286855D50CD9E96C1CDF3D4A913DB01EEB3321'
|
||||
]
|
||||
batch_results = generator.generate_batch(test_args)
|
||||
for arg, res in batch_results.items():
|
||||
print(f'{arg} -> {res}')
|
||||
|
||||
# 方法4:使用自定义密钥
|
||||
custom_key = 'YXBwbHk=' # Base64编码的密钥
|
||||
generator.set_key(custom_key)
|
||||
result = generator.generate(arg1)
|
||||
print(f'使用自定义密钥的acw_sc__v2值: {result}')
|
||||
```
|
||||
|
||||
## 核心算法
|
||||
|
||||
### 1. 字符重排
|
||||
|
||||
根据索引映射数组 `m` 重新排列输入字符串 `arg1`:
|
||||
|
||||
```python
|
||||
m = [0xf, 0x23, 0x1d, 0x18, 0x21, 0x10, 0x1, 0x26, 0xa, 0x9, 0x13, 0x1f, 0x28, 0x1b, 0x16, 0x17, 0x19, 0xd, 0x6, 0xb, 0x27, 0x12, 0x14, 0x8, 0xe, 0x15, 0x20, 0x1a, 0x2, 0x1e, 0x7, 0x4, 0x11, 0x5, 0x3, 0x1c, 0x22, 0x25, 0xc, 0x24]
|
||||
```
|
||||
|
||||
### 2. 密钥解码
|
||||
|
||||
将Base64编码的密钥解码为十六进制字符串:
|
||||
|
||||
```python
|
||||
key_encoded = 'Aw5PDg9JAw4='
|
||||
key_hex = base64.b64decode(key_encoded).hex()
|
||||
# 结果: '030e4f0e0f49030e'
|
||||
```
|
||||
|
||||
### 3. 异或运算
|
||||
|
||||
将重排后的字符串与解码后的密钥进行按位异或:
|
||||
|
||||
```python
|
||||
for x in range(0, min(len(u), len(p)), 2):
|
||||
u_sub = u[x:x+2]
|
||||
p_sub = p[x:x+2]
|
||||
a = format(int(u_sub, 16) ^ int(p_sub, 16), '02x')
|
||||
v += a
|
||||
```
|
||||
|
||||
## 输入输出示例
|
||||
|
||||
### 输入
|
||||
```python
|
||||
arg1 = '1EEB3321F0286855D50CD9E96C1CDF3D4A913DB0'
|
||||
```
|
||||
|
||||
### 输出
|
||||
```python
|
||||
acw_sc__v2 = '5ad70a13004a0290'
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **输入长度**:`arg1` 必须是40位的十六进制字符串
|
||||
2. **输入格式**:`arg1` 必须是有效的十六进制字符串
|
||||
3. **密钥**:默认使用内置密钥,也可以自定义密钥
|
||||
4. **输出格式**:输出为十六进制字符串
|
||||
|
||||
## 错误处理
|
||||
|
||||
完整版生成器包含以下错误处理:
|
||||
|
||||
- 输入长度验证
|
||||
- 输入格式验证
|
||||
- Base64解码错误处理
|
||||
- 异常捕获和错误信息输出
|
||||
|
||||
## 应用场景
|
||||
|
||||
- 网站爬虫:绕过阿里云CDN的反爬虫保护
|
||||
- 安全测试:测试网站的反爬虫机制
|
||||
- 学习研究:了解混淆JavaScript代码的逆向工程
|
||||
|
||||
## 技术细节
|
||||
|
||||
### 混淆技术
|
||||
|
||||
原始HTML代码使用了多种混淆技术:
|
||||
|
||||
1. **字符串编码**:Base64编码 + URL编码
|
||||
2. **变量名混淆**:使用无意义的变量名
|
||||
3. **控制流混淆**:使用闭包和条件函数
|
||||
4. **反调试**:通过数学计算和正则表达式检测调试器
|
||||
|
||||
### 逆向工程
|
||||
|
||||
通过分析原始代码,提取出核心算法:
|
||||
|
||||
1. 移除反调试逻辑
|
||||
2. 简化字符串解码
|
||||
3. 提取核心计算逻辑
|
||||
4. 重构为可读代码
|
||||
|
||||
## 性能优化
|
||||
|
||||
- 使用缓存机制避免重复解码
|
||||
- 批量生成功能提高效率
|
||||
- 简化算法减少计算量
|
||||
|
||||
## 安全提示
|
||||
|
||||
本工具仅供学习和研究使用,请勿用于非法用途。使用本工具绕过网站的反爬虫保护可能违反网站的使用条款。
|
||||
|
||||
## 许可证
|
||||
|
||||
本代码仅供学习和研究使用。
|
||||
|
||||
## 联系方式
|
||||
|
||||
如有问题或建议,请通过以下方式联系:
|
||||
|
||||
- 提交Issue
|
||||
- 发送Pull Request
|
||||
- 联系作者
|
||||
|
||||
## 更新日志
|
||||
|
||||
### v1.0.0 (2024-01-01)
|
||||
- 初始版本发布
|
||||
- 支持JavaScript和Python版本
|
||||
- 包含完整的使用示例
|
||||
|
||||
## 致谢
|
||||
|
||||
感谢所有为逆向工程和反爬虫技术研究做出贡献的开发者。
|
||||
76
domainCheck/add_backlink_count_column.py
Normal file
76
domainCheck/add_backlink_count_column.py
Normal file
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :add_backlink_count_column.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/10 03:50
|
||||
@explain : 为domains表添加backlink_count字段
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from app.config import config
|
||||
|
||||
|
||||
def add_backlink_count_column():
|
||||
"""
|
||||
为domains表添加backlink_count字段
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
print(f"成功连接到数据库: {config.DB_HOST}:{config.DB_PORT}/{config.DB_DATABASE}")
|
||||
|
||||
# 创建游标
|
||||
cur = conn.cursor()
|
||||
|
||||
# 添加backlink_count字段
|
||||
add_column_sql = """
|
||||
ALTER TABLE domains ADD COLUMN IF NOT EXISTS backlink_count INTEGER DEFAULT 0
|
||||
"""
|
||||
cur.execute(add_column_sql)
|
||||
print("添加backlink_count字段成功")
|
||||
|
||||
# 添加字段注释
|
||||
add_comment_sql = """
|
||||
COMMENT ON COLUMN domains.backlink_count IS '友情链接数量'
|
||||
"""
|
||||
cur.execute(add_comment_sql)
|
||||
print("添加字段注释成功")
|
||||
|
||||
# 创建索引
|
||||
create_index_sql = """
|
||||
CREATE INDEX IF NOT EXISTS idx_domains_backlink_count ON domains(backlink_count)
|
||||
"""
|
||||
cur.execute(create_index_sql)
|
||||
print("创建索引成功")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
print("事务提交成功")
|
||||
|
||||
# 关闭游标和连接
|
||||
cur.close()
|
||||
conn.close()
|
||||
print("连接关闭成功")
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"添加backlink_count字段失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("开始为domains表添加backlink_count字段...")
|
||||
success = add_backlink_count_column()
|
||||
if success:
|
||||
print("backlink_count字段添加成功!")
|
||||
else:
|
||||
print("backlink_count字段添加失败!")
|
||||
8
domainCheck/add_backlink_count_column.sql
Normal file
8
domainCheck/add_backlink_count_column.sql
Normal file
@@ -0,0 +1,8 @@
|
||||
-- 为domains表添加backlink_count字段
|
||||
ALTER TABLE domains ADD COLUMN IF NOT EXISTS backlink_count INTEGER DEFAULT 0;
|
||||
|
||||
-- 添加字段注释
|
||||
COMMENT ON COLUMN domains.backlink_count IS '友情链接数量';
|
||||
|
||||
-- 创建索引
|
||||
CREATE INDEX IF NOT EXISTS idx_domains_backlink_count ON domains(backlink_count);
|
||||
51
domainCheck/add_expire_date_field.py
Normal file
51
domainCheck/add_expire_date_field.py
Normal file
@@ -0,0 +1,51 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :add_expire_date_field.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/11 22:40
|
||||
@explain : 为domains表添加expire_date字段
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from loguru import logger
|
||||
from app.config import config
|
||||
|
||||
def add_expire_date_field():
|
||||
"""
|
||||
为domains表添加expire_date字段
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
logger.info("数据库连接成功")
|
||||
|
||||
# 添加expire_date字段
|
||||
try:
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS expire_date DATE")
|
||||
cur.execute("COMMENT ON COLUMN domains.expire_date IS '过期时间'")
|
||||
logger.info("添加expire_date字段成功")
|
||||
except Exception as e:
|
||||
logger.error(f"添加expire_date字段失败: {e}")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
logger.info("字段添加完成")
|
||||
|
||||
# 关闭连接
|
||||
cur.close()
|
||||
conn.close()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"执行失败: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
add_expire_date_field()
|
||||
59
domainCheck/add_fields_to_domains.py
Normal file
59
domainCheck/add_fields_to_domains.py
Normal file
@@ -0,0 +1,59 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :add_fields_to_domains.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/11 22:40
|
||||
@explain : 为domains表添加缺失的字段
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from loguru import logger
|
||||
from app.config import config
|
||||
|
||||
def add_fields_to_domains():
|
||||
"""
|
||||
为domains表添加缺失的字段
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
logger.info("数据库连接成功")
|
||||
|
||||
# 添加company_type字段
|
||||
try:
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS company_type VARCHAR(100)")
|
||||
cur.execute("COMMENT ON COLUMN domains.company_type IS '单位性质'")
|
||||
logger.info("添加company_type字段成功")
|
||||
except Exception as e:
|
||||
logger.error(f"添加company_type字段失败: {e}")
|
||||
|
||||
# 添加website_url字段
|
||||
try:
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS website_url VARCHAR(255)")
|
||||
cur.execute("COMMENT ON COLUMN domains.website_url IS '网站首页网址'")
|
||||
logger.info("添加website_url字段成功")
|
||||
except Exception as e:
|
||||
logger.error(f"添加website_url字段失败: {e}")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
logger.info("字段添加完成")
|
||||
|
||||
# 关闭连接
|
||||
cur.close()
|
||||
conn.close()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"执行失败: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
add_fields_to_domains()
|
||||
58
domainCheck/add_sensitive_words_table.py
Normal file
58
domainCheck/add_sensitive_words_table.py
Normal file
@@ -0,0 +1,58 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :add_sensitive_words_table.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/12 00:03
|
||||
@explain : 为数据库添加sensitive_words表
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from loguru import logger
|
||||
from app.config import config
|
||||
|
||||
def add_sensitive_words_table():
|
||||
"""
|
||||
为数据库添加sensitive_words表
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
logger.info("数据库连接成功")
|
||||
|
||||
# 创建sensitive_words表
|
||||
create_sensitive_words_table = """
|
||||
CREATE TABLE IF NOT EXISTS sensitive_words (
|
||||
id SERIAL PRIMARY KEY,
|
||||
word VARCHAR(255) UNIQUE NOT NULL
|
||||
)
|
||||
"""
|
||||
cur.execute(create_sensitive_words_table)
|
||||
logger.info("创建sensitive_words表成功")
|
||||
|
||||
# 添加字段注释
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.id IS '主键ID'")
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.word IS '敏感词'")
|
||||
logger.info("添加sensitive_words表字段注释成功")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
logger.info("sensitive_words表添加完成")
|
||||
|
||||
# 关闭连接
|
||||
cur.close()
|
||||
conn.close()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"执行失败: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
add_sensitive_words_table()
|
||||
6
domainCheck/app/__init__.py
Normal file
6
domainCheck/app/__init__.py
Normal file
@@ -0,0 +1,6 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
域名库系统
|
||||
'''
|
||||
|
||||
__version__ = "1.0.0"
|
||||
1257
domainCheck/app/chinaz.js
Normal file
1257
domainCheck/app/chinaz.js
Normal file
File diff suppressed because one or more lines are too long
265
domainCheck/app/config.py
Normal file
265
domainCheck/app/config.py
Normal file
@@ -0,0 +1,265 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :config.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:08
|
||||
@explain : 系统配置
|
||||
'''
|
||||
|
||||
import os
|
||||
import sys
|
||||
from dotenv import load_dotenv
|
||||
|
||||
|
||||
def _safe_echo(message):
|
||||
"""
|
||||
在不同 Windows 控制台编码下安全输出文本,避免导入阶段因中文打印失败。
|
||||
"""
|
||||
try:
|
||||
print(message)
|
||||
except UnicodeEncodeError:
|
||||
try:
|
||||
encoding = sys.stdout.encoding or 'utf-8'
|
||||
sys.stdout.buffer.write((message + '\n').encode(encoding, errors='replace'))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 确定基础目录
|
||||
if hasattr(sys, '_MEIPASS'):
|
||||
# PyInstaller 打包后的临时目录
|
||||
BASE_DIR = sys._MEIPASS
|
||||
else:
|
||||
# 开发环境目录
|
||||
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# 加载环境变量
|
||||
env_path = os.path.join(BASE_DIR, '.env')
|
||||
if os.path.exists(env_path):
|
||||
load_dotenv(env_path)
|
||||
_safe_echo(f"成功加载环境变量文件: {env_path}")
|
||||
else:
|
||||
load_dotenv()
|
||||
_safe_echo(f"环境变量文件不存在: {env_path},使用默认环境变量")
|
||||
|
||||
|
||||
class Config:
|
||||
"""
|
||||
系统配置
|
||||
"""
|
||||
|
||||
# 数据库配置
|
||||
DB_HOST = os.getenv('DB_HOST', 'localhost')
|
||||
DB_PORT = int(os.getenv('DB_PORT', 5432))
|
||||
DB_DATABASE = os.getenv('DB_DATABASE', 'domain_scan_db')
|
||||
DB_USER = os.getenv('DB_USER', 'postgres')
|
||||
DB_PASSWORD = os.getenv('DB_PASSWORD', 'postgres')
|
||||
DB_POOL_SIZE = int(os.getenv('DB_POOL_SIZE', 5))
|
||||
|
||||
# 消息队列配置
|
||||
RABBITMQ_HOST = os.getenv('RABBITMQ_HOST', 'localhost')
|
||||
RABBITMQ_PORT = int(os.getenv('RABBITMQ_PORT', 5672))
|
||||
RABBITMQ_USER = os.getenv('RABBITMQ_USER', 'guest')
|
||||
RABBITMQ_PASSWORD = os.getenv('RABBITMQ_PASSWORD', 'guest')
|
||||
RABBITMQ_VHOST = os.getenv('RABBITMQ_VHOST', '/')
|
||||
|
||||
# Redis配置
|
||||
REDIS_HOST = os.getenv('REDIS_HOST', 'localhost')
|
||||
REDIS_PORT = int(os.getenv('REDIS_PORT', 6379))
|
||||
REDIS_PASSWORD = os.getenv('REDIS_PASSWORD', '')
|
||||
REDIS_DB = int(os.getenv('REDIS_DB', 0))
|
||||
|
||||
# 聚名网配置
|
||||
JUMING_COOKIE = os.getenv('JUMING_COOKIE', '')
|
||||
JUMING_REFERER = os.getenv('JUMING_REFERER', 'https://www.juming.com/')
|
||||
JUMING_USER_AGENT = os.getenv('JUMING_USER_AGENT', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36')
|
||||
|
||||
# 代理配置
|
||||
PROXY_ENABLED = os.getenv('PROXY_ENABLED', 'false').lower() == 'true'
|
||||
PROXY_URL = os.getenv('PROXY_URL', '')
|
||||
|
||||
# 检测配置
|
||||
DETECT_TIMEOUT = int(os.getenv('DETECT_TIMEOUT', 30))
|
||||
DETECT_RETRY_COUNT = int(os.getenv('DETECT_RETRY_COUNT', 3))
|
||||
DETECT_CONCURRENCY = int(os.getenv('DETECT_CONCURRENCY', 10))
|
||||
WAYBACK_CDX_TIMEOUT = int(os.getenv('WAYBACK_CDX_TIMEOUT', 15))
|
||||
WAYBACK_SNAPSHOT_TIMEOUT = int(os.getenv('WAYBACK_SNAPSHOT_TIMEOUT', 12))
|
||||
WAYBACK_RETRY_COUNT = int(os.getenv('WAYBACK_RETRY_COUNT', 2))
|
||||
WAYBACK_REQUEST_DELAY = float(os.getenv('WAYBACK_REQUEST_DELAY', 0))
|
||||
WAYBACK_PROGRESS_INTERVAL = int(os.getenv('WAYBACK_PROGRESS_INTERVAL', 500))
|
||||
WAYBACK_DOMAIN_CONCURRENCY = int(os.getenv('WAYBACK_DOMAIN_CONCURRENCY', 3))
|
||||
WAYBACK_TITLE_MAX_BYTES = int(os.getenv('WAYBACK_TITLE_MAX_BYTES', 65536))
|
||||
WAYBACK_TIMESTAMP_CACHE_TTL = int(os.getenv('WAYBACK_TIMESTAMP_CACHE_TTL', 86400))
|
||||
WAYBACK_TITLE_CACHE_TTL = int(os.getenv('WAYBACK_TITLE_CACHE_TTL', 2592000))
|
||||
WAYBACK_USER_AGENT = os.getenv(
|
||||
'WAYBACK_USER_AGENT',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
)
|
||||
|
||||
# 域名配置
|
||||
DOMAIN_TLDS = ['com', 'net']
|
||||
DOMAIN_BATCH_SIZE = int(os.getenv('DOMAIN_BATCH_SIZE', 1000))
|
||||
|
||||
# 日志配置
|
||||
LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')
|
||||
LOG_FILE = os.getenv('LOG_FILE', 'app.log')
|
||||
|
||||
# 任务配置
|
||||
TASK_PRIORITY = {
|
||||
'rdap': 10,
|
||||
'wayback': 8,
|
||||
'baidu': 6,
|
||||
'qihu360': 5,
|
||||
'google': 5,
|
||||
'chinaz': 4,
|
||||
'aizhan': 4,
|
||||
'juziseo': 3,
|
||||
'jucha': 3
|
||||
}
|
||||
|
||||
# 敏感词配置 - 现在从数据库加载
|
||||
SENSITIVE_WORDS = []
|
||||
|
||||
# 检测项配置
|
||||
DETECT_ITEMS = {
|
||||
'rdap': True,
|
||||
'wayback': True,
|
||||
'baidu': True,
|
||||
'qihu360': True,
|
||||
'google': True,
|
||||
'chinaz': True,
|
||||
'aizhan': True,
|
||||
'juziseo': True,
|
||||
'jucha': True
|
||||
}
|
||||
|
||||
# 目录配置
|
||||
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
DATA_DIR = os.path.join(BASE_DIR, 'data')
|
||||
LOG_DIR = os.path.join(BASE_DIR, 'logs')
|
||||
|
||||
# 确保目录存在
|
||||
os.makedirs(DATA_DIR, exist_ok=True)
|
||||
os.makedirs(LOG_DIR, exist_ok=True)
|
||||
|
||||
|
||||
# 从数据库加载敏感词
|
||||
def load_sensitive_words():
|
||||
"""
|
||||
从数据库加载敏感词
|
||||
"""
|
||||
try:
|
||||
from app.utils.database import Database
|
||||
db = Database()
|
||||
sensitive_words = db.get_sensitive_words()
|
||||
words = [word['word'] for word in sensitive_words]
|
||||
db.close()
|
||||
return words
|
||||
except Exception as e:
|
||||
_safe_echo(f"加载敏感词失败: {e}")
|
||||
return []
|
||||
|
||||
# 从文件加载检测选项
|
||||
def load_detect_options():
|
||||
"""
|
||||
从文件加载检测选项
|
||||
"""
|
||||
try:
|
||||
default_order = [
|
||||
'detect_register',
|
||||
'detect_baidu_site',
|
||||
'detect_360_site',
|
||||
'detect_chinaz',
|
||||
'detect_aizhan',
|
||||
'detect_wayback',
|
||||
'detect_jucha',
|
||||
'detect_juziseo',
|
||||
]
|
||||
import json
|
||||
import os
|
||||
defaults = {
|
||||
'detect_register': True,
|
||||
'detect_wayback': True,
|
||||
'detect_chinaz': True,
|
||||
'detect_aizhan': True,
|
||||
'detect_baidu_site': True,
|
||||
'detect_360_site': True,
|
||||
'detect_jucha': False,
|
||||
'detect_juziseo': False,
|
||||
'detect_order': default_order,
|
||||
}
|
||||
if os.path.exists('detect_options.json'):
|
||||
with open('detect_options.json', 'r', encoding='utf-8') as f:
|
||||
detect_options = json.load(f)
|
||||
defaults.update(detect_options)
|
||||
if defaults.get('detect_whois') or defaults.get('detect_beian') or defaults.get('detect_intercept'):
|
||||
defaults['detect_jucha'] = True
|
||||
if defaults.get('detect_juziseo_outlink'):
|
||||
defaults['detect_juziseo'] = True
|
||||
order = defaults.get('detect_order') or []
|
||||
normalized_order = [key for key in order if key in default_order]
|
||||
for key in default_order:
|
||||
if key not in normalized_order:
|
||||
normalized_order.append(key)
|
||||
defaults['detect_order'] = normalized_order
|
||||
return defaults
|
||||
except Exception as e:
|
||||
_safe_echo(f"加载检测选项失败: {e}")
|
||||
return {
|
||||
'detect_register': True,
|
||||
'detect_wayback': True,
|
||||
'detect_chinaz': True,
|
||||
'detect_aizhan': True,
|
||||
'detect_baidu_site': True,
|
||||
'detect_360_site': True,
|
||||
'detect_jucha': False,
|
||||
'detect_juziseo': False,
|
||||
}
|
||||
|
||||
# 导出配置
|
||||
config = Config()
|
||||
# 加载检测选项
|
||||
config.DETECT_OPTIONS = load_detect_options()
|
||||
|
||||
# 延迟加载敏感词,避免循环导入
|
||||
def load_sensitive_words_lazy():
|
||||
"""
|
||||
延迟加载敏感词
|
||||
"""
|
||||
return load_sensitive_words()
|
||||
|
||||
# 设置敏感词属性为延迟加载
|
||||
config.load_sensitive_words = load_sensitive_words_lazy
|
||||
|
||||
# 检查检测类型是否应该执行
|
||||
def should_detect(detect_type):
|
||||
"""
|
||||
检查检测类型是否应该执行
|
||||
|
||||
:param detect_type: 检测类型
|
||||
:return: bool - 是否应该执行
|
||||
"""
|
||||
# 检测类型映射
|
||||
type_mapping = {
|
||||
'register': 'detect_register',
|
||||
'wayback': 'detect_wayback',
|
||||
'chinaz': 'detect_chinaz',
|
||||
'aizhan': 'detect_aizhan',
|
||||
'baidu_site': 'detect_baidu_site',
|
||||
'360_site': 'detect_360_site',
|
||||
'whois': 'detect_jucha',
|
||||
'beian': 'detect_jucha',
|
||||
'intercept': 'detect_jucha',
|
||||
'jucha': 'detect_jucha',
|
||||
'juziseo': 'detect_juziseo',
|
||||
'juziseo_outlink': 'detect_juziseo'
|
||||
}
|
||||
|
||||
# 获取对应的配置键
|
||||
config_key = type_mapping.get(detect_type)
|
||||
if not config_key:
|
||||
return True # 默认执行
|
||||
|
||||
# 检查是否在配置中,默认为True
|
||||
return config.DETECT_OPTIONS.get(config_key, True)
|
||||
0
domainCheck/app/config/sensitive_words.txt
Normal file
0
domainCheck/app/config/sensitive_words.txt
Normal file
4
domainCheck/app/core/__init__.py
Normal file
4
domainCheck/app/core/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
核心功能模块
|
||||
'''
|
||||
286
domainCheck/app/core/detect_engine.py
Normal file
286
domainCheck/app/core/detect_engine.py
Normal file
@@ -0,0 +1,286 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :detect_engine.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:52
|
||||
@explain : 检测引擎
|
||||
'''
|
||||
|
||||
import time
|
||||
from loguru import logger
|
||||
from app.utils.database import Database
|
||||
from app.detectors.rdap_detector import RDAPDetector
|
||||
from app.detectors.wayback_detector import WaybackDetector
|
||||
from app.detectors.baidu_detector import BaiduDetector
|
||||
from app.detectors.qihu360_detector import Qihu360Detector
|
||||
from app.detectors.google_detector import GoogleDetector
|
||||
from app.detectors.chinaz_detector import ChinazDetector
|
||||
from app.detectors.aizhan_detector import AizhanDetector
|
||||
from app.detectors.juziseo_detector import JuziseoDetector
|
||||
from app.detectors.jucha_detector import JuchaDetector
|
||||
from app.utils.status_codes import (
|
||||
DETECT_STATUS_BLACKLISTED,
|
||||
DETECT_STATUS_COMPLETED,
|
||||
DETECT_STATUS_FAILED,
|
||||
DETECT_STATUS_RUNNING,
|
||||
)
|
||||
|
||||
|
||||
class DetectEngine:
|
||||
"""
|
||||
检测引擎
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化检测引擎
|
||||
"""
|
||||
self.db = Database()
|
||||
self.rdap_detector = RDAPDetector()
|
||||
self.wayback_detector = WaybackDetector()
|
||||
self.baidu_detector = BaiduDetector()
|
||||
self.qihu360_detector = Qihu360Detector()
|
||||
self.google_detector = GoogleDetector()
|
||||
self.chinaz_detector = ChinazDetector()
|
||||
self.aizhan_detector = AizhanDetector()
|
||||
self.juziseo_detector = JuziseoDetector()
|
||||
self.jucha_detector = JuchaDetector()
|
||||
|
||||
def detect_domain(self, domain_id):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:return: bool - 是否检测成功
|
||||
"""
|
||||
try:
|
||||
# 获取域名信息
|
||||
domain_info = self.db.get_domain_by_id(domain_id)
|
||||
if not domain_info:
|
||||
logger.error(f"域名不存在: {domain_id}")
|
||||
return False
|
||||
|
||||
domain = domain_info['domain']
|
||||
logger.info(f"开始检测域名: {domain}")
|
||||
|
||||
# 更新检测状态为检测中
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_RUNNING)
|
||||
|
||||
# 1. 基础检测
|
||||
if not self._basic_detect(domain_id, domain):
|
||||
logger.info(f"基础检测失败,停止后续检测: {domain}")
|
||||
return False
|
||||
|
||||
# 2. 深度检测
|
||||
if not self._deep_detect(domain_id, domain):
|
||||
logger.info(f"深度检测失败: {domain}")
|
||||
return False
|
||||
|
||||
# 更新检测状态为正常
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_COMPLETED)
|
||||
logger.info(f"域名检测完成: {domain}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"检测域名出错: {e}")
|
||||
# 更新检测状态为检测失败
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED)
|
||||
return False
|
||||
|
||||
def _basic_detect(self, domain_id, domain):
|
||||
"""
|
||||
基础检测
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:param domain: 域名
|
||||
:return: bool - 是否检测通过
|
||||
"""
|
||||
# 1. 检查是否为一口价域名
|
||||
is_ykj = self.db.is_ykj_domain(domain_id)
|
||||
|
||||
# 2. 注册状态检测(一口价域名跳过)
|
||||
if not is_ykj:
|
||||
register_status = self.rdap_detector.check_register_status(domain)
|
||||
self.db.update_domain_register_status(domain_id, register_status)
|
||||
|
||||
# 3. 黑名单缓存检查
|
||||
if self.db.is_blacklisted(domain):
|
||||
logger.info(f"域名在黑名单中: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
return False
|
||||
|
||||
# 4. 时光机快照年份采集
|
||||
snapshot_years = self.wayback_detector.get_snapshot_years(domain)
|
||||
if snapshot_years:
|
||||
self.db.update_domain_snapshot_years(domain_id, ','.join(map(str, snapshot_years)))
|
||||
|
||||
# 5. 时光机正文抽样与敏感词匹配
|
||||
if self.wayback_detector.has_sensitive_content(domain):
|
||||
logger.info(f"域名包含敏感词: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
self.db.add_to_blacklist(domain, "快照包含敏感词")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def _deep_detect(self, domain_id, domain):
|
||||
"""
|
||||
深度检测
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:param domain: 域名
|
||||
:return: bool - 是否检测通过
|
||||
"""
|
||||
# 1. 百度历史/Site
|
||||
baidu_history = self.baidu_detector.check_history(domain)
|
||||
baidu_site = self.baidu_detector.check_site(domain)
|
||||
|
||||
# 2. 360 Site
|
||||
qihu360_site = self.qihu360_detector.check_site(domain)
|
||||
|
||||
# 3. Google Site
|
||||
google_site = self.google_detector.check_site(domain)
|
||||
|
||||
# 4. 站长之家
|
||||
chinaz_info = self.chinaz_detector.check_domain(domain)
|
||||
|
||||
# 5. 爱站网
|
||||
aizhan_info = self.aizhan_detector.check_domain(domain)
|
||||
|
||||
# 6. 桔子SEO
|
||||
juziseo_info = self.juziseo_detector.check_domain(domain)
|
||||
|
||||
# 7. 聚查
|
||||
jucha_info = self.jucha_detector.check_domain(domain)
|
||||
|
||||
# 检查是否有风险
|
||||
if self._check_risk(domain_id, domain, baidu_history, baidu_site, qihu360_site, google_site, chinaz_info, aizhan_info, juziseo_info, jucha_info):
|
||||
return False
|
||||
|
||||
# 保存检测结果
|
||||
self.db.add_detection_result(domain_id, baidu_history, baidu_site, qihu360_site, google_site, chinaz_info, aizhan_info, juziseo_info, jucha_info)
|
||||
|
||||
return True
|
||||
|
||||
def _check_risk(self, domain_id, domain, baidu_history, baidu_site, qihu360_site, google_site, chinaz_info, aizhan_info, juziseo_info, jucha_info):
|
||||
"""
|
||||
检查风险
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:param domain: 域名
|
||||
:param baidu_history: 百度历史
|
||||
:param baidu_site: 百度site
|
||||
:param qihu360_site: 360 site
|
||||
:param google_site: Google site
|
||||
:param chinaz_info: 站长之家信息
|
||||
:param aizhan_info: 爱站网信息
|
||||
:param juziseo_info: 桔子SEO信息
|
||||
:param jucha_info: 聚查信息
|
||||
:return: bool - 是否有风险
|
||||
"""
|
||||
# 检查百度历史过灰
|
||||
if baidu_history and '灰' in str(baidu_history):
|
||||
logger.info(f"百度历史过灰: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
self.db.add_to_blacklist(domain, "百度历史过灰")
|
||||
return True
|
||||
|
||||
# 检查标题敏感词
|
||||
if chinaz_info and 'title' in chinaz_info:
|
||||
if self._contains_sensitive_words(chinaz_info['title']):
|
||||
logger.info(f"标题包含敏感词: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
self.db.add_to_blacklist(domain, "标题包含敏感词")
|
||||
return True
|
||||
|
||||
# 检查子域名
|
||||
if baidu_site and 'subdomains' in baidu_site:
|
||||
subdomains = baidu_site['subdomains']
|
||||
# 排除 www, @, m
|
||||
valid_subdomains = [sub for sub in subdomains if sub not in ['www', '@', 'm']]
|
||||
if valid_subdomains:
|
||||
logger.info(f"存在子域名: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
self.db.add_to_blacklist(domain, "存在子域名")
|
||||
return True
|
||||
|
||||
# 检查风险提示
|
||||
if aizhan_info and 'risk' in aizhan_info:
|
||||
if aizhan_info['risk'] in ['低风险', '疑似色情博彩风险', '严重影响权重']:
|
||||
logger.info(f"风险提示: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
self.db.add_to_blacklist(domain, f"风险提示: {aizhan_info['risk']}")
|
||||
return True
|
||||
|
||||
# 检查WHOIS状态
|
||||
if jucha_info and 'whois' in jucha_info:
|
||||
if jucha_info['whois'].get('status') in ['clientHold', 'serverHold']:
|
||||
logger.info(f"WHOIS状态异常: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
|
||||
self.db.add_to_blacklist(domain, "WHOIS状态异常")
|
||||
return True
|
||||
|
||||
# 检查拦截检测
|
||||
if jucha_info and 'intercept' in jucha_info:
|
||||
if not jucha_info['intercept'].get('normal', True):
|
||||
logger.info(f"拦截检测异常: {domain}")
|
||||
self.db.update_domain_detect_status(domain_id, 4) # 4 表示黑名单
|
||||
self.db.add_to_blacklist(domain, "拦截检测异常")
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _contains_sensitive_words(self, text):
|
||||
"""
|
||||
检查文本是否包含敏感词
|
||||
|
||||
:param text: 文本
|
||||
:return: bool - 是否包含敏感词
|
||||
"""
|
||||
sensitive_words = self.db.get_all_sensitive_words()
|
||||
for word in sensitive_words:
|
||||
if word in text:
|
||||
return True
|
||||
return False
|
||||
|
||||
def process_task(self, task_id):
|
||||
"""
|
||||
处理检测任务
|
||||
|
||||
:param task_id: 任务ID
|
||||
:return: bool - 是否处理成功
|
||||
"""
|
||||
try:
|
||||
# 获取任务信息
|
||||
task = self.db.get_task_by_id(task_id)
|
||||
if not task:
|
||||
logger.error(f"任务不存在: {task_id}")
|
||||
return False
|
||||
|
||||
domain_id = task['domain_id']
|
||||
|
||||
# 更新任务状态为执行中
|
||||
self.db.update_task_status(task_id, 1) # 1 表示执行中
|
||||
|
||||
# 执行检测
|
||||
success = self.detect_domain(domain_id)
|
||||
|
||||
# 更新任务状态
|
||||
if success:
|
||||
self.db.update_task_status(task_id, 2) # 2 表示完成
|
||||
else:
|
||||
# 增加重试次数
|
||||
retry_count = task.get('retry_count', 0) + 1
|
||||
if retry_count < 3:
|
||||
self.db.update_task_retry_count(task_id, retry_count)
|
||||
self.db.update_task_status(task_id, 0) # 0 表示待执行
|
||||
else:
|
||||
self.db.update_task_status(task_id, 3) # 3 表示失败
|
||||
|
||||
return success
|
||||
except Exception as e:
|
||||
logger.error(f"处理任务出错: {e}")
|
||||
# 更新任务状态为失败
|
||||
self.db.update_task_status(task_id, 3) # 3 表示失败
|
||||
return False
|
||||
261
domainCheck/app/core/domain_collector.py
Normal file
261
domainCheck/app/core/domain_collector.py
Normal file
@@ -0,0 +1,261 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :domain_collector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:50
|
||||
@explain : 域名收集器
|
||||
'''
|
||||
|
||||
import re
|
||||
import tldextract
|
||||
from loguru import logger
|
||||
from app.utils.database import Database
|
||||
from app.utils.domain_utils import normalize_domain
|
||||
from app.config import config
|
||||
|
||||
|
||||
class DomainCollector:
|
||||
"""
|
||||
域名收集器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化域名收集器
|
||||
"""
|
||||
self.db = Database(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
|
||||
def add_domain(self, domain, source_type):
|
||||
"""
|
||||
添加域名
|
||||
|
||||
:param domain: 域名
|
||||
:param source_type: 来源类型
|
||||
:return: bool - 是否添加成功
|
||||
"""
|
||||
try:
|
||||
# 标准化域名
|
||||
normalized_domain = normalize_domain(domain)
|
||||
if not normalized_domain:
|
||||
logger.warning(f"无效域名: {domain}")
|
||||
return False
|
||||
|
||||
# 提取顶级域名
|
||||
ext = tldextract.extract(normalized_domain)
|
||||
tld = ext.suffix
|
||||
|
||||
# 只保留 .com 和 .net
|
||||
if tld not in ['com', 'net']:
|
||||
logger.warning(f"不支持的顶级域名: {tld}")
|
||||
return False
|
||||
|
||||
# 检查是否已存在
|
||||
if self.db.domain_exists(normalized_domain):
|
||||
logger.info(f"域名已存在: {normalized_domain}")
|
||||
return False
|
||||
|
||||
# 添加域名
|
||||
domain_id = self.db.add_domain(normalized_domain, tld, source_type)
|
||||
if domain_id:
|
||||
# 创建检测任务
|
||||
self.db.create_detect_task(domain_id, 1) # 1 表示基础检测
|
||||
logger.info(f"成功添加域名: {normalized_domain}")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"添加域名失败: {normalized_domain}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"添加域名出错: {e}")
|
||||
return False
|
||||
|
||||
def add_domains_batch(self, domains, source_type, batch_size=1000, dry_run=False):
|
||||
"""
|
||||
批量添加域名
|
||||
|
||||
:param domains: 域名列表
|
||||
:param source_type: 来源类型
|
||||
:param batch_size: 批量大小
|
||||
:param dry_run: 是否仅进行干运行(不实际添加域名)
|
||||
:return: dict - 统计信息
|
||||
"""
|
||||
try:
|
||||
# 统计信息
|
||||
stats = {
|
||||
'total': len(domains),
|
||||
'valid': 0,
|
||||
'added': 0,
|
||||
'exists': 0,
|
||||
'invalid': 0,
|
||||
'failed': 0
|
||||
}
|
||||
|
||||
# 处理大规模数据时,分批进行标准化和过滤
|
||||
normalized_domains = []
|
||||
batch_domains = []
|
||||
|
||||
for i, domain in enumerate(domains):
|
||||
normalized = normalize_domain(domain)
|
||||
if not normalized:
|
||||
stats['invalid'] += 1
|
||||
continue
|
||||
|
||||
# 提取顶级域名
|
||||
ext = tldextract.extract(normalized)
|
||||
tld = ext.suffix
|
||||
|
||||
# 只保留 .com 和 .net
|
||||
if tld not in ['com', 'net']:
|
||||
stats['invalid'] += 1
|
||||
continue
|
||||
|
||||
normalized_domains.append((normalized, tld))
|
||||
batch_domains.append(normalized)
|
||||
stats['valid'] += 1
|
||||
|
||||
# 每1000个域名检查一次,避免内存占用过高
|
||||
if (i + 1) % 1000 == 0:
|
||||
logger.info(f"已处理 {i + 1}/{len(domains)} 个域名")
|
||||
|
||||
logger.info(f"域名标准化完成,有效域名: {stats['valid']}")
|
||||
|
||||
# 提取所有域名
|
||||
all_domains = [domain for domain, tld in normalized_domains]
|
||||
|
||||
# 批量检查域名是否存在
|
||||
existing_domains = self.db.check_domains_exist(all_domains)
|
||||
existing_set = set(existing_domains)
|
||||
|
||||
# 准备批量添加数据
|
||||
batch_data = []
|
||||
for domain, tld in normalized_domains:
|
||||
if domain not in existing_set:
|
||||
batch_data.append((domain, tld, source_type))
|
||||
|
||||
stats['exists'] = len(existing_domains)
|
||||
stats['valid'] = len(normalized_domains)
|
||||
|
||||
# 干运行模式下直接返回统计信息
|
||||
if dry_run:
|
||||
stats['added'] = len(batch_data)
|
||||
logger.info(f"干运行模式:准备添加 {len(batch_data)} 个新域名")
|
||||
return stats
|
||||
|
||||
logger.info(f"准备添加 {len(batch_data)} 个新域名")
|
||||
|
||||
# 分批次添加
|
||||
for i in range(0, len(batch_data), batch_size):
|
||||
batch = batch_data[i:i+batch_size]
|
||||
added_count = self.db.add_domains_batch(batch)
|
||||
stats['added'] += added_count
|
||||
stats['failed'] += len(batch) - added_count
|
||||
|
||||
# 每处理一批,记录一次进度
|
||||
if (i + len(batch)) % (batch_size * 10) == 0:
|
||||
logger.info(f"已添加 {i + len(batch)}/{len(batch_data)} 个域名")
|
||||
|
||||
logger.info(f"批量添加域名完成: {stats}")
|
||||
return stats
|
||||
except Exception as e:
|
||||
logger.error(f"批量添加域名出错: {e}")
|
||||
return {
|
||||
'total': len(domains),
|
||||
'valid': 0,
|
||||
'added': 0,
|
||||
'exists': 0,
|
||||
'invalid': 0,
|
||||
'failed': len(domains)
|
||||
}
|
||||
|
||||
def import_from_file(self, file_path, source_type, batch_size=1000):
|
||||
"""
|
||||
从文件导入域名
|
||||
|
||||
:param file_path: 文件路径
|
||||
:param source_type: 来源类型
|
||||
:param batch_size: 批量大小
|
||||
:return: dict - 导入统计信息
|
||||
"""
|
||||
try:
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
domains = f.readlines()
|
||||
|
||||
# 提取域名
|
||||
domain_list = []
|
||||
for domain in domains:
|
||||
domain = domain.strip()
|
||||
if domain:
|
||||
domain_list.append(domain)
|
||||
|
||||
# 批量添加域名
|
||||
stats = self.add_domains_batch(domain_list, source_type, batch_size)
|
||||
|
||||
logger.info(f"从文件导入完成: {stats}")
|
||||
return stats
|
||||
except Exception as e:
|
||||
logger.error(f"从文件导入出错: {e}")
|
||||
return {
|
||||
'total': 0,
|
||||
'valid': 0,
|
||||
'added': 0,
|
||||
'exists': 0,
|
||||
'invalid': 0,
|
||||
'failed': 0
|
||||
}
|
||||
|
||||
def collect_from_juming(self, type_='一口价'):
|
||||
"""
|
||||
从聚名网收集域名
|
||||
|
||||
:param type_: 类型,一口价或过期删除
|
||||
:return: int - 收集到的域名数量
|
||||
"""
|
||||
# 这里可以添加从聚名网收集域名的逻辑
|
||||
logger.info(f"从聚名网收集 {type_} 域名")
|
||||
# 模拟收集结果
|
||||
return 0
|
||||
|
||||
def collect_from_search_engine(self, keyword, limit=100):
|
||||
"""
|
||||
从搜索引擎收集域名
|
||||
|
||||
:param keyword: 关键词
|
||||
:param limit: 限制数量
|
||||
:return: int - 收集到的域名数量
|
||||
"""
|
||||
# 这里可以添加从搜索引擎收集域名的逻辑
|
||||
logger.info(f"从搜索引擎收集域名,关键词: {keyword}, 限制: {limit}")
|
||||
# 模拟收集结果
|
||||
return 0
|
||||
|
||||
def collect_from_enterprise_directory(self, url, limit=100):
|
||||
"""
|
||||
从企业目录收集域名
|
||||
|
||||
:param url: 企业目录URL
|
||||
:param limit: 限制数量
|
||||
:return: int - 收集到的域名数量
|
||||
"""
|
||||
# 这里可以添加从企业目录收集域名的逻辑
|
||||
logger.info(f"从企业目录收集域名,URL: {url}, 限制: {limit}")
|
||||
# 模拟收集结果
|
||||
return 0
|
||||
|
||||
def collect_from_zone_file(self, file_path):
|
||||
"""
|
||||
从Zone File收集域名
|
||||
|
||||
:param file_path: Zone File路径
|
||||
:return: int - 收集到的域名数量
|
||||
"""
|
||||
# 这里可以添加从Zone File收集域名的逻辑
|
||||
logger.info(f"从Zone File收集域名,文件: {file_path}")
|
||||
# 模拟收集结果
|
||||
return 0
|
||||
150
domainCheck/app/core/domain_processor.py
Normal file
150
domainCheck/app/core/domain_processor.py
Normal file
@@ -0,0 +1,150 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :domain_processor.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:51
|
||||
@explain : 域名处理器
|
||||
'''
|
||||
|
||||
import re
|
||||
import tldextract
|
||||
from loguru import logger
|
||||
from app.utils.database import Database
|
||||
from app.utils.domain_utils import normalize_domain
|
||||
|
||||
|
||||
class DomainProcessor:
|
||||
"""
|
||||
域名处理器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化域名处理器
|
||||
"""
|
||||
self.db = Database()
|
||||
|
||||
def process_domain(self, domain):
|
||||
"""
|
||||
处理域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 处理结果
|
||||
"""
|
||||
result = {
|
||||
'original': domain,
|
||||
'normalized': None,
|
||||
'tld': None,
|
||||
'valid': False,
|
||||
'reason': ''
|
||||
}
|
||||
|
||||
try:
|
||||
# 标准化域名
|
||||
normalized = normalize_domain(domain)
|
||||
if not normalized:
|
||||
result['reason'] = '无效域名格式'
|
||||
return result
|
||||
|
||||
# 提取顶级域名
|
||||
ext = tldextract.extract(normalized)
|
||||
tld = ext.suffix
|
||||
|
||||
# 检查顶级域名
|
||||
if tld not in ['com', 'net']:
|
||||
result['reason'] = '不支持的顶级域名'
|
||||
return result
|
||||
|
||||
result['normalized'] = normalized
|
||||
result['tld'] = tld
|
||||
result['valid'] = True
|
||||
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(f"处理域名出错: {e}")
|
||||
result['reason'] = f"处理出错: {str(e)}"
|
||||
return result
|
||||
|
||||
def batch_process(self, domains):
|
||||
"""
|
||||
批量处理域名
|
||||
|
||||
:param domains: 域名列表
|
||||
:return: list - 处理结果列表
|
||||
"""
|
||||
results = []
|
||||
for domain in domains:
|
||||
result = self.process_domain(domain)
|
||||
results.append(result)
|
||||
return results
|
||||
|
||||
def filter_valid_domains(self, domains):
|
||||
"""
|
||||
过滤有效的域名
|
||||
|
||||
:param domains: 域名列表
|
||||
:return: list - 有效的域名列表
|
||||
"""
|
||||
valid_domains = []
|
||||
for domain in domains:
|
||||
result = self.process_domain(domain)
|
||||
if result['valid']:
|
||||
valid_domains.append(result['normalized'])
|
||||
return valid_domains
|
||||
|
||||
def update_domain_status(self, domain_id, status_type, status_value):
|
||||
"""
|
||||
更新域名状态
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:param status_type: 状态类型
|
||||
:param status_value: 状态值
|
||||
:return: bool - 是否更新成功
|
||||
"""
|
||||
try:
|
||||
if status_type == 'use_status':
|
||||
return self.db.update_domain_use_status(domain_id, status_value)
|
||||
elif status_type == 'detect_status':
|
||||
return self.db.update_domain_detect_status(domain_id, status_value)
|
||||
elif status_type == 'register_status':
|
||||
return self.db.update_domain_register_status(domain_id, status_value)
|
||||
else:
|
||||
logger.error(f"未知的状态类型: {status_type}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"更新域名状态出错: {e}")
|
||||
return False
|
||||
|
||||
def blacklist_domain(self, domain_id, reason):
|
||||
"""
|
||||
将域名加入黑名单
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:param reason: 黑名单原因
|
||||
:return: bool - 是否操作成功
|
||||
"""
|
||||
try:
|
||||
# 更新检测状态为黑名单
|
||||
if self.db.update_domain_detect_status(domain_id, 4): # 4 表示黑名单
|
||||
# 添加到黑名单表
|
||||
domain = self.db.get_domain_by_id(domain_id)
|
||||
if domain:
|
||||
return self.db.add_to_blacklist(domain['domain'], reason)
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"将域名加入黑名单出错: {e}")
|
||||
return False
|
||||
|
||||
def get_domain_statistics(self):
|
||||
"""
|
||||
获取域名统计信息
|
||||
|
||||
:return: dict - 统计信息
|
||||
"""
|
||||
try:
|
||||
return self.db.get_domain_statistics()
|
||||
except Exception as e:
|
||||
logger.error(f"获取域名统计信息出错: {e}")
|
||||
return {}
|
||||
289
domainCheck/app/core/export_manager.py
Normal file
289
domainCheck/app/core/export_manager.py
Normal file
@@ -0,0 +1,289 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :export_manager.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:54
|
||||
@explain : 导出管理器
|
||||
'''
|
||||
|
||||
import os
|
||||
from loguru import logger
|
||||
from app.utils.database import Database
|
||||
|
||||
|
||||
class ExportManager:
|
||||
"""
|
||||
导出管理器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化导出管理器
|
||||
"""
|
||||
self.db = Database()
|
||||
|
||||
def export_domains(self, filter_conditions, output_file):
|
||||
"""
|
||||
导出域名
|
||||
|
||||
:param filter_conditions: 筛选条件
|
||||
:param output_file: 输出文件路径
|
||||
:return: int - 导出的域名数量
|
||||
"""
|
||||
try:
|
||||
# 查询符合条件的域名
|
||||
domains = self.db.get_domains_by_conditions(filter_conditions)
|
||||
|
||||
if not domains:
|
||||
logger.warning("没有符合条件的域名")
|
||||
return 0
|
||||
|
||||
# 导出到文件
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
for domain in domains:
|
||||
f.write(domain['domain'] + '\n')
|
||||
|
||||
logger.info(f"成功导出 {len(domains)} 个域名到 {output_file}")
|
||||
return len(domains)
|
||||
except Exception as e:
|
||||
logger.error(f"导出域名出错: {e}")
|
||||
return 0
|
||||
|
||||
def export_with_details(self, filter_conditions, output_file):
|
||||
"""
|
||||
导出域名及其详细信息
|
||||
|
||||
:param filter_conditions: 筛选条件
|
||||
:param output_file: 输出文件路径
|
||||
:return: int - 导出的域名数量
|
||||
"""
|
||||
try:
|
||||
# 查询符合条件的域名及其详细信息
|
||||
domains = self.db.get_domains_with_details(filter_conditions)
|
||||
|
||||
if not domains:
|
||||
logger.warning("没有符合条件的域名")
|
||||
return 0
|
||||
|
||||
# 导出到文件
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
# 写入表头
|
||||
f.write('域名,注册状态,使用状态,检测状态,备案历史,备案年份,快照年份,友情链接数量\n')
|
||||
|
||||
# 写入数据
|
||||
for domain in domains:
|
||||
line = f"{domain['domain']},{domain['register_status']},{domain['use_status']},{domain['detect_status']},{domain['has_beian']},{domain['beian_year']},{domain['snapshot_years']},{domain['backlink_count']}\n"
|
||||
f.write(line)
|
||||
|
||||
logger.info(f"成功导出 {len(domains)} 个域名及其详细信息到 {output_file}")
|
||||
return len(domains)
|
||||
except Exception as e:
|
||||
logger.error(f"导出域名详细信息出错: {e}")
|
||||
return 0
|
||||
|
||||
def batch_update_status(self, domain_ids, status_type, status_value):
|
||||
"""
|
||||
批量更新域名状态
|
||||
|
||||
:param domain_ids: 域名ID列表
|
||||
:param status_type: 状态类型
|
||||
:param status_value: 状态值
|
||||
:return: int - 更新成功的域名数量
|
||||
"""
|
||||
try:
|
||||
success_count = 0
|
||||
for domain_id in domain_ids:
|
||||
if self.db.update_domain_status(domain_id, status_type, status_value):
|
||||
success_count += 1
|
||||
|
||||
logger.info(f"成功更新 {success_count} 个域名的状态")
|
||||
return success_count
|
||||
except Exception as e:
|
||||
logger.error(f"批量更新域名状态出错: {e}")
|
||||
return 0
|
||||
|
||||
def get_filtered_domains(self, filter_conditions, limit=1000):
|
||||
"""
|
||||
获取符合条件的域名
|
||||
|
||||
:param filter_conditions: 筛选条件
|
||||
:param limit: 限制数量
|
||||
:return: list - 域名列表
|
||||
"""
|
||||
try:
|
||||
domains = self.db.get_domains_by_conditions(filter_conditions, limit)
|
||||
logger.info(f"获取到 {len(domains)} 个符合条件的域名")
|
||||
return domains
|
||||
except Exception as e:
|
||||
logger.error(f"获取符合条件的域名出错: {e}")
|
||||
return []
|
||||
|
||||
def generate_report(self, output_file):
|
||||
"""
|
||||
生成统计报告
|
||||
|
||||
:param output_file: 输出文件路径
|
||||
:return: bool - 是否生成成功
|
||||
"""
|
||||
try:
|
||||
# 获取域名统计信息
|
||||
domain_stats = self.db.get_domain_statistics()
|
||||
|
||||
# 获取任务统计信息
|
||||
task_stats = self.db.get_task_statistics()
|
||||
|
||||
# 生成报告
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
f.write('域名库系统统计报告\n')
|
||||
f.write('=' * 50 + '\n')
|
||||
|
||||
f.write('\n域名统计:\n')
|
||||
f.write(f'总域名数: {domain_stats.get("total", 0)}\n')
|
||||
f.write(f'可注册域名: {domain_stats.get("available", 0)}\n')
|
||||
f.write(f'已注册域名: {domain_stats.get("registered", 0)}\n')
|
||||
f.write(f'黑名单域名: {domain_stats.get("blacklisted", 0)}\n')
|
||||
|
||||
f.write('\n任务统计:\n')
|
||||
f.write(f'总任务数: {task_stats.get("total", 0)}\n')
|
||||
f.write(f'待执行任务: {task_stats.get("pending", 0)}\n')
|
||||
f.write(f'执行中任务: {task_stats.get("running", 0)}\n')
|
||||
f.write(f'已完成任务: {task_stats.get("completed", 0)}\n')
|
||||
f.write(f'失败任务: {task_stats.get("failed", 0)}\n')
|
||||
|
||||
logger.info(f"成功生成统计报告到 {output_file}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"生成统计报告出错: {e}")
|
||||
return False
|
||||
|
||||
def export_to_excel(self, domains, output_file):
|
||||
"""
|
||||
导出域名到Excel文件
|
||||
|
||||
:param domains: 域名列表
|
||||
:param output_file: 输出文件路径
|
||||
:return: int - 导出的域名数量
|
||||
"""
|
||||
try:
|
||||
# 尝试导入openpyxl
|
||||
try:
|
||||
from openpyxl import Workbook
|
||||
except ImportError:
|
||||
# 如果没有安装openpyxl,使用CSV格式作为替代
|
||||
logger.warning("openpyxl库未安装,将使用CSV格式导出")
|
||||
return self.export_to_csv(domains, output_file.replace('.xlsx', '.csv'))
|
||||
|
||||
# 创建工作簿
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
|
||||
# 写入表头
|
||||
headers = ['域名', '注册状态', '使用状态', '检测状态', '人工复核状态', '过期时间', '单位性质', '网站首页网址', '检测时间', '备案历史', '备案年份', '快照年份', '百度历史', '百度Site', '是否中文标题', '360 Site', 'Google Site', '友情链接数量']
|
||||
ws.append(headers)
|
||||
|
||||
# 写入数据
|
||||
for domain in domains:
|
||||
row = [
|
||||
domain.get('domain', ''),
|
||||
domain.get('register_status', ''),
|
||||
domain.get('use_status', ''),
|
||||
domain.get('detect_status', ''),
|
||||
domain.get('review_status', ''),
|
||||
domain.get('expire_date', ''),
|
||||
domain.get('company_type', ''),
|
||||
domain.get('website_url', ''),
|
||||
domain.get('detect_time', ''),
|
||||
domain.get('has_beian', ''),
|
||||
domain.get('beian_year', ''),
|
||||
domain.get('snapshot_years', ''),
|
||||
domain.get('baidu_history', ''),
|
||||
domain.get('baidu_site', ''),
|
||||
domain.get('is_chinese_title', ''),
|
||||
domain.get('qihu360_site', ''),
|
||||
domain.get('google_site', ''),
|
||||
domain.get('backlink_count', '')
|
||||
]
|
||||
ws.append(row)
|
||||
|
||||
# 保存文件
|
||||
wb.save(output_file)
|
||||
|
||||
logger.info(f"成功导出 {len(domains)} 个域名到Excel文件: {output_file}")
|
||||
return len(domains)
|
||||
except Exception as e:
|
||||
logger.error(f"导出Excel文件出错: {e}")
|
||||
# 尝试使用CSV格式作为替代
|
||||
try:
|
||||
csv_file = output_file.replace('.xlsx', '.csv')
|
||||
logger.info(f"尝试使用CSV格式导出到: {csv_file}")
|
||||
return self.export_to_csv(domains, csv_file)
|
||||
except Exception as e2:
|
||||
logger.error(f"导出CSV文件也失败: {e2}")
|
||||
raise
|
||||
|
||||
def export_to_csv(self, domains, output_file):
|
||||
"""
|
||||
导出域名到CSV文件
|
||||
|
||||
:param domains: 域名列表
|
||||
:param output_file: 输出文件路径
|
||||
:return: int - 导出的域名数量
|
||||
"""
|
||||
try:
|
||||
import csv
|
||||
|
||||
# 写入文件
|
||||
with open(output_file, 'w', encoding='utf-8', newline='') as f:
|
||||
writer = csv.writer(f)
|
||||
|
||||
# 写入表头
|
||||
headers = ['域名', '注册状态', '使用状态', '检测状态', '人工复核状态', '过期时间', '单位性质', '网站首页网址', '检测时间', '备案历史', '备案年份', '快照年份', '百度历史', '百度Site', '是否中文标题', '360 Site', 'Google Site', '友情链接数量']
|
||||
writer.writerow(headers)
|
||||
|
||||
# 写入数据
|
||||
for domain in domains:
|
||||
row = [
|
||||
domain.get('domain', ''),
|
||||
domain.get('register_status', ''),
|
||||
domain.get('use_status', ''),
|
||||
domain.get('detect_status', ''),
|
||||
domain.get('review_status', ''),
|
||||
domain.get('expire_date', ''),
|
||||
domain.get('company_type', ''),
|
||||
domain.get('website_url', ''),
|
||||
domain.get('detect_time', ''),
|
||||
domain.get('has_beian', ''),
|
||||
domain.get('beian_year', ''),
|
||||
domain.get('snapshot_years', ''),
|
||||
domain.get('baidu_history', ''),
|
||||
domain.get('baidu_site', ''),
|
||||
domain.get('is_chinese_title', ''),
|
||||
domain.get('qihu360_site', ''),
|
||||
domain.get('google_site', ''),
|
||||
domain.get('backlink_count', '')
|
||||
]
|
||||
writer.writerow(row)
|
||||
|
||||
logger.info(f"成功导出 {len(domains)} 个域名到CSV文件: {output_file}")
|
||||
return len(domains)
|
||||
except Exception as e:
|
||||
logger.error(f"导出CSV文件出错: {e}")
|
||||
raise
|
||||
|
||||
def export_to_txt(self, domains, output_file):
|
||||
"""
|
||||
导出域名到TXT文件,一行一个域名。
|
||||
"""
|
||||
try:
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
for domain in domains:
|
||||
value = domain.get('domain', '').strip()
|
||||
if value:
|
||||
f.write(value + '\n')
|
||||
logger.info(f"成功导出 {len(domains)} 个域名到TXT文件: {output_file}")
|
||||
return len(domains)
|
||||
except Exception as e:
|
||||
logger.error(f"导出TXT文件出错: {e}")
|
||||
raise
|
||||
162
domainCheck/app/core/task_scheduler.py
Normal file
162
domainCheck/app/core/task_scheduler.py
Normal file
@@ -0,0 +1,162 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :task_scheduler.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:53
|
||||
@explain : 任务调度器
|
||||
'''
|
||||
|
||||
import time
|
||||
import threading
|
||||
from loguru import logger
|
||||
from app.utils.database import Database
|
||||
from app.core.detect_engine import DetectEngine
|
||||
|
||||
|
||||
class TaskScheduler:
|
||||
"""
|
||||
任务调度器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化任务调度器
|
||||
"""
|
||||
self.db = Database()
|
||||
self.detect_engine = DetectEngine()
|
||||
self.running = False
|
||||
self.threads = []
|
||||
self.max_threads = 10
|
||||
|
||||
def start(self):
|
||||
"""
|
||||
启动任务调度器
|
||||
"""
|
||||
if self.running:
|
||||
logger.info("任务调度器已经在运行中")
|
||||
return
|
||||
|
||||
self.running = True
|
||||
logger.info("启动任务调度器")
|
||||
|
||||
# 启动多个线程处理任务
|
||||
for i in range(self.max_threads):
|
||||
thread = threading.Thread(target=self._process_tasks, daemon=True)
|
||||
thread.start()
|
||||
self.threads.append(thread)
|
||||
logger.info(f"启动任务处理线程 {i+1}")
|
||||
|
||||
def stop(self):
|
||||
"""
|
||||
停止任务调度器
|
||||
"""
|
||||
self.running = False
|
||||
logger.info("停止任务调度器")
|
||||
|
||||
# 等待线程结束
|
||||
for thread in self.threads:
|
||||
thread.join(timeout=5)
|
||||
|
||||
self.threads.clear()
|
||||
logger.info("任务调度器已停止")
|
||||
|
||||
def _process_tasks(self):
|
||||
"""
|
||||
处理任务
|
||||
"""
|
||||
while self.running:
|
||||
try:
|
||||
# 获取待执行的任务
|
||||
task = self.db.get_pending_task()
|
||||
if not task:
|
||||
# 没有任务,休眠一段时间
|
||||
time.sleep(1)
|
||||
continue
|
||||
|
||||
task_id = task['id']
|
||||
domain_id = task['domain_id']
|
||||
|
||||
logger.info(f"处理任务: {task_id}, 域名ID: {domain_id}")
|
||||
|
||||
# 执行任务
|
||||
success = self.detect_engine.process_task(task_id)
|
||||
|
||||
if success:
|
||||
logger.info(f"任务处理成功: {task_id}")
|
||||
else:
|
||||
logger.warning(f"任务处理失败: {task_id}")
|
||||
|
||||
# 短暂休眠,避免过于频繁的数据库操作
|
||||
time.sleep(0.1)
|
||||
except Exception as e:
|
||||
logger.error(f"处理任务出错: {e}")
|
||||
# 休眠一段时间,避免出错后无限循环
|
||||
time.sleep(5)
|
||||
|
||||
def add_task(self, domain_id, task_type=1, priority=0):
|
||||
"""
|
||||
添加任务
|
||||
|
||||
:param domain_id: 域名ID
|
||||
:param task_type: 任务类型,1-基础检测,2-深度检测
|
||||
:param priority: 优先级,0-低,1-中,2-高
|
||||
:return: int - 任务ID
|
||||
"""
|
||||
try:
|
||||
task_id = self.db.create_detect_task(domain_id, task_type, priority)
|
||||
logger.info(f"添加任务成功: {task_id}, 域名ID: {domain_id}")
|
||||
return task_id
|
||||
except Exception as e:
|
||||
logger.error(f"添加任务失败: {e}")
|
||||
return None
|
||||
|
||||
def get_task_stats(self):
|
||||
"""
|
||||
获取任务统计信息
|
||||
|
||||
:return: dict - 任务统计信息
|
||||
"""
|
||||
try:
|
||||
return self.db.get_task_statistics()
|
||||
except Exception as e:
|
||||
logger.error(f"获取任务统计信息出错: {e}")
|
||||
return {}
|
||||
|
||||
def retry_failed_tasks(self):
|
||||
"""
|
||||
重试失败的任务
|
||||
|
||||
:return: int - 重试的任务数量
|
||||
"""
|
||||
try:
|
||||
tasks = self.db.get_failed_tasks()
|
||||
retry_count = 0
|
||||
|
||||
for task in tasks:
|
||||
task_id = task['id']
|
||||
self.db.update_task_status(task_id, 0) # 0 表示待执行
|
||||
self.db.update_task_retry_count(task_id, 0) # 重置重试次数
|
||||
retry_count += 1
|
||||
|
||||
logger.info(f"重试 {retry_count} 个失败的任务")
|
||||
return retry_count
|
||||
except Exception as e:
|
||||
logger.error(f"重试失败任务出错: {e}")
|
||||
return 0
|
||||
|
||||
def clear_completed_tasks(self, days=7):
|
||||
"""
|
||||
清理已完成的任务
|
||||
|
||||
:param days: 保留天数
|
||||
:return: int - 清理的任务数量
|
||||
"""
|
||||
try:
|
||||
count = self.db.clear_completed_tasks(days)
|
||||
logger.info(f"清理 {count} 个已完成的任务")
|
||||
return count
|
||||
except Exception as e:
|
||||
logger.error(f"清理已完成任务出错: {e}")
|
||||
return 0
|
||||
20
domainCheck/app/detect_options.json
Normal file
20
domainCheck/app/detect_options.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"detect_register": true,
|
||||
"detect_wayback": true,
|
||||
"detect_chinaz": true,
|
||||
"detect_aizhan": true,
|
||||
"detect_baidu_site": true,
|
||||
"detect_360_site": true,
|
||||
"detect_jucha": false,
|
||||
"detect_juziseo": false,
|
||||
"detect_order": [
|
||||
"detect_register",
|
||||
"detect_baidu_site",
|
||||
"detect_360_site",
|
||||
"detect_chinaz",
|
||||
"detect_aizhan",
|
||||
"detect_wayback",
|
||||
"detect_jucha",
|
||||
"detect_juziseo"
|
||||
]
|
||||
}
|
||||
4
domainCheck/app/detectors/__init__.py
Normal file
4
domainCheck/app/detectors/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
检测插件模块
|
||||
'''
|
||||
130
domainCheck/app/detectors/aizhan_detector.py
Normal file
130
domainCheck/app/detectors/aizhan_detector.py
Normal file
@@ -0,0 +1,130 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :aizhan_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:02
|
||||
@explain : 爱站网检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class AizhanDetector(BaseDetector):
|
||||
"""
|
||||
爱站网检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化爱站网检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.url = 'https://www.aizhan.com'
|
||||
self.query_url = 'https://www.aizhan.com/cha/{domain}'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.query_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取标题
|
||||
title = self._extract_title(content)
|
||||
|
||||
# 提取风险信息
|
||||
risk = self._extract_risk(content)
|
||||
|
||||
# 检查是否包含敏感词
|
||||
has_sensitive = self._check_sensitive(title, risk)
|
||||
|
||||
return {
|
||||
'title': title,
|
||||
'risk': risk,
|
||||
'has_sensitive': has_sensitive
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"爱站网查询失败: {response.status_code}")
|
||||
return {'title': '', 'risk': '', 'has_sensitive': False}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def _extract_title(self, content):
|
||||
"""
|
||||
提取标题
|
||||
|
||||
:param content: 页面内容
|
||||
:return: str - 标题
|
||||
"""
|
||||
try:
|
||||
pattern = r'<title>(.*?)</title>'
|
||||
match = re.search(pattern, content)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return ''
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_title')
|
||||
return ''
|
||||
|
||||
def _extract_risk(self, content):
|
||||
"""
|
||||
提取风险信息
|
||||
|
||||
:param content: 页面内容
|
||||
:return: str - 风险信息
|
||||
"""
|
||||
try:
|
||||
# 这里需要根据实际页面结构调整正则表达式
|
||||
pattern = r'百度网址检测:<span[^>]+>(.*?)</span>'
|
||||
match = re.search(pattern, content)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return ''
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_risk')
|
||||
return ''
|
||||
|
||||
def _check_sensitive(self, title, risk):
|
||||
"""
|
||||
检查是否包含敏感词
|
||||
|
||||
:param title: 标题
|
||||
:param risk: 风险信息
|
||||
:return: bool - 是否包含敏感词
|
||||
"""
|
||||
# 风险类型
|
||||
sensitive_risks = ['低风险', '疑似色情博彩风险', '严重影响权重']
|
||||
|
||||
# 敏感词
|
||||
sensitive_words = ['色情', '赌博', '博彩', '毒品', '暴力', '诈骗']
|
||||
|
||||
# 检查风险
|
||||
for r in sensitive_risks:
|
||||
if r in risk:
|
||||
return True
|
||||
|
||||
# 检查标题
|
||||
for word in sensitive_words:
|
||||
if word in title:
|
||||
return True
|
||||
|
||||
return False
|
||||
151
domainCheck/app/detectors/baidu_detector.py
Normal file
151
domainCheck/app/detectors/baidu_detector.py
Normal file
@@ -0,0 +1,151 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :baidu_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:58
|
||||
@explain : 百度检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class BaiduDetector(BaseDetector):
|
||||
"""
|
||||
百度检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化百度检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.site_url = 'https://www.baidu.com/s'
|
||||
self.history_url = 'https://www.baidu.com/s'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 检查百度site
|
||||
site_result = self.check_site(domain)
|
||||
|
||||
# 检查百度历史
|
||||
history_result = self.check_history(domain)
|
||||
|
||||
return {
|
||||
'site': site_result,
|
||||
'history': history_result
|
||||
}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def check_site(self, domain):
|
||||
"""
|
||||
检查百度site收录
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
params = {
|
||||
'wd': f'site:{domain}',
|
||||
'rn': '50'
|
||||
}
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(self.site_url, params=params, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取子域名
|
||||
subdomains = self._extract_subdomains(content, domain)
|
||||
|
||||
# 检查是否有收录
|
||||
has_收录 = '没有找到相关结果' not in content
|
||||
|
||||
return {
|
||||
'has_收录': has_收录,
|
||||
'subdomains': subdomains
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"百度site查询失败: {response.status_code}")
|
||||
return {'has_收录': False, 'subdomains': []}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_收录': False, 'subdomains': []}
|
||||
|
||||
def check_history(self, domain):
|
||||
"""
|
||||
检查百度历史收录
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
params = {
|
||||
'wd': f'cache:{domain}',
|
||||
'rn': '50'
|
||||
}
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(self.history_url, params=params, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 检查是否有历史收录
|
||||
has_history = '百度快照' in content
|
||||
|
||||
# 检查是否有灰色内容
|
||||
has_gray = '风险提示' in content or '安全警告' in content
|
||||
|
||||
return {
|
||||
'has_history': has_history,
|
||||
'has_gray': has_gray
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"百度历史查询失败: {response.status_code}")
|
||||
return {'has_history': False, 'has_gray': False}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_history': False, 'has_gray': False}
|
||||
|
||||
def _extract_subdomains(self, content, domain):
|
||||
"""
|
||||
提取子域名
|
||||
|
||||
:param content: 搜索结果内容
|
||||
:param domain: 主域名
|
||||
:return: list - 子域名列表
|
||||
"""
|
||||
try:
|
||||
# 提取所有包含域名的链接
|
||||
pattern = r'https?://([a-zA-Z0-9-]+)\.' + re.escape(domain)
|
||||
matches = re.findall(pattern, content)
|
||||
|
||||
# 去重并过滤空值
|
||||
subdomains = list(set(matches))
|
||||
subdomains = [sub for sub in subdomains if sub]
|
||||
|
||||
return subdomains
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return []
|
||||
70
domainCheck/app/detectors/base.py
Normal file
70
domainCheck/app/detectors/base.py
Normal file
@@ -0,0 +1,70 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :base.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:55
|
||||
@explain : 基础检测类
|
||||
'''
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class BaseDetector(ABC):
|
||||
"""
|
||||
基础检测类
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化检测类
|
||||
"""
|
||||
self.name = self.__class__.__name__
|
||||
logger.info(f"初始化检测器: {self.name}")
|
||||
|
||||
@abstractmethod
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
pass
|
||||
|
||||
def _log_info(self, message):
|
||||
"""
|
||||
记录信息日志
|
||||
|
||||
:param message: 消息
|
||||
"""
|
||||
logger.info(f"[{self.name}] {message}")
|
||||
|
||||
def _log_warning(self, message):
|
||||
"""
|
||||
记录警告日志
|
||||
|
||||
:param message: 消息
|
||||
"""
|
||||
logger.warning(f"[{self.name}] {message}")
|
||||
|
||||
def _log_error(self, message):
|
||||
"""
|
||||
记录错误日志
|
||||
|
||||
:param message: 消息
|
||||
"""
|
||||
logger.error(f"[{self.name}] {message}")
|
||||
|
||||
def _handle_exception(self, e, domain):
|
||||
"""
|
||||
处理异常
|
||||
|
||||
:param e: 异常
|
||||
:param domain: 域名
|
||||
:return: dict - 错误结果
|
||||
"""
|
||||
self._log_error(f"检测域名 {domain} 出错: {e}")
|
||||
return {'error': str(e)}
|
||||
130
domainCheck/app/detectors/chinaz_detector.py
Normal file
130
domainCheck/app/detectors/chinaz_detector.py
Normal file
@@ -0,0 +1,130 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :chinaz_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:01
|
||||
@explain : 站长之家检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class ChinazDetector(BaseDetector):
|
||||
"""
|
||||
站长之家检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化站长之家检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.url = 'https://seo.chinaz.com'
|
||||
self.query_url = 'https://seo.chinaz.com/{domain}'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.query_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取标题
|
||||
title = self._extract_title(content)
|
||||
|
||||
# 提取网站分类
|
||||
category = self._extract_category(content)
|
||||
|
||||
# 检查是否包含敏感词
|
||||
has_sensitive = self._check_sensitive(title, category)
|
||||
|
||||
return {
|
||||
'title': title,
|
||||
'category': category,
|
||||
'has_sensitive': has_sensitive
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"站长之家查询失败: {response.status_code}")
|
||||
return {'title': '', 'category': '', 'has_sensitive': False}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def _extract_title(self, content):
|
||||
"""
|
||||
提取标题
|
||||
|
||||
:param content: 页面内容
|
||||
:return: str - 标题
|
||||
"""
|
||||
try:
|
||||
pattern = r'<title>(.*?)</title>'
|
||||
match = re.search(pattern, content)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return ''
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_title')
|
||||
return ''
|
||||
|
||||
def _extract_category(self, content):
|
||||
"""
|
||||
提取网站分类
|
||||
|
||||
:param content: 页面内容
|
||||
:return: str - 分类
|
||||
"""
|
||||
try:
|
||||
# 这里需要根据实际页面结构调整正则表达式
|
||||
pattern = r'网站分类:<a[^>]+>(.*?)</a>'
|
||||
match = re.search(pattern, content)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return ''
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_category')
|
||||
return ''
|
||||
|
||||
def _check_sensitive(self, title, category):
|
||||
"""
|
||||
检查是否包含敏感词
|
||||
|
||||
:param title: 标题
|
||||
:param category: 分类
|
||||
:return: bool - 是否包含敏感词
|
||||
"""
|
||||
# 敏感分类
|
||||
sensitive_categories = ['视频电影', '体育运动', '常用查询']
|
||||
|
||||
# 敏感词
|
||||
sensitive_words = ['色情', '赌博', '博彩', '毒品', '暴力', '诈骗']
|
||||
|
||||
# 检查分类
|
||||
for cat in sensitive_categories:
|
||||
if cat in category:
|
||||
return True
|
||||
|
||||
# 检查标题
|
||||
for word in sensitive_words:
|
||||
if word in title:
|
||||
return True
|
||||
|
||||
return False
|
||||
80
domainCheck/app/detectors/google_detector.py
Normal file
80
domainCheck/app/detectors/google_detector.py
Normal file
@@ -0,0 +1,80 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :google_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:00
|
||||
@explain : Google检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class GoogleDetector(BaseDetector):
|
||||
"""
|
||||
Google检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化Google检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.site_url = 'https://www.google.com/search'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 检查Google site
|
||||
site_result = self.check_site(domain)
|
||||
|
||||
return {
|
||||
'site': site_result
|
||||
}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def check_site(self, domain):
|
||||
"""
|
||||
检查Google site收录
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
params = {
|
||||
'q': f'site:{domain}',
|
||||
'num': '50'
|
||||
}
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(self.site_url, params=params, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 检查是否有收录
|
||||
has_收录 = 'No results found for' not in content
|
||||
|
||||
return {
|
||||
'has_收录': has_收录
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"Google site查询失败: {response.status_code}")
|
||||
return {'has_收录': False}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_收录': False}
|
||||
228
domainCheck/app/detectors/jucha_detector.py
Normal file
228
domainCheck/app/detectors/jucha_detector.py
Normal file
@@ -0,0 +1,228 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :jucha_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:04
|
||||
@explain : 聚查检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class JuchaDetector(BaseDetector):
|
||||
"""
|
||||
聚查检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化聚查检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.url = 'https://www.jucha.com'
|
||||
self.whois_url = 'https://www.jucha.com/whois/{domain}'
|
||||
self.beian_url = 'https://www.jucha.com/beian/{domain}'
|
||||
self.intercept_url = 'https://www.jucha.com/intercept/{domain}'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 检查WHOIS
|
||||
whois_result = self.check_whois(domain)
|
||||
|
||||
# 检查备案
|
||||
beian_result = self.check_beian(domain)
|
||||
|
||||
# 检查拦截
|
||||
intercept_result = self.check_intercept(domain)
|
||||
|
||||
return {
|
||||
'whois': whois_result,
|
||||
'beian': beian_result,
|
||||
'intercept': intercept_result
|
||||
}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def check_whois(self, domain):
|
||||
"""
|
||||
检查WHOIS
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.whois_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取WHOIS信息
|
||||
whois_info = self._extract_whois_info(content)
|
||||
|
||||
return whois_info
|
||||
else:
|
||||
self._log_warning(f"聚查WHOIS查询失败: {response.status_code}")
|
||||
return {'status': ''}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'status': ''}
|
||||
|
||||
def check_beian(self, domain):
|
||||
"""
|
||||
检查备案
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.beian_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取备案信息
|
||||
beian_info = self._extract_beian_info(content)
|
||||
|
||||
return beian_info
|
||||
else:
|
||||
self._log_warning(f"聚查备案查询失败: {response.status_code}")
|
||||
return {'has_beian': False, 'beian_year': '', 'is_enterprise': False, 'beian_match': False}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_beian': False, 'beian_year': '', 'is_enterprise': False, 'beian_match': False}
|
||||
|
||||
def check_intercept(self, domain):
|
||||
"""
|
||||
检查拦截
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.intercept_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 检查是否被拦截
|
||||
is_normal = self._check_intercept_status(content)
|
||||
|
||||
return {
|
||||
'normal': is_normal
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"聚查拦截查询失败: {response.status_code}")
|
||||
return {'normal': False}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'normal': False}
|
||||
|
||||
def _extract_whois_info(self, content):
|
||||
"""
|
||||
提取WHOIS信息
|
||||
|
||||
:param content: 页面内容
|
||||
:return: dict - WHOIS信息
|
||||
"""
|
||||
try:
|
||||
# 提取状态信息
|
||||
pattern = r'域名状态:<span[^>]+>(.*?)</span>'
|
||||
match = re.search(pattern, content)
|
||||
status = match.group(1).strip() if match else ''
|
||||
|
||||
return {
|
||||
'status': status
|
||||
}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_whois_info')
|
||||
return {'status': ''}
|
||||
|
||||
def _extract_beian_info(self, content):
|
||||
"""
|
||||
提取备案信息
|
||||
|
||||
:param content: 页面内容
|
||||
:return: dict - 备案信息
|
||||
"""
|
||||
try:
|
||||
# 检查是否有备案
|
||||
has_beian = '备案信息' in content
|
||||
|
||||
# 提取备案年份
|
||||
beian_year = ''
|
||||
pattern = r'审核时间:(\d{4})-\d{2}-\d{2}'
|
||||
match = re.search(pattern, content)
|
||||
if match:
|
||||
beian_year = match.group(1)
|
||||
|
||||
# 提取单位性质
|
||||
is_enterprise = '企业' in content
|
||||
|
||||
# 检查首网址和备案网址是否一致
|
||||
beian_match = '网站首页网址' in content
|
||||
|
||||
return {
|
||||
'has_beian': has_beian,
|
||||
'beian_year': beian_year,
|
||||
'is_enterprise': is_enterprise,
|
||||
'beian_match': beian_match
|
||||
}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_beian_info')
|
||||
return {'has_beian': False, 'beian_year': '', 'is_enterprise': False, 'beian_match': False}
|
||||
|
||||
def _check_intercept_status(self, content):
|
||||
"""
|
||||
检查拦截状态
|
||||
|
||||
:param content: 页面内容
|
||||
:return: bool - 是否正常
|
||||
"""
|
||||
try:
|
||||
# 检查是否包含正常标识
|
||||
if '正常' in content:
|
||||
return True
|
||||
|
||||
# 检查是否包含拦截标识
|
||||
if '拦截' in content:
|
||||
return False
|
||||
|
||||
return False
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'check_intercept_status')
|
||||
return False
|
||||
214
domainCheck/app/detectors/juziseo_detector.py
Normal file
214
domainCheck/app/detectors/juziseo_detector.py
Normal file
@@ -0,0 +1,214 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :juziseo_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:03
|
||||
@explain : 桔子SEO检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class JuziseoDetector(BaseDetector):
|
||||
"""
|
||||
桔子SEO检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化桔子SEO检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.url = 'https://seo.juziseo.com'
|
||||
self.history_url = 'https://seo.juziseo.com/history/{domain}'
|
||||
self.backlink_url = 'https://seo.juziseo.com/backlink/{domain}'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 检查历史信息
|
||||
history_result = self.check_history(domain)
|
||||
|
||||
# 检查外链
|
||||
backlink_result = self.check_backlink(domain)
|
||||
|
||||
return {
|
||||
'history': history_result,
|
||||
'backlink': backlink_result
|
||||
}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def check_history(self, domain):
|
||||
"""
|
||||
检查历史信息
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.history_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取历史信息
|
||||
history_info = self._extract_history_info(content)
|
||||
|
||||
# 检查是否包含敏感词
|
||||
has_sensitive = self._check_sensitive(history_info)
|
||||
|
||||
# 检查是否有百度历史收录
|
||||
has_baidu_history = '百度历史收录' in content
|
||||
|
||||
# 检查是否有子域名
|
||||
has_subdomains = '子域名' in content
|
||||
|
||||
# 检查是否为简体中文
|
||||
is_simplified = self._check_simplified(content)
|
||||
|
||||
return {
|
||||
'has_sensitive': has_sensitive,
|
||||
'has_baidu_history': has_baidu_history,
|
||||
'has_subdomains': has_subdomains,
|
||||
'is_simplified': is_simplified
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"桔子SEO历史查询失败: {response.status_code}")
|
||||
return {'has_sensitive': False, 'has_baidu_history': False, 'has_subdomains': False, 'is_simplified': True}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_sensitive': False, 'has_baidu_history': False, 'has_subdomains': False, 'is_simplified': True}
|
||||
|
||||
def check_backlink(self, domain):
|
||||
"""
|
||||
检查外链
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 构建查询URL
|
||||
query_url = self.backlink_url.format(domain=domain)
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(query_url, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 检查是否包含敏感词
|
||||
has_sensitive = self._check_backlink_sensitive(content)
|
||||
|
||||
# 检查是否有子域名
|
||||
has_subdomains = '子域名' in content
|
||||
|
||||
return {
|
||||
'has_sensitive': has_sensitive,
|
||||
'has_subdomains': has_subdomains
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"桔子SEO外链查询失败: {response.status_code}")
|
||||
return {'has_sensitive': False, 'has_subdomains': False}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_sensitive': False, 'has_subdomains': False}
|
||||
|
||||
def _extract_history_info(self, content):
|
||||
"""
|
||||
提取历史信息
|
||||
|
||||
:param content: 页面内容
|
||||
:return: str - 历史信息
|
||||
"""
|
||||
try:
|
||||
# 这里需要根据实际页面结构调整正则表达式
|
||||
pattern = r'<div class="history-info">(.*?)</div>'
|
||||
match = re.search(pattern, content, re.DOTALL)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return ''
|
||||
except Exception as e:
|
||||
self._handle_exception(e, 'extract_history_info')
|
||||
return ''
|
||||
|
||||
def _check_sensitive(self, history_info):
|
||||
"""
|
||||
检查是否包含敏感词
|
||||
|
||||
:param history_info: 历史信息
|
||||
:return: bool - 是否包含敏感词
|
||||
"""
|
||||
# 敏感词
|
||||
sensitive_words = [
|
||||
'色情', '赌博', '博彩', '毒品', '暴力', '诈骗',
|
||||
'足球', '直播', '证券', '配资', '软件',
|
||||
'体育', '商行', '下载', '影视', '网络',
|
||||
'计算', 'app', 'HTML SiteMap', '模拟器', '传媒',
|
||||
'二次元', '成人', '米乐', '小说', '凯发',
|
||||
'人才', '华体', '娱乐', '开户'
|
||||
]
|
||||
|
||||
for word in sensitive_words:
|
||||
if word in history_info:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _check_simplified(self, content):
|
||||
"""
|
||||
检查是否为简体中文
|
||||
|
||||
:param content: 页面内容
|
||||
:return: bool - 是否为简体中文
|
||||
"""
|
||||
# 检查是否包含简体中文标识
|
||||
if '简体中文' in content:
|
||||
return True
|
||||
|
||||
# 检查是否包含繁体中文标识
|
||||
if '繁体中文' in content:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def _check_backlink_sensitive(self, content):
|
||||
"""
|
||||
检查外链是否包含敏感词
|
||||
|
||||
:param content: 页面内容
|
||||
:return: bool - 是否包含敏感词
|
||||
"""
|
||||
# 敏感词
|
||||
sensitive_words = [
|
||||
'内幕', '猛料', '精料', '高手', '绝杀',
|
||||
'权威', '澳门', '色情', '赌博', '博彩'
|
||||
]
|
||||
|
||||
for word in sensitive_words:
|
||||
if word in content:
|
||||
return True
|
||||
|
||||
return False
|
||||
107
domainCheck/app/detectors/qihu360_detector.py
Normal file
107
domainCheck/app/detectors/qihu360_detector.py
Normal file
@@ -0,0 +1,107 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :qihu360_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:59
|
||||
@explain : 360检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
import re
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class Qihu360Detector(BaseDetector):
|
||||
"""
|
||||
360检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化360检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.site_url = 'https://www.so.com/s'
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 检查360 site
|
||||
site_result = self.check_site(domain)
|
||||
|
||||
return {
|
||||
'site': site_result
|
||||
}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def check_site(self, domain):
|
||||
"""
|
||||
检查360 site收录
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
params = {
|
||||
'q': f'site:{domain}',
|
||||
'pn': '1',
|
||||
'rn': '50'
|
||||
}
|
||||
|
||||
headers = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'
|
||||
}
|
||||
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(self.site_url, params=params, headers=headers, impersonate='chrome', timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
content = response.text
|
||||
|
||||
# 提取子域名
|
||||
subdomains = self._extract_subdomains(content, domain)
|
||||
|
||||
# 检查是否有收录
|
||||
has_收录 = '没有找到相关结果' not in content
|
||||
|
||||
return {
|
||||
'has_收录': has_收录,
|
||||
'subdomains': subdomains
|
||||
}
|
||||
else:
|
||||
self._log_warning(f"360 site查询失败: {response.status_code}")
|
||||
return {'has_收录': False, 'subdomains': []}
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return {'has_收录': False, 'subdomains': []}
|
||||
|
||||
def _extract_subdomains(self, content, domain):
|
||||
"""
|
||||
提取子域名
|
||||
|
||||
:param content: 搜索结果内容
|
||||
:param domain: 主域名
|
||||
:return: list - 子域名列表
|
||||
"""
|
||||
try:
|
||||
# 提取所有包含域名的链接
|
||||
pattern = r'https?://([a-zA-Z0-9-]+)\.' + re.escape(domain)
|
||||
matches = re.findall(pattern, content)
|
||||
|
||||
# 去重并过滤空值
|
||||
subdomains = list(set(matches))
|
||||
subdomains = [sub for sub in subdomains if sub]
|
||||
|
||||
return subdomains
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return []
|
||||
128
domainCheck/app/detectors/rdap_detector.py
Normal file
128
domainCheck/app/detectors/rdap_detector.py
Normal file
@@ -0,0 +1,128 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :rdap_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:56
|
||||
@explain : RDAP检测器
|
||||
'''
|
||||
|
||||
import requests
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class RDAPDetector(BaseDetector):
|
||||
"""
|
||||
RDAP检测器
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化RDAP检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.rdap_urls = {
|
||||
'com': 'https://rdap.verisign.com/com/v1/domain/',
|
||||
'net': 'https://rdap.verisign.com/net/v1/domain/'
|
||||
}
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
# 提取顶级域名
|
||||
tld = domain.split('.')[-1]
|
||||
if tld not in self.rdap_urls:
|
||||
return {'error': '不支持的顶级域名'}
|
||||
|
||||
# 构建RDAP查询URL
|
||||
url = self.rdap_urls[tld] + domain
|
||||
|
||||
# 发送请求
|
||||
response = requests.get(url, timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
return self._parse_rdap_response(data)
|
||||
elif response.status_code == 404:
|
||||
return {'status': 'available'}
|
||||
else:
|
||||
return {'error': f'RDAP查询失败: {response.status_code}'}
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def check_register_status(self, domain):
|
||||
"""
|
||||
检查注册状态
|
||||
|
||||
:param domain: 域名
|
||||
:return: int - 注册状态码
|
||||
"""
|
||||
try:
|
||||
result = self.check_domain(domain)
|
||||
|
||||
if 'error' in result:
|
||||
return 9 # 状态未知
|
||||
|
||||
if result.get('status') == 'available':
|
||||
return 2 # 可注册
|
||||
|
||||
# 检查域名状态
|
||||
statuses = result.get('status', [])
|
||||
if 'clientHold' in statuses:
|
||||
return 7 # clientHold
|
||||
elif 'serverHold' in statuses:
|
||||
return 8 # serverHold
|
||||
elif 'autoRenewPeriod' in statuses:
|
||||
return 4 # 宽限期
|
||||
elif 'redemptionPeriod' in statuses:
|
||||
return 5 # 赎回期
|
||||
elif 'pendingDelete' in statuses:
|
||||
return 6 # 删除期
|
||||
else:
|
||||
return 3 # 已注册
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return 10 # 检测失败
|
||||
|
||||
def _parse_rdap_response(self, data):
|
||||
"""
|
||||
解析RDAP响应
|
||||
|
||||
:param data: RDAP响应数据
|
||||
:return: dict - 解析结果
|
||||
"""
|
||||
result = {
|
||||
'status': 'registered',
|
||||
'domain': data.get('ldhName'),
|
||||
'statuses': data.get('status', []),
|
||||
'registrar': None,
|
||||
'creation_date': None,
|
||||
'expiration_date': None,
|
||||
'last_update': None
|
||||
}
|
||||
|
||||
# 解析注册商信息
|
||||
for entity in data.get('entities', []):
|
||||
if 'registrar' in entity.get('roles', []):
|
||||
result['registrar'] = entity.get('vcardArray', [[], []])[1][1][3]
|
||||
break
|
||||
|
||||
# 解析时间信息
|
||||
for event in data.get('events', []):
|
||||
event_action = event.get('eventAction')
|
||||
event_date = event.get('eventDate')
|
||||
|
||||
if event_action == 'registration':
|
||||
result['creation_date'] = event_date
|
||||
elif event_action == 'expiration':
|
||||
result['expiration_date'] = event_date
|
||||
elif event_action == 'last update':
|
||||
result['last_update'] = event_date
|
||||
|
||||
return result
|
||||
562
domainCheck/app/detectors/wayback_detector.py
Normal file
562
domainCheck/app/detectors/wayback_detector.py
Normal file
@@ -0,0 +1,562 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :wayback_detector.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:57
|
||||
@explain : Wayback检测器
|
||||
'''
|
||||
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
import zlib
|
||||
from base64 import b64decode, b64encode
|
||||
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
|
||||
|
||||
import redis
|
||||
import requests
|
||||
from requests.adapters import HTTPAdapter
|
||||
from urllib3.util.retry import Retry
|
||||
|
||||
from app.config import config
|
||||
from app.detectors.base import BaseDetector
|
||||
|
||||
|
||||
class WaybackDetector(BaseDetector):
|
||||
"""
|
||||
Wayback检测器
|
||||
"""
|
||||
|
||||
TITLE_PATTERN = re.compile(r'<title[^>]*>(.*?)</title>', re.IGNORECASE | re.DOTALL)
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化Wayback检测器
|
||||
"""
|
||||
super().__init__()
|
||||
self.cdx_api_url = 'https://web.archive.org/cdx/search/cdx'
|
||||
# 使用 id_ 直接取快照内容,避免回放页面额外壳层干扰正文匹配。
|
||||
self.snapshot_url = 'https://web.archive.org/web/{timestamp}id_/{domain}'
|
||||
self._timestamp_cache = {}
|
||||
self._title_cache = {}
|
||||
self._cache_lock = threading.Lock()
|
||||
self.session = self._build_session()
|
||||
self.redis_client = self._build_redis_client()
|
||||
|
||||
def _build_session(self):
|
||||
session = requests.Session()
|
||||
retry = Retry(
|
||||
total=max(0, config.WAYBACK_RETRY_COUNT),
|
||||
backoff_factor=0.5,
|
||||
status_forcelist=(429, 500, 502, 503, 504),
|
||||
allowed_methods=frozenset(["GET"]),
|
||||
raise_on_status=False,
|
||||
)
|
||||
adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=10)
|
||||
session.mount('http://', adapter)
|
||||
session.mount('https://', adapter)
|
||||
session.headers.update({
|
||||
'User-Agent': config.WAYBACK_USER_AGENT,
|
||||
'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
|
||||
})
|
||||
return session
|
||||
|
||||
def _build_redis_client(self):
|
||||
try:
|
||||
client = redis.Redis(
|
||||
host=config.REDIS_HOST,
|
||||
port=config.REDIS_PORT,
|
||||
password=config.REDIS_PASSWORD,
|
||||
db=config.REDIS_DB,
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=5,
|
||||
socket_timeout=5,
|
||||
)
|
||||
client.ping()
|
||||
return client
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _title_cache_key(self, domain, timestamp):
|
||||
return f"domain_tool:wayback_title:{domain}:{timestamp}"
|
||||
|
||||
def _timestamp_cache_key(self, domain):
|
||||
return f"domain_tool:wayback_timestamps:{domain}"
|
||||
|
||||
def _record_cache_key(self, domain):
|
||||
return f"domain_tool:wayback_records:{domain}"
|
||||
|
||||
def _normalize_title(self, title):
|
||||
normalized = html.unescape(title or '')
|
||||
normalized = re.sub(r'\s+', ' ', normalized, flags=re.DOTALL).strip().lower()
|
||||
return normalized
|
||||
|
||||
def _extract_title(self, content):
|
||||
match = self.TITLE_PATTERN.search(content or '')
|
||||
if not match:
|
||||
return ''
|
||||
return html.unescape(match.group(1)).strip()
|
||||
|
||||
def _load_cached_title(self, domain, timestamp):
|
||||
cache_key = self._title_cache_key(domain, timestamp)
|
||||
with self._cache_lock:
|
||||
if cache_key in self._title_cache:
|
||||
return self._title_cache[cache_key]
|
||||
if self.redis_client:
|
||||
try:
|
||||
raw_value = self.redis_client.get(cache_key)
|
||||
if raw_value:
|
||||
data = json.loads(raw_value)
|
||||
with self._cache_lock:
|
||||
self._title_cache[cache_key] = data
|
||||
return data
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
def _save_cached_title(self, domain, timestamp, data):
|
||||
cache_key = self._title_cache_key(domain, timestamp)
|
||||
with self._cache_lock:
|
||||
self._title_cache[cache_key] = data
|
||||
if self.redis_client:
|
||||
try:
|
||||
self.redis_client.set(
|
||||
cache_key,
|
||||
json.dumps(data, ensure_ascii=False),
|
||||
ex=max(0, config.WAYBACK_TITLE_CACHE_TTL) or None,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _load_cached_timestamps(self, domain):
|
||||
if domain in self._timestamp_cache:
|
||||
return list(self._timestamp_cache[domain])
|
||||
if not self.redis_client:
|
||||
return None
|
||||
try:
|
||||
raw_value = self.redis_client.get(self._timestamp_cache_key(domain))
|
||||
if not raw_value:
|
||||
return None
|
||||
compressed = b64decode(raw_value.encode('ascii'))
|
||||
timestamps = json.loads(zlib.decompress(compressed).decode('utf-8'))
|
||||
if isinstance(timestamps, list):
|
||||
self._timestamp_cache[domain] = tuple(timestamps)
|
||||
return list(timestamps)
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
def _save_cached_timestamps(self, domain, timestamps):
|
||||
self._timestamp_cache[domain] = tuple(timestamps)
|
||||
if not self.redis_client:
|
||||
return
|
||||
try:
|
||||
payload = json.dumps(timestamps, separators=(',', ':')).encode('utf-8')
|
||||
compressed = zlib.compress(payload, level=6)
|
||||
self.redis_client.set(
|
||||
self._timestamp_cache_key(domain),
|
||||
b64encode(compressed).decode('ascii'),
|
||||
ex=max(0, config.WAYBACK_TIMESTAMP_CACHE_TTL) or None,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _load_cached_records(self, domain):
|
||||
cache_key = self._record_cache_key(domain)
|
||||
with self._cache_lock:
|
||||
if cache_key in self._timestamp_cache:
|
||||
return [
|
||||
{'timestamp': item[0], 'digest': item[1]}
|
||||
for item in self._timestamp_cache[cache_key]
|
||||
]
|
||||
if not self.redis_client:
|
||||
return None
|
||||
try:
|
||||
raw_value = self.redis_client.get(cache_key)
|
||||
if not raw_value:
|
||||
return None
|
||||
compressed = b64decode(raw_value.encode('ascii'))
|
||||
records = json.loads(zlib.decompress(compressed).decode('utf-8'))
|
||||
if isinstance(records, list):
|
||||
normalized = tuple(
|
||||
(item.get('timestamp', ''), item.get('digest', ''))
|
||||
for item in records if isinstance(item, dict)
|
||||
)
|
||||
with self._cache_lock:
|
||||
self._timestamp_cache[cache_key] = normalized
|
||||
return [
|
||||
{'timestamp': item[0], 'digest': item[1]}
|
||||
for item in normalized if item[0]
|
||||
]
|
||||
except Exception:
|
||||
return None
|
||||
return None
|
||||
|
||||
def _save_cached_records(self, domain, records):
|
||||
cache_key = self._record_cache_key(domain)
|
||||
normalized = tuple(
|
||||
(item.get('timestamp', ''), item.get('digest', ''))
|
||||
for item in (records or [])
|
||||
if item and item.get('timestamp')
|
||||
)
|
||||
with self._cache_lock:
|
||||
self._timestamp_cache[cache_key] = normalized
|
||||
if not self.redis_client:
|
||||
return
|
||||
try:
|
||||
payload = json.dumps(
|
||||
[{'timestamp': item[0], 'digest': item[1]} for item in normalized],
|
||||
separators=(',', ':')
|
||||
).encode('utf-8')
|
||||
compressed = zlib.compress(payload, level=6)
|
||||
self.redis_client.set(
|
||||
cache_key,
|
||||
b64encode(compressed).decode('ascii'),
|
||||
ex=max(0, config.WAYBACK_TIMESTAMP_CACHE_TTL) or None,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _fetch_cdx_records(self, domain, limit=None, fast_latest=False):
|
||||
response = None
|
||||
try:
|
||||
params = {
|
||||
'url': domain,
|
||||
'output': 'txt',
|
||||
'fl': 'timestamp,digest',
|
||||
'filter': ['statuscode:200', 'mimetype:text/html'],
|
||||
}
|
||||
if limit is not None:
|
||||
params['limit'] = str(limit)
|
||||
if fast_latest:
|
||||
params['fastLatest'] = 'true'
|
||||
response = self.session.get(
|
||||
self.cdx_api_url,
|
||||
params=params,
|
||||
timeout=config.WAYBACK_CDX_TIMEOUT,
|
||||
stream=True,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
self._log_warning(f"获取快照记录失败: {response.status_code}")
|
||||
return []
|
||||
records = []
|
||||
seen = set()
|
||||
for raw_line in response.iter_lines(decode_unicode=True):
|
||||
line = (raw_line or '').strip()
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split()
|
||||
timestamp = parts[0].strip() if parts else ''
|
||||
digest = parts[1].strip() if len(parts) > 1 else ''
|
||||
if not timestamp or timestamp in seen:
|
||||
continue
|
||||
seen.add(timestamp)
|
||||
records.append({'timestamp': timestamp, 'digest': digest})
|
||||
return records
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return []
|
||||
finally:
|
||||
try:
|
||||
if response is not None:
|
||||
response.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def get_latest_snapshot_record(self, domain):
|
||||
records = self._fetch_cdx_records(domain, limit=-1, fast_latest=True)
|
||||
return records[0] if records else None
|
||||
|
||||
def get_snapshot_records(self, domain):
|
||||
cached_records = self._load_cached_records(domain)
|
||||
if cached_records is not None:
|
||||
return cached_records
|
||||
records = self._fetch_cdx_records(domain)
|
||||
if records:
|
||||
self._save_cached_records(domain, records)
|
||||
self._save_cached_timestamps(domain, [item['timestamp'] for item in records])
|
||||
return records
|
||||
|
||||
def check_domain(self, domain):
|
||||
"""
|
||||
检测域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: dict - 检测结果
|
||||
"""
|
||||
try:
|
||||
return self.scan_snapshots(domain)
|
||||
except Exception as e:
|
||||
return self._handle_exception(e, domain)
|
||||
|
||||
def get_snapshot_timestamps(self, domain):
|
||||
cached_records = self._load_cached_records(domain)
|
||||
if cached_records is not None:
|
||||
return [item['timestamp'] for item in cached_records if item.get('timestamp')]
|
||||
cached_timestamps = self._load_cached_timestamps(domain)
|
||||
if cached_timestamps is not None:
|
||||
return cached_timestamps
|
||||
records = self.get_snapshot_records(domain)
|
||||
return [item['timestamp'] for item in records if item.get('timestamp')]
|
||||
|
||||
def _fetch_snapshot_title(self, domain, timestamp):
|
||||
cached = self._load_cached_title(domain, timestamp)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
snapshot_url = self.snapshot_url.format(timestamp=timestamp, domain=domain)
|
||||
response = None
|
||||
try:
|
||||
response = self.session.get(
|
||||
snapshot_url,
|
||||
timeout=config.WAYBACK_SNAPSHOT_TIMEOUT,
|
||||
stream=True,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
data = {'timestamp': timestamp, 'title': '', 'ok': False}
|
||||
self._save_cached_title(domain, timestamp, data)
|
||||
return data
|
||||
|
||||
content_type = (response.headers.get('Content-Type') or '').lower()
|
||||
if content_type and 'text/html' not in content_type and 'application/xhtml+xml' not in content_type:
|
||||
data = {'timestamp': timestamp, 'title': '', 'ok': False}
|
||||
self._save_cached_title(domain, timestamp, data)
|
||||
return data
|
||||
|
||||
chunks = []
|
||||
total_bytes = 0
|
||||
found_title = False
|
||||
for chunk in response.iter_content(chunk_size=4096, decode_unicode=True):
|
||||
if not chunk:
|
||||
continue
|
||||
chunks.append(chunk)
|
||||
total_bytes += len(chunk.encode('utf-8', errors='ignore'))
|
||||
current_text = ''.join(chunks)
|
||||
if '</title>' in current_text.lower():
|
||||
found_title = True
|
||||
break
|
||||
if total_bytes >= config.WAYBACK_TITLE_MAX_BYTES:
|
||||
break
|
||||
|
||||
content = ''.join(chunks)
|
||||
title = self._extract_title(content) if found_title or content else ''
|
||||
data = {'timestamp': timestamp, 'title': title, 'ok': True}
|
||||
self._save_cached_title(domain, timestamp, data)
|
||||
return data
|
||||
except Exception:
|
||||
data = {'timestamp': timestamp, 'title': '', 'ok': False}
|
||||
self._save_cached_title(domain, timestamp, data)
|
||||
return data
|
||||
finally:
|
||||
try:
|
||||
if response is not None:
|
||||
response.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def get_snapshot_years(self, domain):
|
||||
"""
|
||||
获取快照年份
|
||||
|
||||
:param domain: 域名
|
||||
:return: list - 快照年份列表
|
||||
"""
|
||||
try:
|
||||
years = set()
|
||||
for timestamp in self.get_snapshot_timestamps(domain):
|
||||
if len(timestamp) >= 4:
|
||||
years.add(int(timestamp[:4]))
|
||||
return sorted(years)
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return []
|
||||
|
||||
def has_sensitive_content(self, domain):
|
||||
"""
|
||||
检查是否包含敏感内容
|
||||
|
||||
:param domain: 域名
|
||||
:return: bool - 是否包含敏感内容
|
||||
"""
|
||||
try:
|
||||
result = self.scan_snapshots(domain)
|
||||
return result.get('has_sensitive_content', False)
|
||||
except Exception as e:
|
||||
self._handle_exception(e, domain)
|
||||
return False
|
||||
|
||||
def scan_snapshots(self, domain, sensitive_words=None, stop_on_first_hit=True):
|
||||
sensitive_words = sensitive_words or config.load_sensitive_words()
|
||||
latest_record = self.get_latest_snapshot_record(domain)
|
||||
latest_timestamp = (latest_record or {}).get('timestamp')
|
||||
latest_digest = (latest_record or {}).get('digest', '')
|
||||
matched_word = None
|
||||
matched_timestamp = None
|
||||
matched_title = None
|
||||
fetched_snapshot_count = 0
|
||||
failed_snapshot_count = 0
|
||||
unique_title_count = 0
|
||||
duplicate_title_skipped = 0
|
||||
digest_duplicate_skipped = 0
|
||||
started_at = time.time()
|
||||
progress_interval = max(1, config.WAYBACK_PROGRESS_INTERVAL)
|
||||
title_seen = set()
|
||||
digest_seen = set()
|
||||
checked_snapshot_count = 0
|
||||
domain_concurrency = max(1, config.WAYBACK_DOMAIN_CONCURRENCY)
|
||||
|
||||
if latest_timestamp:
|
||||
latest_result = self._fetch_snapshot_title(domain, latest_timestamp)
|
||||
checked_snapshot_count = 1
|
||||
if latest_result and latest_result.get('ok'):
|
||||
fetched_snapshot_count = 1
|
||||
title = latest_result.get('title', '')
|
||||
normalized_title = self._normalize_title(title)
|
||||
if latest_digest:
|
||||
digest_seen.add(latest_digest)
|
||||
if normalized_title:
|
||||
title_seen.add(normalized_title)
|
||||
unique_title_count = 1
|
||||
matched_word = self._find_sensitive_word(title, sensitive_words)
|
||||
if matched_word and stop_on_first_hit:
|
||||
matched_timestamp = latest_timestamp
|
||||
matched_title = title
|
||||
return {
|
||||
'snapshot_years': [int(latest_timestamp[:4])] if len(latest_timestamp) >= 4 else [],
|
||||
'has_sensitive_content': True,
|
||||
'matched_word': matched_word,
|
||||
'matched_timestamp': matched_timestamp,
|
||||
'matched_title': matched_title,
|
||||
'backlink_count': 0,
|
||||
'backlink_count_gt_10': False,
|
||||
'checked_snapshot_count': checked_snapshot_count,
|
||||
'fetched_snapshot_count': fetched_snapshot_count,
|
||||
'failed_snapshot_count': failed_snapshot_count,
|
||||
'unique_title_count': unique_title_count,
|
||||
'duplicate_title_skipped': duplicate_title_skipped,
|
||||
'digest_duplicate_skipped': digest_duplicate_skipped,
|
||||
'elapsed_seconds': round(time.time() - started_at, 2),
|
||||
}
|
||||
else:
|
||||
failed_snapshot_count += 1
|
||||
|
||||
records = sorted(self.get_snapshot_records(domain), key=lambda item: item.get('timestamp', ''), reverse=True)
|
||||
years = sorted({int(item['timestamp'][:4]) for item in records if len(item.get('timestamp', '')) >= 4})
|
||||
checked_snapshot_count = len(records)
|
||||
pending_records = []
|
||||
for item in records:
|
||||
timestamp = item.get('timestamp', '')
|
||||
digest = item.get('digest', '')
|
||||
if not timestamp:
|
||||
continue
|
||||
if latest_timestamp and timestamp == latest_timestamp:
|
||||
continue
|
||||
if digest and digest in digest_seen:
|
||||
digest_duplicate_skipped += 1
|
||||
continue
|
||||
if digest:
|
||||
digest_seen.add(digest)
|
||||
pending_records.append(item)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=domain_concurrency) as executor:
|
||||
pending = {}
|
||||
index = 0
|
||||
finished_count = 1 if latest_timestamp else 0
|
||||
stop_requested = False
|
||||
|
||||
while (index < len(pending_records) or pending) and not stop_requested:
|
||||
while index < len(pending_records) and len(pending) < domain_concurrency and not stop_requested:
|
||||
timestamp = pending_records[index]['timestamp']
|
||||
future = executor.submit(self._fetch_snapshot_title, domain, timestamp)
|
||||
pending[future] = timestamp
|
||||
index += 1
|
||||
|
||||
if not pending:
|
||||
break
|
||||
|
||||
done, _ = wait(list(pending.keys()), return_when=FIRST_COMPLETED)
|
||||
for future in done:
|
||||
timestamp = pending.pop(future, None)
|
||||
finished_count += 1
|
||||
try:
|
||||
result = future.result()
|
||||
except Exception as e:
|
||||
failed_snapshot_count += 1
|
||||
self._handle_exception(e, domain)
|
||||
continue
|
||||
|
||||
if not result or not result.get('ok'):
|
||||
failed_snapshot_count += 1
|
||||
continue
|
||||
|
||||
fetched_snapshot_count += 1
|
||||
title = result.get('title', '')
|
||||
normalized_title = self._normalize_title(title)
|
||||
if normalized_title:
|
||||
if normalized_title in title_seen:
|
||||
duplicate_title_skipped += 1
|
||||
else:
|
||||
title_seen.add(normalized_title)
|
||||
unique_title_count += 1
|
||||
matched_word = self._find_sensitive_word(title, sensitive_words)
|
||||
if matched_word:
|
||||
matched_timestamp = timestamp
|
||||
matched_title = title
|
||||
if stop_on_first_hit:
|
||||
stop_requested = True
|
||||
if finished_count % progress_interval == 0:
|
||||
elapsed = round(time.time() - started_at, 2)
|
||||
self._log_info(
|
||||
f"{domain} 时光机进度: {finished_count}/{checked_snapshot_count},成功 {fetched_snapshot_count},失败 {failed_snapshot_count},唯一标题 {unique_title_count},标题重复跳过 {duplicate_title_skipped},digest 重复跳过 {digest_duplicate_skipped},耗时 {elapsed}s"
|
||||
)
|
||||
if config.WAYBACK_REQUEST_DELAY > 0:
|
||||
time.sleep(config.WAYBACK_REQUEST_DELAY)
|
||||
|
||||
if stop_requested:
|
||||
for future in pending:
|
||||
future.cancel()
|
||||
|
||||
return {
|
||||
'snapshot_years': years,
|
||||
'has_sensitive_content': matched_word is not None,
|
||||
'matched_word': matched_word,
|
||||
'matched_timestamp': matched_timestamp,
|
||||
'matched_title': matched_title,
|
||||
'backlink_count': 0,
|
||||
'backlink_count_gt_10': False,
|
||||
'checked_snapshot_count': checked_snapshot_count,
|
||||
'fetched_snapshot_count': fetched_snapshot_count,
|
||||
'failed_snapshot_count': failed_snapshot_count,
|
||||
'unique_title_count': unique_title_count,
|
||||
'duplicate_title_skipped': duplicate_title_skipped,
|
||||
'digest_duplicate_skipped': digest_duplicate_skipped,
|
||||
'elapsed_seconds': round(time.time() - started_at, 2),
|
||||
}
|
||||
|
||||
def _check_sensitive_words(self, content):
|
||||
"""
|
||||
检查敏感词
|
||||
|
||||
:param content: 内容
|
||||
:return: bool - 是否包含敏感词
|
||||
"""
|
||||
return self._find_sensitive_word(content, config.load_sensitive_words()) is not None
|
||||
|
||||
def _find_sensitive_word(self, content, sensitive_words):
|
||||
for word in sensitive_words or []:
|
||||
if word and word in (content or ''):
|
||||
return word
|
||||
return None
|
||||
|
||||
def get_backlink_count(self, domain):
|
||||
"""
|
||||
当前策略仅扫描标题,不再抓取正文,友链数量默认返回 0。
|
||||
"""
|
||||
return 0
|
||||
|
||||
def _count_backlinks(self, content):
|
||||
return 0
|
||||
3
domainCheck/app/domain_suffixes.json
Normal file
3
domainCheck/app/domain_suffixes.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"suffixes": ".com,.net"
|
||||
}
|
||||
68
domainCheck/app/main.py
Normal file
68
domainCheck/app/main.py
Normal file
@@ -0,0 +1,68 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :main.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:09
|
||||
@explain : 系统主入口
|
||||
'''
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# 添加项目根目录到 sys.path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from loguru import logger
|
||||
from PySide6.QtWidgets import QApplication
|
||||
from app.ui.main_window import MainWindow
|
||||
from app.utils.database import Database
|
||||
from app.config import config
|
||||
|
||||
# 配置日志
|
||||
logger.add(
|
||||
os.path.join(config.LOG_DIR, config.LOG_FILE),
|
||||
level=config.LOG_LEVEL,
|
||||
rotation="10 MB",
|
||||
compression="zip"
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
"""
|
||||
主函数
|
||||
"""
|
||||
try:
|
||||
# 初始化数据库连接
|
||||
db = Database(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
|
||||
# 测试数据库连接
|
||||
db.execute("SELECT 1")
|
||||
logger.info("数据库连接成功")
|
||||
|
||||
# 初始化应用程序
|
||||
app = QApplication(sys.argv)
|
||||
window = MainWindow()
|
||||
window.show()
|
||||
|
||||
# 运行应用程序
|
||||
sys.exit(app.exec())
|
||||
except Exception as e:
|
||||
logger.error(f"启动应用程序失败: {e}")
|
||||
sys.exit(1)
|
||||
finally:
|
||||
# 关闭数据库连接
|
||||
if 'db' in locals():
|
||||
db.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
4
domainCheck/app/proxy_config.json
Normal file
4
domainCheck/app/proxy_config.json
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"proxy_enable": false,
|
||||
"proxy_url": ""
|
||||
}
|
||||
15108
domainCheck/app/sdk_leg.js
Normal file
15108
domainCheck/app/sdk_leg.js
Normal file
File diff suppressed because one or more lines are too long
897
domainCheck/app/sdk_leg_env.js
Normal file
897
domainCheck/app/sdk_leg_env.js
Normal file
@@ -0,0 +1,897 @@
|
||||
process_ = process;
|
||||
require_ = require;
|
||||
delete Buffer;
|
||||
// delete process;
|
||||
delete require;
|
||||
delete global;
|
||||
delete module;
|
||||
delete exports;
|
||||
delete __filename;
|
||||
delete __dirname;
|
||||
delete SharedArrayBuffer;
|
||||
|
||||
AsObj = {
|
||||
// print: console.log,
|
||||
print: function () { },
|
||||
// print_:console.log,
|
||||
}
|
||||
|
||||
no_print = ['Boolean','String','parseFloat','Array','Object','prepareStackTrace_'];
|
||||
function watch(object, WatchName) {
|
||||
const handler = {
|
||||
get(target, property, receiver) {
|
||||
if (
|
||||
property !== 'isNaN' &&
|
||||
property !== 'encodeURI' &&
|
||||
property !== "Uint8Array" &&
|
||||
property !== 'undefined' &&
|
||||
property !== 'JSON' &&
|
||||
property !== 'Number' &&
|
||||
!no_print.includes(property) &&
|
||||
property !== Symbol.for('nodejs.util.inspect.custom') &&
|
||||
typeof property !== 'symbol'
|
||||
) {
|
||||
|
||||
if (property === 'global') {
|
||||
return undefined;
|
||||
}
|
||||
if (property === 'Buffer') {
|
||||
return undefined;
|
||||
}
|
||||
if (property === 'process') {
|
||||
return undefined;
|
||||
}
|
||||
if (WatchName === 'config_data') {
|
||||
debugger
|
||||
}
|
||||
if (WatchName.indexOf('.prototype') != -1 && target[property] != undefined) {
|
||||
return Reflect.get(target, property, receiver);
|
||||
}
|
||||
|
||||
AsObj.print(
|
||||
"方法:", "get",
|
||||
"对象:", WatchName,
|
||||
"属性:", property,
|
||||
"属性类型:", typeof property,
|
||||
"属性值:", typeof target[property] == 'object' ? "object" : target[property],
|
||||
"属性值类型:", typeof target[property]
|
||||
);
|
||||
}
|
||||
|
||||
if (WatchName === 'top') {
|
||||
return window;
|
||||
}
|
||||
|
||||
return Reflect.get(target, property, receiver);
|
||||
},
|
||||
|
||||
set(target, property, value, receiver) {
|
||||
if (WatchName.indexOf('.prototype') != -1 && value != undefined) {
|
||||
return Reflect.set(target, property, value, receiver);
|
||||
}
|
||||
AsObj.print(
|
||||
"方法:", "set",
|
||||
"对象:", WatchName,
|
||||
"属性:", property,
|
||||
"属性类型:", typeof property,
|
||||
"属性值:", typeof value == 'object' ? "object" : value,
|
||||
"属性值类型:", typeof target[property]
|
||||
);
|
||||
return Reflect.set(target, property, value, receiver);
|
||||
},
|
||||
// in操作 检测
|
||||
has(target, property) {
|
||||
AsObj.print(
|
||||
"代理对象:", WatchName,
|
||||
"方法:", "has",
|
||||
"检查属性:", property,
|
||||
"结果:", typeof target[property] == 'object' ? "object" : target[property],
|
||||
);
|
||||
return Reflect.has(target, property);
|
||||
},
|
||||
// Object.key 检测
|
||||
ownKeys(target) {
|
||||
AsObj.print(
|
||||
"方法:", "ownKeys",
|
||||
"对象:", target+''
|
||||
);
|
||||
return Reflect.ownKeys(target);
|
||||
}
|
||||
};
|
||||
|
||||
return new Proxy(object, handler);
|
||||
}
|
||||
// function watch(object, WatchName) {
|
||||
// return object
|
||||
// }
|
||||
|
||||
// 保护函数,toString检测
|
||||
const safeFunction = function safeFunction(func) {
|
||||
//处理安全函数
|
||||
Function.prototype.$call = Function.prototype.call;
|
||||
const $toString = Function.toString;
|
||||
const myFunction_toString_symbol = Symbol('('.concat('', ')'));
|
||||
|
||||
const myToString = function myToString() {
|
||||
return typeof this === 'function' && this[myFunction_toString_symbol] || $toString.$call(this);
|
||||
}
|
||||
|
||||
const set_native = function set_native(func, key, value) {
|
||||
Object.defineProperty(func, key, {
|
||||
"enumerable": false,
|
||||
"configurable": true,
|
||||
"writable": true,
|
||||
"value": value
|
||||
});
|
||||
}
|
||||
|
||||
delete Function.prototype['toString'];
|
||||
set_native(Function.prototype, "toString", myToString);
|
||||
set_native(Function.prototype.toString, myFunction_toString_symbol, "function toString() { [native code] }");
|
||||
|
||||
const safe_Function = function safe_Function(func) {
|
||||
set_native(func, myFunction_toString_symbol, "function" + (func.name ? " " + func.name : "") + "() { [native code] }");
|
||||
}
|
||||
|
||||
return safe_Function(func)
|
||||
}
|
||||
|
||||
//创建函数,并代理上
|
||||
const makeFunction = function makeFunction(name) {
|
||||
v_log = AsObj.print;
|
||||
// 使用 Function 保留函数名
|
||||
func = new Function("v_log", `
|
||||
return function ${name}() {
|
||||
v_log('函数${name}传参-->', arguments);
|
||||
};
|
||||
`)(v_log); // 传递 v_log 到动态函数
|
||||
|
||||
safeFunction(func);
|
||||
func = watch(func,`${name}`);
|
||||
func.prototype = watch(func.prototype, `${name}.prototype`);
|
||||
return func;
|
||||
}
|
||||
|
||||
!(function () {
|
||||
"use strict";
|
||||
const $toString = Function.toString;
|
||||
const myFunction_toString_symbol = Symbol('('.concat('', ')_', (Math.random() + '').toString(36)));
|
||||
const mytoString = function () {
|
||||
return typeof this == 'function' && this[myFunction_toString_symbol] || $toString.call(this);
|
||||
};
|
||||
|
||||
function set_native(func, key, value) {
|
||||
Object.defineProperty(func, key, {
|
||||
"enumerable": false,
|
||||
"configurable": true,
|
||||
"writable": true,
|
||||
"value": value
|
||||
})
|
||||
};
|
||||
delete Function.prototype['toString'];
|
||||
set_native(Function.prototype, "toString", mytoString);
|
||||
set_native(Function.prototype.toString, myFunction_toString_symbol, "function toString() { [native code] }");
|
||||
this.func_set_native = function (func) {
|
||||
set_native(func, myFunction_toString_symbol, `function ${myFunction_toString_symbol, func.name || ''}() { [native code] }`)
|
||||
}
|
||||
}).call(globalThis);
|
||||
|
||||
// 重写全局对象原型链
|
||||
function setTostringAndstringTag(obj) {
|
||||
Object.defineProperties(obj.prototype, {
|
||||
[Symbol.toStringTag]: {
|
||||
configurable: true,
|
||||
value: obj.name
|
||||
}
|
||||
});
|
||||
safeFunction(obj);
|
||||
};
|
||||
|
||||
// 创建标签原型
|
||||
function createTagProto(propObj,portotypeObj) {
|
||||
let res = propObj + ' = ' + 'function ' + propObj + '() { throw new TypeError("Illegal constructor"); };\n';
|
||||
res += 'setTostringAndstringTag(' + propObj + ',null);\n';
|
||||
if (portotypeObj) {
|
||||
for (let key in portotypeObj) {
|
||||
res += propObj + '.prototype.' + portotypeObj[key] + '= function ' + portotypeObj[key] + '() {AsObj.print("'+propObj+'.prototype.' + portotypeObj[key] + '原型方法(需在实例对象上补该方法)::",arguments)};\n';
|
||||
res += 'globalThis.func_set_native(' + propObj + '.prototype.' + portotypeObj[key] + ');\n';
|
||||
}
|
||||
}
|
||||
eval(res);
|
||||
}
|
||||
|
||||
Object.defineProperties(globalThis, {
|
||||
[Symbol.toStringTag]: {
|
||||
configurable: true,
|
||||
value: 'Window'
|
||||
}
|
||||
});
|
||||
|
||||
for (let key in globalThis) {
|
||||
if (typeof globalThis[key] === 'function') {
|
||||
safeFunction(globalThis[key])
|
||||
}
|
||||
}
|
||||
for (let key in console) {
|
||||
if (typeof console[key] === 'function') {
|
||||
safeFunction(console[key])
|
||||
}
|
||||
}
|
||||
|
||||
createTagProto('EventTarget',['addEventListener']);
|
||||
createTagProto('WindowProperties');
|
||||
createTagProto('Window');
|
||||
|
||||
window = globalThis;
|
||||
window.__proto__ = Window.prototype;
|
||||
window.__proto__.__proto__ = WindowProperties.prototype;
|
||||
window.__proto__.__proto__.__proto__ = EventTarget.prototype;
|
||||
Window.__proto__ = EventTarget;
|
||||
|
||||
Object.defineProperty(window, 'WindowProperties', {
|
||||
get: function () {
|
||||
return undefined;
|
||||
}
|
||||
})
|
||||
|
||||
function randoms(min, max) {
|
||||
return Math.floor(Math.random() * (max - min + 1) + min)
|
||||
}
|
||||
|
||||
function getRandomValues(buf) {
|
||||
var min = 0,
|
||||
max = 255;
|
||||
if (buf instanceof Uint16Array) {
|
||||
max = 65535;
|
||||
} else if (buf instanceof Uint32Array) {
|
||||
max = 4294967295;
|
||||
}
|
||||
for (var element in buf) {
|
||||
buf[element] = randoms(min, max);
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
|
||||
self = window.self = window;
|
||||
frames = window.frames = window;
|
||||
top = window.top = window;
|
||||
parent = window.parent = window;
|
||||
global = window.global = window;
|
||||
|
||||
Object.defineProperty(window, "global", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function global(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "top", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function top(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "self", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function self(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "parent", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function parent(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "frames", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function frames(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
|
||||
innerWidth = 1536
|
||||
innerHeight = 715
|
||||
outerWidth = 1536
|
||||
outerHeight = 824
|
||||
devicePixelRatio = 1.25;
|
||||
screenLeft = 0;
|
||||
screenX = 0;
|
||||
screenTop = 0;
|
||||
screenY = 0;
|
||||
opener = null;
|
||||
isSecureContext = true;
|
||||
crypto = {
|
||||
getRandomValues:getRandomValues
|
||||
};
|
||||
|
||||
createTagProto('DOMStringMap')
|
||||
createTagProto('HTMLHeadElement',['insertBefore','removeChild'])
|
||||
createTagProto('HTMLBodyElement',['addEventListener','appendChild','removeChild'])
|
||||
createTagProto('HTMLHtmlElement',['getAttribute'])
|
||||
createTagProto('HTMLDocument')
|
||||
createTagProto('Document',['browsingTopics','appendChild','querySelector','evaluate','querySelectorAll','removeChild','requestStorageAccess','requestStorageAccessFor','hasStorageAccess','getElementsByTagName','hasPrivateToken','createElement','hasRedemptionRecord','hasFocus'])
|
||||
createTagProto('Node')
|
||||
document = {};
|
||||
document.__proto__ = HTMLDocument.prototype;
|
||||
document.__proto__.__proto__ = Document.prototype;
|
||||
document.__proto__.__proto__.__proto__ = Node.prototype;
|
||||
document.__proto__.__proto__.__proto__.__proto__ = EventTarget.prototype;
|
||||
HTMLDocument.__proto__ = Document;
|
||||
HTMLDocument.__proto__.__proto__ = Node;
|
||||
HTMLDocument.__proto__.__proto__.__proto__ = EventTarget;
|
||||
Document.__proto__ = Node;
|
||||
Document.__proto__.__proto__ = EventTarget;
|
||||
Node.__proto__ = EventTarget;
|
||||
|
||||
createTagProto('Plugin');
|
||||
createTagProto('PluginArray');
|
||||
plugins0 = {
|
||||
name: 'PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type: 'application/pdf',
|
||||
},
|
||||
'1':{
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins0['0'].enabledPlugin = plugins0;
|
||||
plugins0['1'].enabledPlugin = plugins0;
|
||||
plugins1 = {
|
||||
name: 'Chrome PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1': {
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins1['0'].enabledPlugin = plugins1;
|
||||
plugins1['1'].enabledPlugin = plugins1;
|
||||
plugins2 = {
|
||||
name: 'Chromium PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1': {
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins2['0'].enabledPlugin = plugins2;
|
||||
plugins2['1'].enabledPlugin = plugins2;
|
||||
plugins3 = {
|
||||
name: 'Microsoft Edge PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0':{
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1': {
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins3['0'].enabledPlugin = plugins3;
|
||||
plugins3['1'].enabledPlugin = plugins3;
|
||||
plugins4 = {
|
||||
name: 'WebKit built-in PDF',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1':{
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins4['0'].enabledPlugin = plugins4;
|
||||
plugins4['1'].enabledPlugin = plugins4;
|
||||
plugins = {
|
||||
length: 5,
|
||||
'0': plugins0,
|
||||
'1': plugins1,
|
||||
'2': plugins2,
|
||||
'3': plugins3,
|
||||
'4': plugins4,
|
||||
namedItem : function (name) {
|
||||
AsObj.print('Plugin-namedItem:', name)
|
||||
},
|
||||
item: function (index) {
|
||||
AsObj.print('Plugin-item:', index)
|
||||
return watch(plugins0,'item-'+index);
|
||||
},
|
||||
refresh: function () {
|
||||
AsObj.print('Plugin-refresh:',arguments)
|
||||
},
|
||||
}
|
||||
plugins.__proto__ = PluginArray.prototype;
|
||||
|
||||
MimeTypeArray = function MimeTypeArray() {
|
||||
this.length = 2;
|
||||
this['0'] = {
|
||||
suffixes: 'pdf',
|
||||
type: 'application/pdf',
|
||||
description:"Portable Document Format",
|
||||
enabledPlugin: plugins0
|
||||
};
|
||||
this['1'] = {
|
||||
suffixes: 'pdf',
|
||||
type: 'text/pdf',
|
||||
description:"Portable Document Format",
|
||||
enabledPlugin: plugins0
|
||||
};
|
||||
};
|
||||
MimeTypeArray.prototype.toString = function () { return '[object MimeTypeArray]'; }
|
||||
MimeTypeArray.toString = function () { return 'function MimeTypeArray() { [native code] }'; }
|
||||
Object.defineProperties(MimeTypeArray.prototype, { [Symbol.toStringTag]: { value: 'MimeTypeArray' } })
|
||||
MimeTypeArrayc = new MimeTypeArray();
|
||||
MimeTypeArrayc[Symbol.iterator] = function* () {
|
||||
for (let key in this) {
|
||||
yield this[key];
|
||||
}
|
||||
}
|
||||
|
||||
// 创建电池管理器对象原型
|
||||
const BatteryManager = {
|
||||
level: 1,
|
||||
charging: true,
|
||||
chargingTime: 0,
|
||||
dischargingTime: null,
|
||||
onchargingchange: null,
|
||||
onlevelchange: null,
|
||||
toString: function toString() {
|
||||
return `BatteryManager {
|
||||
charging: ${this.charging},
|
||||
level: ${this.level},
|
||||
chargingTime: ${this.chargingTime},
|
||||
dischargingTime: ${this.dischargingTime}
|
||||
}`
|
||||
}
|
||||
}
|
||||
window.BatteryManager = BatteryManager;
|
||||
|
||||
Promise2 = {
|
||||
then: function () {
|
||||
return this;
|
||||
},
|
||||
catch: function (){},
|
||||
};
|
||||
|
||||
createTagProto('Bluetooth');
|
||||
createTagProto('Navigator');
|
||||
Navigator.prototype.hardwareConcurrency = 8;
|
||||
Navigator.prototype.userAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36';
|
||||
Navigator.prototype.appVersion = '5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36'
|
||||
Navigator.prototype.appName = 'Netscape';
|
||||
Navigator.prototype.appCodeName = 'Mozilla';
|
||||
Navigator.prototype.vendor = 'Google Inc.';
|
||||
Navigator.prototype.maxTouchPoints = 10;
|
||||
Navigator.prototype.platform = 'Win32';
|
||||
Navigator.prototype.adAuctionComponents = function adAuctionComponents() {
|
||||
AsObj.print('adAuctionComponents:::', arguments)
|
||||
}
|
||||
safeFunction(Navigator.prototype.adAuctionComponents)
|
||||
Navigator.prototype.runAdAuction = function runAdAuction() {
|
||||
AsObj.print('runAdAuction:::', arguments)
|
||||
}
|
||||
safeFunction(Navigator.prototype.runAdAuction)
|
||||
Navigator.prototype.canLoadAdAuctionFencedFrame = makeFunction('canLoadAdAuctionFencedFrame')
|
||||
Navigator.prototype.deprecatedReplaceInURN = makeFunction('deprecatedReplaceInURN')
|
||||
Navigator.prototype.deprecatedURNToURL = makeFunction('deprecatedURNToURL')
|
||||
Navigator.prototype.joinAdInterestGroup = makeFunction('joinAdInterestGroup')
|
||||
Navigator.prototype.leaveAdInterestGroup = makeFunction('leaveAdInterestGroup')
|
||||
Navigator.prototype.updateAdInterestGroups = makeFunction('updateAdInterestGroups')
|
||||
Navigator.prototype.connection = watch({
|
||||
downlink: 9.1,
|
||||
effectiveType: '4g',
|
||||
rtt: 0,
|
||||
saveData: false,
|
||||
},'connection')
|
||||
Navigator.prototype.language = 'zh-CN';
|
||||
Navigator.prototype.languages = ["zh-CN"];
|
||||
Navigator.prototype.plugins = plugins;
|
||||
Navigator.prototype.webdriver = false;
|
||||
Navigator.prototype.cookieEnabled = true;
|
||||
Navigator.prototype.onLine = true;
|
||||
Navigator.prototype.doNotTrack = null;
|
||||
Navigator.prototype.bluetooth = {};
|
||||
Navigator.prototype.product = 'Gecko'
|
||||
Navigator.prototype.deviceMemory = 8
|
||||
Navigator.prototype.mediaDevices = watch({
|
||||
enumerateDevices: function enumerateDevices() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = [
|
||||
{deviceId: '', kind: 'audioinput', label: '', groupId: ''},
|
||||
{deviceId: '', kind: 'videoinput', label: '', groupId: ''},
|
||||
{deviceId: '', kind: 'audiooutput', label: '', groupId: ''},
|
||||
]
|
||||
resolve(offer);
|
||||
});
|
||||
},
|
||||
getUserMedia: function getUserMedia() {
|
||||
AsObj.print('getUserMedia:::', arguments)
|
||||
}
|
||||
},'mediaDevices')
|
||||
Navigator.prototype.storage = {
|
||||
estimate: function estimate() {
|
||||
AsObj.print('estimate:::', arguments)
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
usage: 0, // 1GB
|
||||
quota: 2147483648, // 1GB,
|
||||
usageDetails: {caches: 512, indexedDB: 2855}
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
}
|
||||
Navigator.prototype.webkitPersistentStorage = watch({},'webkitPersistentStorage')
|
||||
Navigator.prototype.webkitTemporaryStorage = watch({
|
||||
queryUsageAndQuota: function queryUsageAndQuota() {
|
||||
AsObj.print('queryUsageAndQuota:::', arguments)
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
usage: 1024 * 1024 * 1024, // 1GB
|
||||
quota: 1024 * 1024 * 1024, // 1GB
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
},'webkitTemporaryStorage')
|
||||
Navigator.prototype.bluetooth.__proto__ = Bluetooth.prototype;
|
||||
Navigator.prototype.javaEnabled = function javaEnabled() {
|
||||
return false
|
||||
};
|
||||
safeFunction(Navigator.prototype.javaEnabled)
|
||||
Navigator.prototype.getBattery = function getBattery() {
|
||||
AsObj.print('getBattery:::', arguments)
|
||||
return Promise.resolve({
|
||||
__proto__: BatteryManager,
|
||||
// 动态参数配置(示例值)
|
||||
level: 1,
|
||||
charging: true,
|
||||
dischargingTime: null // 2小时放电时间
|
||||
})
|
||||
}
|
||||
safeFunction(Navigator.prototype.getBattery)
|
||||
Navigator.prototype.registerProtocolHandler = function registerProtocolHandler() {
|
||||
AsObj.print('registerProtocolHandler:::',arguments)
|
||||
}
|
||||
safeFunction(Navigator.prototype.registerProtocolHandler)
|
||||
Navigator.prototype.mimeTypes = watch(MimeTypeArrayc,'mimeTypes');
|
||||
Navigator.prototype.geolocation = {
|
||||
getCurrentPosition: function getCurrentPosition() {
|
||||
return Promise2;
|
||||
}
|
||||
}
|
||||
Navigator.prototype.pdfViewerEnabled = true;
|
||||
Navigator.prototype.doNotTrack = null;
|
||||
Navigator.prototype.keyboard = watch({
|
||||
getLayoutMap: function getLayoutMap() {
|
||||
AsObj.print('Navigator.prototype.keyboard:', arguments)
|
||||
return {
|
||||
then: function () {
|
||||
// arguments[0](watch({
|
||||
// size: 48,
|
||||
// values: function () {
|
||||
// return ['k', 'g', '2', '0', 'v', 'a', '`', 'l', '\\', "'", 'w', '8', 'm', 'h', '.', '7', '1', 'p', 'd', 'f', 'o', 'q', 'c', 'n', '[', 'z', 'y', '3', '6', '5', 'x', '/', '\\', ',', '-', '4', 'b', 't', '9', 's', 'i', 'u', '=', 'j', ';', 'r', ']', 'e']
|
||||
// }
|
||||
// }, 'navigator.keyboard.getLayoutMap.then'))
|
||||
return {
|
||||
catch: function () {
|
||||
arguments[0]({
|
||||
message:'getLayoutMap() must be called from a top-level browsing context or allowed by the permission policy.'
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},'navigator.keyboard')
|
||||
Navigator.prototype.permissions = watch({
|
||||
query: function query() {
|
||||
arg_obj = arguments[0];
|
||||
if (arg_obj.name === 'audio_capture') {
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
state: 'prompt',
|
||||
onchange: null,
|
||||
name:arg_obj.name
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
if (arg_obj.name === 'microphone') {
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0](watch({
|
||||
state: 'denied',
|
||||
onchange: null,
|
||||
name: 'audio_capture'
|
||||
},'permissions.query.microphone'));
|
||||
return {catch:function(){}}
|
||||
},
|
||||
catch:function(){}
|
||||
}
|
||||
}
|
||||
if (arg_obj.name === 'camera') {
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0](watch({
|
||||
state: 'prompt',
|
||||
onchange: null,
|
||||
name: 'video_capture'
|
||||
},'permissions.query.camera'));
|
||||
return {catch:function(){}}
|
||||
},
|
||||
catch:function(){}
|
||||
}
|
||||
}
|
||||
AsObj.print('permissions.query:::', arguments)
|
||||
|
||||
}
|
||||
},'permissions')
|
||||
Navigator.prototype.productSub = '20030107'
|
||||
Navigator.prototype.getGamepads = function getGamepads() {
|
||||
AsObj.print('getGamepads:::', arguments)
|
||||
return [null,null,null,null]
|
||||
}
|
||||
safeFunction(Navigator.prototype.getGamepads)
|
||||
|
||||
Navigator.prototype.sendBeacon = makeFunction('sendBeacon')
|
||||
|
||||
Navigator.prototype.deprecatedRunAdAuctionEnforcesKAnonymity = false
|
||||
Navigator.prototype.gpu = watch({
|
||||
getPreferredCanvasFormat: function getPreferredCanvasFormat() {
|
||||
AsObj.print('gpu.getPreferredCanvasFormat:', arguments)
|
||||
return 'bgra8unorm'
|
||||
},
|
||||
wgslLanguageFeatures: watch({
|
||||
size: 7,
|
||||
values: function values() {
|
||||
debugger
|
||||
AsObj.print('wgslLanguageFeatures.values')
|
||||
return ['packed_4x8_integer_dot_product', 'unrestricted_pointer_parameters', 'subgroup_uniformity', 'subgroup_id', 'pointer_composite_access', 'readonly_and_readwrite_storage_textures', 'uniform_buffer_standard_layout']
|
||||
},
|
||||
}, 'gpu.wgslLanguageFeatures'),
|
||||
requestAdapter: function requestAdapter() {
|
||||
AsObj.print('gpu.requestAdapter:', arguments)
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0](watch({
|
||||
features: watch({
|
||||
size: 19,
|
||||
values: function () {
|
||||
return ['depth32float-stencil8', 'rg11b10ufloat-renderable', 'bgra8unorm-storage', 'texture-formats-tier1', 'texture-compression-bc', 'dual-source-blending', 'core-features-and-limits', 'float32-filterable', 'indirect-first-instance', 'float32-blendable', 'depth-clip-control', 'texture-compression-bc-sliced-3d', 'timestamp-query', 'texture-formats-tier2', 'clip-distances', 'shader-f16', 'primitive-index', 'texture-component-swizzle', 'subgroups']
|
||||
}
|
||||
}, 'gpu.requestAdapter.features'),
|
||||
info: watch({ vendor: 'intel', architecture: 'gen-11', device: '', description: '', subgroupMinSize: 16 }, 'gpu.requestAdapter.info'),
|
||||
limits: watch({
|
||||
maxBufferSize: 2147483648,
|
||||
maxStorageBufferBindingSize:2147483644
|
||||
}, 'gpu.requestAdapter.limits'),
|
||||
catch:function(){}
|
||||
}, 'gpu.requestAdapter'));
|
||||
return {
|
||||
catch: function () {
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0]()
|
||||
return {catch:function(){}}
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
},
|
||||
catch: function () {
|
||||
}
|
||||
}
|
||||
}
|
||||
},'navigator.gpu')
|
||||
Navigator.prototype.userAgentData = watch({
|
||||
brands:[
|
||||
{
|
||||
"brand": "Google Chrome",
|
||||
"version": "143"
|
||||
},
|
||||
{
|
||||
"brand": "Chromium",
|
||||
"version": "143"
|
||||
},
|
||||
{
|
||||
"brand": "Not A(Brand",
|
||||
"version": "24"
|
||||
}
|
||||
],
|
||||
mobile: false,
|
||||
platform: "Windows",
|
||||
getHighEntropyValues: function getHighEntropyValues() {
|
||||
if (arguments[0] + '' === 'architecture,bitness,model,platformVersion,uaFullVersion,wow64') {
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
"architecture": "x86",
|
||||
"bitness": "64",
|
||||
"brands": [
|
||||
{
|
||||
"brand": "Not:A-Brand",
|
||||
"version": "99"
|
||||
},
|
||||
{
|
||||
"brand": "Google Chrome",
|
||||
"version": "145"
|
||||
},
|
||||
{
|
||||
"brand": "Chromium",
|
||||
"version": "145"
|
||||
}
|
||||
],
|
||||
"mobile": false,
|
||||
"model": "",
|
||||
"platform": "Windows",
|
||||
"platformVersion": "10.0.0",
|
||||
"uaFullVersion": "145.0.7632.117",
|
||||
"wow64": false
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
AsObj.print('getHighEntropyValues:::', arguments)
|
||||
}
|
||||
},'userAgentData')
|
||||
|
||||
navigator = {};
|
||||
navigator.__proto__ = Navigator.prototype;
|
||||
|
||||
createTagProto('Location');
|
||||
location = {
|
||||
"ancestorOrigins": {},
|
||||
"href": "https://www.neimanmarcus.com/",
|
||||
"origin": "https://www.neimanmarcus.com",
|
||||
"protocol": "https:",
|
||||
"host": "www.neimanmarcus.com",
|
||||
"hostname": "www.neimanmarcus.com",
|
||||
"port": "",
|
||||
"pathname": "/",
|
||||
"search": "",
|
||||
"hash": ""
|
||||
};
|
||||
|
||||
location.__proto__ = Location.prototype;
|
||||
location.toString = function toString() {
|
||||
return this.href;
|
||||
}
|
||||
|
||||
createTagProto('Screen');
|
||||
Screen.prototype = Object.assign(Screen.prototype, {
|
||||
availHeight: 824,
|
||||
availLeft: 0,
|
||||
availTop: 0,
|
||||
availWidth: 1536,
|
||||
colorDepth: 32,
|
||||
height: 864,
|
||||
isExtended: true,
|
||||
onchange: null,
|
||||
pixelDepth: 24,
|
||||
width: 1536,
|
||||
orientation: {
|
||||
angle: 0,
|
||||
type: "landscape-primary",
|
||||
onchange: null
|
||||
}
|
||||
})
|
||||
screen = {};
|
||||
screen.__proto__ = Screen.prototype;
|
||||
|
||||
createTagProto('History',['replaceState']);
|
||||
history = {};
|
||||
history.__proto__ = History.prototype;
|
||||
|
||||
chrome = {
|
||||
loadTimes: function loadTimes() { },
|
||||
csi: function csi() { },
|
||||
app: {
|
||||
InstallState: { DISABLED: 'disabled', INSTALLED: 'installed', NOT_INSTALLED: 'not_installed' },
|
||||
RunningState: { CANNOT_RUN: 'cannot_run', READY_TO_RUN: 'ready_to_run', RUNNING: 'running' },
|
||||
getDetails:function getDetails(){},
|
||||
getIsInstalled:function getIsInstalled(){},
|
||||
installState:function installState(){},
|
||||
isInstalled: false,
|
||||
runningState: function runningState(){}
|
||||
},
|
||||
}
|
||||
|
||||
createTagProto('Storage');
|
||||
local = {
|
||||
};
|
||||
localStorage = {
|
||||
getItem: function getItem(key) {
|
||||
AsObj.print("localStorage.getItem::", arguments);
|
||||
if (!local[key]) {
|
||||
return null;
|
||||
}
|
||||
return local[key];
|
||||
},
|
||||
setItem: function setItem(key, value) {
|
||||
AsObj.print("localStorage.setItem::", arguments);
|
||||
local[key] = value;
|
||||
},
|
||||
clear: function clear() {
|
||||
local = {};
|
||||
},
|
||||
removeItem: function removeItem(key) {
|
||||
AsObj.print("localStorage.removeItem::", arguments);
|
||||
delete local[key];
|
||||
}
|
||||
}
|
||||
localStorage.__proto__ = Storage.prototype;
|
||||
sessionStorage = {
|
||||
getItem: function getItem(key) {
|
||||
AsObj.print("sessionStorage.getItem::", arguments);
|
||||
if (!local[key]) {
|
||||
return null;
|
||||
}
|
||||
return local[key];
|
||||
},
|
||||
setItem: function setItem(key, value) {
|
||||
AsObj.print("sessionStorage.setItem::", arguments);
|
||||
local[key] = value;
|
||||
},
|
||||
clear: function clear() {
|
||||
local = {};
|
||||
},
|
||||
removeItem: function removeItem(key) {
|
||||
AsObj.print("sessionStorage.removeItem::", arguments);
|
||||
delete local[key];
|
||||
}
|
||||
}
|
||||
sessionStorage.__proto__ = Storage.prototype;
|
||||
|
||||
// window = watch(window, 'window');
|
||||
// global = watch(global, 'global');
|
||||
// globalThis = watch(globalThis, 'globalThis');
|
||||
// self = watch(self, 'self');
|
||||
// crypto = watch(crypto, 'crypto');
|
||||
// performance = watch(performance, 'performance');
|
||||
// document = watch(document, 'document');
|
||||
// navigator = watch(navigator, 'navigator');
|
||||
// location = watch(location, 'location');
|
||||
// screen = watch(screen, 'screen');
|
||||
// history = watch(history, 'history');
|
||||
// localStorage = watch(localStorage, 'localStorage');
|
||||
// sessionStorage = watch(sessionStorage, 'sessionStorage');
|
||||
// chrome = watch(chrome, 'chrome');
|
||||
|
||||
require_('./sdk_leg.js');
|
||||
|
||||
let input = '';
|
||||
// 收集数据
|
||||
process.stdin.on('data', chunk => {
|
||||
input += chunk;
|
||||
});
|
||||
process.stdin.on('end', async () => {
|
||||
var config_data = JSON.parse(input);
|
||||
var cryptoManager = await CaptchaSDKCorecc();
|
||||
var encryptData = await buildEncryptedVerifyRequestcc(config_data, cryptoManager);
|
||||
console.log(JSON.stringify(encryptData));
|
||||
process.exit(0);
|
||||
})
|
||||
3
domainCheck/app/thread_count.json
Normal file
3
domainCheck/app/thread_count.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"thread_count": "1"
|
||||
}
|
||||
4
domainCheck/app/ui/__init__.py
Normal file
4
domainCheck/app/ui/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
UI 模块
|
||||
'''
|
||||
1423
domainCheck/app/ui/domain_filter.py
Normal file
1423
domainCheck/app/ui/domain_filter.py
Normal file
File diff suppressed because it is too large
Load Diff
510
domainCheck/app/ui/domain_import.py
Normal file
510
domainCheck/app/ui/domain_import.py
Normal file
@@ -0,0 +1,510 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :domain_import.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:47
|
||||
@explain : 域名导入界面
|
||||
'''
|
||||
|
||||
from PySide6.QtWidgets import QWidget, QVBoxLayout, QHBoxLayout, QPushButton, QLabel, QTextEdit, QFileDialog, QProgressBar
|
||||
from PySide6.QtCore import Qt, QThread, Signal
|
||||
from loguru import logger
|
||||
|
||||
from app.core.domain_collector import DomainCollector
|
||||
|
||||
|
||||
class ImportThread(QThread):
|
||||
"""
|
||||
导入线程
|
||||
"""
|
||||
progress_updated = Signal(int)
|
||||
finished = Signal(bool, str)
|
||||
|
||||
def __init__(self, domain_list, source_type):
|
||||
"""
|
||||
初始化导入线程
|
||||
|
||||
:param domain_list: 域名列表
|
||||
:param source_type: 来源类型
|
||||
"""
|
||||
super().__init__()
|
||||
self.domain_list = domain_list
|
||||
self.source_type = source_type
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
运行导入线程
|
||||
"""
|
||||
try:
|
||||
collector = DomainCollector()
|
||||
total = len(self.domain_list)
|
||||
|
||||
# 实时更新进度:开始
|
||||
self.progress_updated.emit(0)
|
||||
|
||||
# 标准化域名和检查是否存在(占30%进度)
|
||||
normalized_domains = []
|
||||
for i, domain in enumerate(self.domain_list):
|
||||
from app.utils.domain_utils import normalize_domain
|
||||
import tldextract
|
||||
normalized = normalize_domain(domain)
|
||||
if normalized:
|
||||
# 提取顶级域名
|
||||
ext = tldextract.extract(normalized)
|
||||
tld = ext.suffix
|
||||
normalized_domains.append((normalized, tld))
|
||||
|
||||
# 更新进度
|
||||
progress = int((i + 1) / total * 30)
|
||||
self.progress_updated.emit(progress)
|
||||
|
||||
# 批量检查域名是否存在
|
||||
batch_data = []
|
||||
existing_domains = []
|
||||
if normalized_domains:
|
||||
all_domains = [domain for domain, tld in normalized_domains]
|
||||
existing_domains = collector.db.check_domains_exist(all_domains)
|
||||
existing_set = set(existing_domains)
|
||||
|
||||
# 准备批量添加数据
|
||||
for domain, tld in normalized_domains:
|
||||
if domain not in existing_set:
|
||||
batch_data.append((domain, tld, self.source_type))
|
||||
|
||||
# 分批次添加域名(占70%进度)
|
||||
batch_size = 1000
|
||||
total_batches = len(batch_data)
|
||||
for i in range(0, len(batch_data), batch_size):
|
||||
batch = batch_data[i:i+batch_size]
|
||||
collector.db.add_domains_batch(batch)
|
||||
|
||||
# 更新进度
|
||||
processed = min(i + len(batch), total_batches)
|
||||
progress = 30 + int(processed / total_batches * 70)
|
||||
self.progress_updated.emit(progress)
|
||||
|
||||
# 完成导入
|
||||
self.progress_updated.emit(100)
|
||||
|
||||
# 计算统计信息
|
||||
stats = {
|
||||
'total': total,
|
||||
'valid': len(normalized_domains),
|
||||
'added': len(batch_data),
|
||||
'exists': len(existing_domains),
|
||||
'invalid': total - len(normalized_domains),
|
||||
'failed': 0
|
||||
}
|
||||
|
||||
# 根据统计信息生成消息
|
||||
if stats['added'] > 0:
|
||||
message = f"导入完成: 总域名数 {stats['total']}, 有效域名数 {stats['valid']}, 新增域名数 {stats['added']}, 已存在域名数 {stats['exists']}, 无效域名数 {stats['invalid']}"
|
||||
else:
|
||||
message = f"导入完成: 所有域名已存在,未添加新域名"
|
||||
|
||||
self.finished.emit(True, message)
|
||||
except Exception as e:
|
||||
logger.error(f"导入失败: {e}")
|
||||
self.finished.emit(False, f"导入失败: {str(e)}")
|
||||
|
||||
|
||||
class ImportFileThread(QThread):
|
||||
"""
|
||||
文件导入线程,用于处理大文件
|
||||
"""
|
||||
progress_updated = Signal(int)
|
||||
finished = Signal(bool, str)
|
||||
|
||||
def __init__(self, file_path, source_type):
|
||||
"""
|
||||
初始化文件导入线程
|
||||
|
||||
:param file_path: 文件路径
|
||||
:param source_type: 来源类型
|
||||
"""
|
||||
super().__init__()
|
||||
self.file_path = file_path
|
||||
self.source_type = source_type
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
运行文件导入线程
|
||||
"""
|
||||
try:
|
||||
collector = DomainCollector()
|
||||
|
||||
# 首先计算文件中的域名数量
|
||||
total = 0
|
||||
encodings = ['utf-8', 'utf-8-sig', 'gbk', 'gb2312', 'cp936', 'latin-1', 'ascii']
|
||||
encoding = 'utf-8' # 默认编码
|
||||
|
||||
# 尝试不同的编码格式计算域名数量
|
||||
for enc in encodings:
|
||||
try:
|
||||
with open(self.file_path, 'r', encoding=enc) as f:
|
||||
total = sum(1 for line in f if line.strip())
|
||||
encoding = enc
|
||||
break
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
|
||||
if total == 0:
|
||||
# 尝试使用二进制模式读取
|
||||
try:
|
||||
import chardet
|
||||
with open(self.file_path, 'rb') as f:
|
||||
raw_data = f.read()
|
||||
result = chardet.detect(raw_data)
|
||||
encoding = result['encoding']
|
||||
if encoding:
|
||||
total = sum(1 for line in raw_data.decode(encoding).split('\n') if line.strip())
|
||||
else:
|
||||
# 最后尝试使用 replace 模式读取
|
||||
with open(self.file_path, 'r', encoding='utf-8', errors='replace') as f:
|
||||
total = sum(1 for line in f if line.strip())
|
||||
encoding = 'utf-8'
|
||||
except Exception:
|
||||
# 最后尝试使用 replace 模式读取
|
||||
with open(self.file_path, 'r', encoding='utf-8', errors='replace') as f:
|
||||
total = sum(1 for line in f if line.strip())
|
||||
encoding = 'utf-8'
|
||||
|
||||
# 实时更新进度:开始
|
||||
self.progress_updated.emit(0)
|
||||
|
||||
# 逐行读取文件并处理域名
|
||||
normalized_domains = []
|
||||
processed = 0
|
||||
|
||||
with open(self.file_path, 'r', encoding=encoding, errors='replace') as f:
|
||||
for line in f:
|
||||
domain = line.strip()
|
||||
if domain:
|
||||
from app.utils.domain_utils import normalize_domain
|
||||
import tldextract
|
||||
normalized = normalize_domain(domain)
|
||||
if normalized:
|
||||
# 提取顶级域名
|
||||
ext = tldextract.extract(normalized)
|
||||
tld = ext.suffix
|
||||
normalized_domains.append((normalized, tld))
|
||||
|
||||
processed += 1
|
||||
# 更新进度(占30%)
|
||||
progress = int(processed / total * 30)
|
||||
self.progress_updated.emit(progress)
|
||||
|
||||
# 批量检查域名是否存在
|
||||
batch_data = []
|
||||
existing_domains = []
|
||||
if normalized_domains:
|
||||
all_domains = [domain for domain, tld in normalized_domains]
|
||||
existing_domains = collector.db.check_domains_exist(all_domains)
|
||||
existing_set = set(existing_domains)
|
||||
|
||||
# 准备批量添加数据
|
||||
for domain, tld in normalized_domains:
|
||||
if domain not in existing_set:
|
||||
batch_data.append((domain, tld, self.source_type))
|
||||
|
||||
# 分批次添加域名(占70%进度)
|
||||
batch_size = 1000
|
||||
total_batches = len(batch_data)
|
||||
for i in range(0, len(batch_data), batch_size):
|
||||
batch = batch_data[i:i+batch_size]
|
||||
collector.db.add_domains_batch(batch)
|
||||
|
||||
# 更新进度
|
||||
processed_batches = min(i + len(batch), total_batches)
|
||||
progress = 30 + int(processed_batches / total_batches * 70)
|
||||
self.progress_updated.emit(progress)
|
||||
|
||||
# 完成导入
|
||||
self.progress_updated.emit(100)
|
||||
|
||||
# 计算统计信息
|
||||
stats = {
|
||||
'total': total,
|
||||
'valid': len(normalized_domains),
|
||||
'added': len(batch_data),
|
||||
'exists': len(existing_domains),
|
||||
'invalid': total - len(normalized_domains),
|
||||
'failed': 0
|
||||
}
|
||||
|
||||
# 根据统计信息生成消息
|
||||
if stats['added'] > 0:
|
||||
message = f"导入完成: 总域名数 {stats['total']}, 有效域名数 {stats['valid']}, 新增域名数 {stats['added']}, 已存在域名数 {stats['exists']}, 无效域名数 {stats['invalid']}"
|
||||
else:
|
||||
message = f"导入完成: 所有域名已存在,未添加新域名"
|
||||
|
||||
self.finished.emit(True, message)
|
||||
except Exception as e:
|
||||
logger.error(f"导入失败: {e}")
|
||||
self.finished.emit(False, f"导入失败: {str(e)}")
|
||||
|
||||
|
||||
class DomainImportWidget(QWidget):
|
||||
"""
|
||||
域名导入界面
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化域名导入界面
|
||||
"""
|
||||
super().__init__()
|
||||
|
||||
# 创建布局
|
||||
layout = QVBoxLayout(self)
|
||||
|
||||
|
||||
|
||||
# 创建文本编辑框
|
||||
self.text_edit = QTextEdit()
|
||||
self.text_edit.setPlaceholderText("请输入域名,一行一个")
|
||||
self.text_edit.setStyleSheet("""
|
||||
QTextEdit {
|
||||
font-size: 14px;
|
||||
padding: 10px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
background-color: #f9f9f9;
|
||||
min-height: 300px;
|
||||
}
|
||||
""")
|
||||
layout.addWidget(self.text_edit)
|
||||
|
||||
# 创建按钮布局
|
||||
button_layout = QHBoxLayout()
|
||||
|
||||
# 导入文件按钮
|
||||
self.import_file_btn = QPushButton("导入文件")
|
||||
self.import_file_btn.clicked.connect(self.import_file)
|
||||
self.import_file_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #2196F3;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #0b7dda;
|
||||
}
|
||||
QPushButton:disabled {
|
||||
background-color: #cccccc;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.import_file_btn)
|
||||
|
||||
# 开始导入按钮
|
||||
self.start_import_btn = QPushButton("开始导入")
|
||||
self.start_import_btn.clicked.connect(self.start_import)
|
||||
self.start_import_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #4CAF50;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #45a049;
|
||||
}
|
||||
QPushButton:disabled {
|
||||
background-color: #cccccc;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.start_import_btn)
|
||||
button_layout.setContentsMargins(0, 15, 0, 15)
|
||||
layout.addLayout(button_layout)
|
||||
|
||||
# 创建进度条
|
||||
self.progress_bar = QProgressBar()
|
||||
self.progress_bar.setVisible(False)
|
||||
self.progress_bar.setStyleSheet("""
|
||||
QProgressBar {
|
||||
height: 20px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 10px;
|
||||
background-color: #f0f0f0;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
QProgressBar::chunk {
|
||||
background-color: #4CAF50;
|
||||
border-radius: 10px;
|
||||
}
|
||||
""")
|
||||
layout.addWidget(self.progress_bar)
|
||||
|
||||
# 创建状态标签
|
||||
self.status_label = QLabel("")
|
||||
self.status_label.setAlignment(Qt.AlignCenter)
|
||||
self.status_label.setStyleSheet("font-size: 14px; color: #333; padding: 10px; background-color: #f0f8ff; border-radius: 4px;")
|
||||
layout.addWidget(self.status_label)
|
||||
|
||||
logger.info("域名导入界面创建完成")
|
||||
|
||||
def import_file(self):
|
||||
"""
|
||||
导入文件
|
||||
"""
|
||||
file_path, _ = QFileDialog.getOpenFileName(self, "选择文件", "", "文本文件 (*.txt)")
|
||||
if file_path:
|
||||
try:
|
||||
# 尝试不同的编码格式
|
||||
encodings = ['utf-8', 'utf-8-sig', 'gbk', 'gb2312', 'cp936', 'latin-1', 'ascii']
|
||||
domain_count = 0
|
||||
|
||||
# 尝试使用不同编码读取并计数
|
||||
for encoding in encodings:
|
||||
try:
|
||||
with open(file_path, 'r', encoding=encoding) as f:
|
||||
domain_count = sum(1 for line in f if line.strip())
|
||||
logger.info(f"使用编码 {encoding} 成功读取文件")
|
||||
break
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
|
||||
# 如果仍然失败,尝试使用二进制模式读取并猜测编码
|
||||
if domain_count == 0:
|
||||
try:
|
||||
import chardet
|
||||
with open(file_path, 'rb') as f:
|
||||
raw_data = f.read()
|
||||
result = chardet.detect(raw_data)
|
||||
encoding = result['encoding']
|
||||
if encoding:
|
||||
domain_count = sum(1 for line in raw_data.decode(encoding).split('\n') if line.strip())
|
||||
logger.info(f"使用 chardet 检测到编码 {encoding} 并成功读取文件")
|
||||
else:
|
||||
raise Exception("无法识别文件编码")
|
||||
except Exception as e:
|
||||
logger.warning(f"chardet 检测失败: {e}")
|
||||
# 最后尝试使用 replace 模式读取
|
||||
with open(file_path, 'r', encoding='utf-8', errors='replace') as f:
|
||||
domain_count = sum(1 for line in f if line.strip())
|
||||
logger.info("使用 utf-8 replace 模式读取文件")
|
||||
|
||||
# 对于大文件,不显示所有域名,只显示文件路径和域名数量
|
||||
if domain_count > 1000:
|
||||
self.text_edit.setText(f"文件路径: {file_path}\n域名数量: {domain_count}\n\n提示: 由于文件较大,仅显示文件信息,不显示具体域名。")
|
||||
# 保存文件路径,用于后续导入
|
||||
self.imported_file_path = file_path
|
||||
else:
|
||||
# 对于小文件,显示所有域名
|
||||
domains = []
|
||||
for encoding in encodings:
|
||||
try:
|
||||
with open(file_path, 'r', encoding=encoding) as f:
|
||||
domains = f.readlines()
|
||||
break
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
if not domains:
|
||||
# 尝试使用二进制模式读取
|
||||
try:
|
||||
import chardet
|
||||
with open(file_path, 'rb') as f:
|
||||
raw_data = f.read()
|
||||
result = chardet.detect(raw_data)
|
||||
encoding = result['encoding']
|
||||
if encoding:
|
||||
domains = raw_data.decode(encoding).split('\n')
|
||||
else:
|
||||
with open(file_path, 'r', encoding='utf-8', errors='replace') as f:
|
||||
domains = f.readlines()
|
||||
except Exception:
|
||||
with open(file_path, 'r', encoding='utf-8', errors='replace') as f:
|
||||
domains = f.readlines()
|
||||
domains = [domain.strip() for domain in domains if domain.strip()]
|
||||
self.text_edit.setText('\n'.join(domains))
|
||||
# 清除文件路径,使用文本框中的域名
|
||||
self.imported_file_path = None
|
||||
|
||||
self.status_label.setText(f"成功读取 {domain_count} 个域名")
|
||||
logger.info(f"成功读取文件: {file_path}, 共 {domain_count} 个域名")
|
||||
except Exception as e:
|
||||
self.status_label.setText(f"读取文件失败: {str(e)}")
|
||||
logger.error(f"读取文件失败: {e}")
|
||||
|
||||
def start_import(self):
|
||||
"""
|
||||
开始导入
|
||||
"""
|
||||
# 检查是否有导入的文件路径
|
||||
if hasattr(self, 'imported_file_path') and self.imported_file_path:
|
||||
# 大文件导入,使用文件路径
|
||||
file_path = self.imported_file_path
|
||||
|
||||
# 显示进度条
|
||||
self.progress_bar.setVisible(True)
|
||||
self.progress_bar.setValue(0)
|
||||
self.status_label.setText("正在导入...")
|
||||
|
||||
# 禁用按钮
|
||||
self.import_file_btn.setEnabled(False)
|
||||
self.start_import_btn.setEnabled(False)
|
||||
|
||||
# 创建并启动导入线程
|
||||
self.import_thread = ImportFileThread(file_path, 7) # 7 表示 TXT 导入
|
||||
self.import_thread.progress_updated.connect(self.update_progress)
|
||||
self.import_thread.finished.connect(self.import_finished)
|
||||
self.import_thread.start()
|
||||
|
||||
logger.info(f"开始从文件导入: {file_path}")
|
||||
else:
|
||||
# 小文件或手动输入的域名
|
||||
domains = self.text_edit.toPlainText().split('\n')
|
||||
domains = [domain.strip() for domain in domains if domain.strip()]
|
||||
|
||||
if not domains:
|
||||
self.status_label.setText("请输入域名")
|
||||
return
|
||||
|
||||
# 显示进度条
|
||||
self.progress_bar.setVisible(True)
|
||||
self.progress_bar.setValue(0)
|
||||
self.status_label.setText("正在导入...")
|
||||
|
||||
# 禁用按钮
|
||||
self.import_file_btn.setEnabled(False)
|
||||
self.start_import_btn.setEnabled(False)
|
||||
|
||||
# 创建并启动导入线程
|
||||
self.import_thread = ImportThread(domains, 7) # 7 表示 TXT 导入
|
||||
self.import_thread.progress_updated.connect(self.update_progress)
|
||||
self.import_thread.finished.connect(self.import_finished)
|
||||
self.import_thread.start()
|
||||
|
||||
logger.info(f"开始导入 {len(domains)} 个域名")
|
||||
|
||||
def update_progress(self, progress):
|
||||
"""
|
||||
更新进度
|
||||
|
||||
:param progress: 进度值
|
||||
"""
|
||||
self.progress_bar.setValue(progress)
|
||||
|
||||
def import_finished(self, success, message):
|
||||
"""
|
||||
导入完成
|
||||
|
||||
:param success: 是否成功
|
||||
:param message: 消息
|
||||
"""
|
||||
self.status_label.setText(message)
|
||||
self.progress_bar.setVisible(False)
|
||||
|
||||
# 启用按钮
|
||||
self.import_file_btn.setEnabled(True)
|
||||
self.start_import_btn.setEnabled(True)
|
||||
|
||||
logger.info(f"导入完成: {message}")
|
||||
577
domainCheck/app/ui/juming_crawler.py
Normal file
577
domainCheck/app/ui/juming_crawler.py
Normal file
@@ -0,0 +1,577 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :juming_crawler.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 15:00
|
||||
@explain : 聚名网爬取页面
|
||||
'''
|
||||
|
||||
from PySide6.QtWidgets import QWidget, QVBoxLayout, QHBoxLayout, QPushButton, QLabel, QTextEdit, QProgressBar, QLineEdit, QComboBox, QDateEdit, QCheckBox
|
||||
from PySide6.QtCore import Qt, QThread, Signal, QDate
|
||||
from PySide6.QtGui import QIntValidator
|
||||
from loguru import logger
|
||||
import re
|
||||
import datetime
|
||||
import time
|
||||
|
||||
from app.core.domain_collector import DomainCollector
|
||||
from detect.juming import JM
|
||||
|
||||
|
||||
class JumingCrawlThread(QThread):
|
||||
"""
|
||||
聚名爬取线程
|
||||
"""
|
||||
progress_updated = Signal(int)
|
||||
status_updated = Signal(str)
|
||||
finished = Signal(bool, str)
|
||||
|
||||
def __init__(self, crawl_type, page_start=1, page_size=50, crawl_date=None, auto_date=True):
|
||||
"""
|
||||
初始化聚名爬取线程
|
||||
|
||||
:param crawl_type: 爬取类型 (1: 一口价, 2: 删除列表)
|
||||
:param page_start: 起始页码
|
||||
:param page_size: 每页数量
|
||||
:param crawl_date: 爬取日期(删除列表用)
|
||||
:param auto_date: 是否自动新增日期
|
||||
"""
|
||||
super().__init__()
|
||||
self.crawl_type = crawl_type
|
||||
self.page_start = page_start
|
||||
self.page_size = page_size
|
||||
self.crawl_date = crawl_date
|
||||
self.auto_date = auto_date
|
||||
self.is_paused = False
|
||||
self.is_stopped = False
|
||||
self.current_page = 0
|
||||
self.total_count = 0
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
运行聚名爬取线程
|
||||
"""
|
||||
try:
|
||||
# 初始化聚名客户端
|
||||
jm = JM()
|
||||
|
||||
# 加载 Cookie
|
||||
jm.load_cookies()
|
||||
logger.info("已加载 Cookie")
|
||||
self.status_updated.emit("已加载 Cookie")
|
||||
|
||||
# 直接开始爬取,不需要登录,因为 Cookie 已经在系统设置页面加载了
|
||||
self.progress_updated.emit(10)
|
||||
|
||||
if self.crawl_type == 1: # 一口价
|
||||
self.progress_updated.emit(30)
|
||||
logger.info("开始获取一口价域名")
|
||||
self.status_updated.emit("开始获取一口价域名")
|
||||
|
||||
# 自动爬取多页
|
||||
page = self.page_start
|
||||
while not self.is_stopped:
|
||||
if self.is_paused:
|
||||
time.sleep(0.1)
|
||||
continue
|
||||
|
||||
self.current_page = page
|
||||
self.status_updated.emit(f"正在爬取第 {page} 页")
|
||||
logger.info(f"正在爬取第 {page} 页")
|
||||
|
||||
# 获取当前页
|
||||
success, html = jm.ykj_get_list(page=page, page_size=self.page_size)
|
||||
if success:
|
||||
pattern_ym = r"<a class='yda1 ydz' ym='([^']*)'"
|
||||
results = re.findall(pattern_ym, html)
|
||||
domains = [domain.strip() for domain in results if domain.strip()]
|
||||
domain_count = len(domains)
|
||||
self.total_count += domain_count
|
||||
|
||||
self.status_updated.emit(f"第 {page} 页找到 {domain_count} 个域名,累计 {self.total_count} 个")
|
||||
logger.info(f"第 {page} 页找到 {domain_count} 个域名,累计 {self.total_count} 个")
|
||||
|
||||
# 自动入库
|
||||
if domains:
|
||||
collector = DomainCollector()
|
||||
stats = collector.add_domains_batch(domains, 1) # 1 表示一口价
|
||||
logger.info(f"自动入库完成: {stats}")
|
||||
self.status_updated.emit(f"自动入库完成: 成功添加 {stats['added']} 个域名")
|
||||
|
||||
# 如果返回的数量小于指定的数量,停止爬取
|
||||
if domain_count < self.page_size:
|
||||
logger.info(f"返回数量小于指定数量,停止爬取")
|
||||
self.status_updated.emit("返回数量小于指定数量,停止爬取")
|
||||
break
|
||||
|
||||
# 增加页码
|
||||
page += 1
|
||||
|
||||
# 模拟网络延迟
|
||||
time.sleep(1)
|
||||
else:
|
||||
logger.error(f"获取一口价域名失败: {html}")
|
||||
self.status_updated.emit(f"获取一口价域名失败: {html}")
|
||||
break
|
||||
|
||||
elif self.crawl_type == 2: # 删除列表
|
||||
self.progress_updated.emit(30)
|
||||
logger.info("开始获取删除域名列表")
|
||||
self.status_updated.emit("开始获取删除域名列表")
|
||||
|
||||
# 使用传入的日期或默认今天
|
||||
start_date_str = self.crawl_date if self.crawl_date else datetime.date.today().strftime("%Y-%m-%d")
|
||||
start_date = datetime.datetime.strptime(start_date_str, "%Y-%m-%d").date()
|
||||
# 计算结束日期:今天 + 4天
|
||||
end_date = datetime.date.today() + datetime.timedelta(days=4)
|
||||
|
||||
if self.auto_date:
|
||||
# 自动新增日期,从起始日期到今天+4天
|
||||
current_date = start_date
|
||||
while current_date <= end_date and not self.is_stopped:
|
||||
if self.is_paused:
|
||||
time.sleep(0.1)
|
||||
continue
|
||||
|
||||
crawl_date = current_date.strftime("%Y-%m-%d")
|
||||
self.status_updated.emit(f"正在爬取 {crawl_date} 的删除域名")
|
||||
logger.info(f"正在爬取 {crawl_date} 的删除域名")
|
||||
|
||||
deleted_domains = jm.new_cha_del(crawl_date)
|
||||
domains = [domain.strip() for domain in deleted_domains if domain.strip()]
|
||||
domain_count = len(domains)
|
||||
self.total_count += domain_count
|
||||
|
||||
self.status_updated.emit(f"{crawl_date} 找到 {domain_count} 个删除域名,累计 {self.total_count} 个")
|
||||
logger.info(f"{crawl_date} 找到 {domain_count} 个删除域名,累计 {self.total_count} 个")
|
||||
|
||||
# 自动入库
|
||||
if domains:
|
||||
collector = DomainCollector()
|
||||
stats = collector.add_domains_batch(domains, 2) # 2 表示删除列表
|
||||
logger.info(f"{crawl_date} 自动入库完成: {stats}")
|
||||
self.status_updated.emit(f"{crawl_date} 自动入库完成: 成功添加 {stats['added']} 个域名")
|
||||
|
||||
# 增加日期
|
||||
current_date = current_date + datetime.timedelta(days=1)
|
||||
|
||||
# 模拟网络延迟
|
||||
time.sleep(1)
|
||||
else:
|
||||
# 只爬取指定日期
|
||||
crawl_date = start_date_str
|
||||
self.status_updated.emit(f"正在爬取 {crawl_date} 的删除域名")
|
||||
logger.info(f"正在爬取 {crawl_date} 的删除域名")
|
||||
|
||||
deleted_domains = jm.new_cha_del(crawl_date)
|
||||
domains = [domain.strip() for domain in deleted_domains if domain.strip()]
|
||||
domain_count = len(domains)
|
||||
self.total_count = domain_count
|
||||
|
||||
self.status_updated.emit(f"找到 {domain_count} 个删除域名")
|
||||
logger.info(f"找到 {domain_count} 个删除域名")
|
||||
|
||||
# 自动入库
|
||||
if domains:
|
||||
collector = DomainCollector()
|
||||
stats = collector.add_domains_batch(domains, 2) # 2 表示删除列表
|
||||
logger.info(f"自动入库完成: {stats}")
|
||||
self.status_updated.emit(f"自动入库完成: 成功添加 {stats['added']} 个域名")
|
||||
|
||||
self.progress_updated.emit(90)
|
||||
|
||||
self.progress_updated.emit(100)
|
||||
self.finished.emit(True, f"成功获取 {self.total_count} 个域名")
|
||||
except Exception as e:
|
||||
logger.error(f"从聚名网爬取失败: {e}")
|
||||
self.status_updated.emit(f"爬取失败: {str(e)}")
|
||||
self.finished.emit(False, f"爬取失败: {str(e)}")
|
||||
|
||||
def pause(self):
|
||||
"""
|
||||
暂停爬取
|
||||
"""
|
||||
self.is_paused = True
|
||||
logger.info("爬取已暂停")
|
||||
self.status_updated.emit("爬取已暂停")
|
||||
|
||||
def resume(self):
|
||||
"""
|
||||
恢复爬取
|
||||
"""
|
||||
self.is_paused = False
|
||||
logger.info("爬取已恢复")
|
||||
self.status_updated.emit("爬取已恢复")
|
||||
|
||||
def stop(self):
|
||||
"""
|
||||
停止爬取
|
||||
"""
|
||||
self.is_stopped = True
|
||||
logger.info("爬取已停止")
|
||||
self.status_updated.emit("爬取已停止")
|
||||
|
||||
|
||||
class JumingCrawlerWidget(QWidget):
|
||||
"""
|
||||
聚名网爬取页面
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化聚名网爬取页面
|
||||
"""
|
||||
super().__init__()
|
||||
|
||||
# 创建布局
|
||||
layout = QVBoxLayout(self)
|
||||
|
||||
|
||||
|
||||
# 爬取类型选择
|
||||
type_layout = QHBoxLayout()
|
||||
type_label = QLabel("爬取类型:")
|
||||
type_label.setStyleSheet("font-size: 14px; color: #666; min-width: 80px;")
|
||||
self.type_combo = QComboBox()
|
||||
self.type_combo.addItem("一口价域名", 1)
|
||||
self.type_combo.addItem("删除列表域名", 2)
|
||||
self.type_combo.setStyleSheet("font-size: 14px; padding: 5px; border: 1px solid #ddd; border-radius: 4px;")
|
||||
# 监听类型变化
|
||||
self.type_combo.currentIndexChanged.connect(self.on_type_changed)
|
||||
type_layout.addWidget(type_label)
|
||||
type_layout.addWidget(self.type_combo)
|
||||
type_layout.setContentsMargins(0, 0, 0, 15)
|
||||
layout.addLayout(type_layout)
|
||||
|
||||
# 页码和每页数量设置
|
||||
page_layout = QHBoxLayout()
|
||||
|
||||
# 起始页码
|
||||
page_start_label = QLabel("起始页码:")
|
||||
page_start_label.setStyleSheet("font-size: 14px; color: #666; min-width: 80px;")
|
||||
self.page_start_edit = QLineEdit("1")
|
||||
# 移除所有限制,允许输入任意正整数
|
||||
self.page_start_edit.setStyleSheet("font-size: 14px; padding: 5px; border: 1px solid #ddd; border-radius: 4px; width: 120px;")
|
||||
page_layout.addWidget(page_start_label)
|
||||
page_layout.addWidget(self.page_start_edit)
|
||||
|
||||
# 每页数量
|
||||
page_size_label = QLabel("每页数量:")
|
||||
page_size_label.setStyleSheet("font-size: 14px; color: #666; min-width: 80px; margin-left: 20px;")
|
||||
self.page_size_edit = QLineEdit("500")
|
||||
self.page_size_edit.setValidator(QIntValidator(1, 1000))
|
||||
self.page_size_edit.setStyleSheet("font-size: 14px; padding: 5px; border: 1px solid #ddd; border-radius: 4px; width: 80px;")
|
||||
page_layout.addWidget(page_size_label)
|
||||
page_layout.addWidget(self.page_size_edit)
|
||||
page_layout.setContentsMargins(0, 0, 0, 20)
|
||||
layout.addLayout(page_layout)
|
||||
|
||||
# 日期设置(删除列表用)
|
||||
date_container = QWidget()
|
||||
date_layout = QHBoxLayout(date_container)
|
||||
|
||||
# 起始日期
|
||||
date_label = QLabel("起始日期:")
|
||||
date_label.setStyleSheet("font-size: 14px; color: #666; min-width: 80px;")
|
||||
self.date_edit = QDateEdit()
|
||||
self.date_edit.setDate(QDate.currentDate())
|
||||
self.date_edit.setCalendarPopup(True)
|
||||
self.date_edit.setStyleSheet("font-size: 14px; padding: 5px; border: 1px solid #ddd; border-radius: 4px; width: 150px;")
|
||||
# 设置最大日期为今天+4天,最小日期为今天的前4天
|
||||
max_date = QDate.currentDate().addDays(4)
|
||||
min_date = QDate.currentDate().addDays(-4)
|
||||
self.date_edit.setMinimumDate(min_date)
|
||||
self.date_edit.setMaximumDate(max_date)
|
||||
date_layout.addWidget(date_label)
|
||||
date_layout.addWidget(self.date_edit)
|
||||
|
||||
# 自动新增日期选项
|
||||
auto_date_checkbox = QCheckBox("自动新增日期")
|
||||
auto_date_checkbox.setChecked(True)
|
||||
auto_date_checkbox.setStyleSheet("font-size: 14px; margin-left: 20px;")
|
||||
self.auto_date_checkbox = auto_date_checkbox
|
||||
date_layout.addWidget(auto_date_checkbox)
|
||||
|
||||
date_layout.setContentsMargins(0, 0, 0, 20)
|
||||
self.date_container = date_container
|
||||
layout.addWidget(date_container)
|
||||
# 默认隐藏日期输入框
|
||||
self.date_container.setVisible(False)
|
||||
|
||||
# 按钮布局
|
||||
button_layout = QHBoxLayout()
|
||||
|
||||
# 创建开始按钮
|
||||
self.start_btn = QPushButton("开始爬取")
|
||||
self.start_btn.clicked.connect(self.start_crawl)
|
||||
self.start_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #4CAF50;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #45a049;
|
||||
}
|
||||
QPushButton:disabled {
|
||||
background-color: #cccccc;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.start_btn)
|
||||
|
||||
# 创建暂停按钮
|
||||
self.pause_btn = QPushButton("暂停爬取")
|
||||
self.pause_btn.clicked.connect(self.pause_crawl)
|
||||
self.pause_btn.setEnabled(False)
|
||||
self.pause_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #ff9800;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #f57c00;
|
||||
}
|
||||
QPushButton:disabled {
|
||||
background-color: #cccccc;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.pause_btn)
|
||||
|
||||
# 创建停止按钮
|
||||
self.stop_btn = QPushButton("停止爬取")
|
||||
self.stop_btn.clicked.connect(self.stop_crawl)
|
||||
self.stop_btn.setEnabled(False)
|
||||
self.stop_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #f44336;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #d32f2f;
|
||||
}
|
||||
QPushButton:disabled {
|
||||
background-color: #cccccc;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.stop_btn)
|
||||
button_layout.setContentsMargins(0, 0, 0, 20)
|
||||
layout.addLayout(button_layout)
|
||||
|
||||
# 创建日志显示区域
|
||||
self.log_edit = QTextEdit()
|
||||
self.log_edit.setPlaceholderText("爬取日志将显示在这里")
|
||||
self.log_edit.setReadOnly(True)
|
||||
self.log_edit.setStyleSheet("""
|
||||
QTextEdit {
|
||||
font-size: 13px;
|
||||
font-family: Consolas, Monaco, monospace;
|
||||
padding: 10px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
background-color: #f9f9f9;
|
||||
}
|
||||
""")
|
||||
layout.addWidget(self.log_edit)
|
||||
|
||||
# 创建进度条
|
||||
self.progress_bar = QProgressBar()
|
||||
self.progress_bar.setVisible(False)
|
||||
self.progress_bar.setStyleSheet("""
|
||||
QProgressBar {
|
||||
height: 20px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 10px;
|
||||
background-color: #f0f0f0;
|
||||
margin-top: 10px;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
QProgressBar::chunk {
|
||||
background-color: #4CAF50;
|
||||
border-radius: 10px;
|
||||
}
|
||||
""")
|
||||
layout.addWidget(self.progress_bar)
|
||||
|
||||
# 创建状态标签
|
||||
self.status_label = QLabel("")
|
||||
self.status_label.setAlignment(Qt.AlignCenter)
|
||||
self.status_label.setStyleSheet("font-size: 14px; color: #333; margin-top: 10px; padding: 8px; background-color: #f0f8ff; border-radius: 4px;")
|
||||
layout.addWidget(self.status_label)
|
||||
|
||||
# 存储爬取线程
|
||||
self.crawl_thread = None
|
||||
|
||||
logger.info("聚名网爬取页面创建完成")
|
||||
|
||||
def start_crawl(self):
|
||||
"""
|
||||
开始爬取
|
||||
"""
|
||||
crawl_type = self.type_combo.currentData()
|
||||
|
||||
# 获取起始页码和每页数量
|
||||
try:
|
||||
page_start = int(self.page_start_edit.text())
|
||||
page_size = int(self.page_size_edit.text())
|
||||
if page_start < 1:
|
||||
self.status_label.setText("起始页码必须大于0")
|
||||
return
|
||||
if page_size < 1:
|
||||
self.status_label.setText("每页数量必须大于0")
|
||||
return
|
||||
except ValueError:
|
||||
self.status_label.setText("请输入有效的页码和每页数量")
|
||||
return
|
||||
|
||||
# 显示进度条
|
||||
self.progress_bar.setVisible(True)
|
||||
self.progress_bar.setValue(0)
|
||||
self.status_label.setText("正在爬取域名...")
|
||||
|
||||
# 清空日志
|
||||
self.log_edit.clear()
|
||||
|
||||
# 启用/禁用按钮
|
||||
self.start_btn.setEnabled(False)
|
||||
self.pause_btn.setEnabled(True)
|
||||
self.stop_btn.setEnabled(True)
|
||||
|
||||
# 获取爬取日期和自动新增日期选项(删除列表用)
|
||||
crawl_date = None
|
||||
auto_date = False
|
||||
if crawl_type == 2: # 删除列表
|
||||
crawl_date = self.date_edit.date().toString("yyyy-MM-dd")
|
||||
auto_date = self.auto_date_checkbox.isChecked()
|
||||
|
||||
# 创建并启动爬取线程
|
||||
self.crawl_thread = JumingCrawlThread(crawl_type, page_start, page_size, crawl_date, auto_date)
|
||||
self.crawl_thread.progress_updated.connect(self.update_progress)
|
||||
self.crawl_thread.status_updated.connect(self.update_status)
|
||||
self.crawl_thread.finished.connect(self.crawl_finished)
|
||||
self.crawl_thread.start()
|
||||
|
||||
if crawl_type == 2: # 删除列表
|
||||
auto_date_str = "是" if auto_date else "否"
|
||||
logger.info(f"开始爬取聚名网域名, 类型: {crawl_type}, 爬取日期: {crawl_date}, 自动新增日期: {auto_date_str}")
|
||||
self.log_edit.append(f"开始爬取聚名网域名, 类型: {crawl_type}, 爬取日期: {crawl_date}, 自动新增日期: {auto_date_str}")
|
||||
else: # 一口价
|
||||
logger.info(f"开始爬取聚名网域名, 类型: {crawl_type}, 起始页码: {page_start}, 每页数量: {page_size}")
|
||||
self.log_edit.append(f"开始爬取聚名网域名, 类型: {crawl_type}, 起始页码: {page_start}, 每页数量: {page_size}")
|
||||
|
||||
def pause_crawl(self):
|
||||
"""
|
||||
暂停爬取
|
||||
"""
|
||||
if self.crawl_thread:
|
||||
if self.crawl_thread.is_paused:
|
||||
self.crawl_thread.resume()
|
||||
self.pause_btn.setText("暂停爬取")
|
||||
else:
|
||||
self.crawl_thread.pause()
|
||||
self.pause_btn.setText("恢复爬取")
|
||||
|
||||
def stop_crawl(self):
|
||||
"""
|
||||
停止爬取
|
||||
"""
|
||||
if self.crawl_thread:
|
||||
self.crawl_thread.stop()
|
||||
|
||||
def update_progress(self, progress):
|
||||
"""
|
||||
更新进度
|
||||
|
||||
:param progress: 进度值
|
||||
"""
|
||||
self.progress_bar.setValue(progress)
|
||||
|
||||
def update_status(self, status):
|
||||
"""
|
||||
更新状态
|
||||
|
||||
:param status: 状态消息
|
||||
"""
|
||||
self.status_label.setText(status)
|
||||
self.log_edit.append(status)
|
||||
|
||||
def crawl_finished(self, success, message):
|
||||
"""
|
||||
爬取完成
|
||||
|
||||
:param success: 是否成功
|
||||
:param message: 消息
|
||||
"""
|
||||
self.status_label.setText(message)
|
||||
self.log_edit.append(message)
|
||||
|
||||
# 启用/禁用按钮
|
||||
self.start_btn.setEnabled(True)
|
||||
self.pause_btn.setEnabled(False)
|
||||
self.pause_btn.setText("暂停爬取")
|
||||
self.stop_btn.setEnabled(False)
|
||||
|
||||
self.progress_bar.setVisible(False)
|
||||
|
||||
logger.info(f"聚名网爬取完成: {message}")
|
||||
|
||||
def on_type_changed(self, index):
|
||||
"""
|
||||
爬取类型变化时的处理
|
||||
|
||||
:param index: 选择的索引
|
||||
"""
|
||||
crawl_type = self.type_combo.currentData()
|
||||
if crawl_type == 2: # 删除列表
|
||||
self.date_container.setVisible(True)
|
||||
else: # 一口价
|
||||
self.date_container.setVisible(False)
|
||||
|
||||
|
||||
class ImportThread(QThread):
|
||||
"""
|
||||
导入线程
|
||||
"""
|
||||
progress_updated = Signal(int)
|
||||
finished = Signal(bool, str)
|
||||
|
||||
def __init__(self, domain_list, source_type):
|
||||
"""
|
||||
初始化导入线程
|
||||
|
||||
:param domain_list: 域名列表
|
||||
:param source_type: 来源类型
|
||||
"""
|
||||
super().__init__()
|
||||
self.domain_list = domain_list
|
||||
self.source_type = source_type
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
运行导入线程
|
||||
"""
|
||||
try:
|
||||
collector = DomainCollector()
|
||||
total = len(self.domain_list)
|
||||
for i, domain in enumerate(self.domain_list):
|
||||
collector.add_domain(domain, self.source_type)
|
||||
progress = int((i + 1) / total * 100)
|
||||
self.progress_updated.emit(progress)
|
||||
self.finished.emit(True, f"成功导入 {total} 个域名")
|
||||
except Exception as e:
|
||||
logger.error(f"导入失败: {e}")
|
||||
self.finished.emit(False, f"导入失败: {str(e)}")
|
||||
137
domainCheck/app/ui/main_window.py
Normal file
137
domainCheck/app/ui/main_window.py
Normal file
@@ -0,0 +1,137 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :main_window.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:46
|
||||
@explain : 主窗口
|
||||
'''
|
||||
|
||||
from PySide6.QtWidgets import QMainWindow, QWidget, QVBoxLayout, QHBoxLayout, QPushButton, QTabWidget, QLabel, QScrollArea, QFrame
|
||||
from PySide6.QtCore import Qt
|
||||
from PySide6.QtGui import QIcon
|
||||
import os
|
||||
from loguru import logger
|
||||
|
||||
from app.ui.domain_import import DomainImportWidget
|
||||
from app.ui.domain_filter import DomainFilterWidget
|
||||
from app.ui.sensitive_words import SensitiveWordsWidget
|
||||
from app.ui.juming_crawler import JumingCrawlerWidget
|
||||
from app.ui.system_settings import SystemSettingsWidget
|
||||
|
||||
|
||||
class MainWindow(QMainWindow):
|
||||
"""
|
||||
主窗口
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化主窗口
|
||||
"""
|
||||
super().__init__()
|
||||
|
||||
# 设置窗口标题和大小
|
||||
self.setWindowTitle("域名工具")
|
||||
self.setGeometry(100, 100, 2000, 800)
|
||||
self.setMinimumSize(1200, 720)
|
||||
|
||||
# 设置窗口图标
|
||||
icon_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), "favicon.ico")
|
||||
if os.path.exists(icon_path):
|
||||
self.setWindowIcon(QIcon(icon_path))
|
||||
logger.info(f"设置窗口图标成功: {icon_path}")
|
||||
else:
|
||||
logger.warning(f"窗口图标文件不存在: {icon_path}")
|
||||
|
||||
# 创建中央部件
|
||||
central_widget = QWidget()
|
||||
self.setCentralWidget(central_widget)
|
||||
|
||||
# 创建主布局
|
||||
main_layout = QVBoxLayout(central_widget)
|
||||
|
||||
# 创建标签页
|
||||
self.tab_widget = QTabWidget()
|
||||
self.tab_widget.setStyleSheet("""
|
||||
QTabWidget {
|
||||
font-size: 14px;
|
||||
}
|
||||
QTabBar::tab {
|
||||
padding: 10px 20px;
|
||||
background-color: #f0f0f0;
|
||||
border: 1px solid #ddd;
|
||||
border-bottom: none;
|
||||
border-top-left-radius: 4px;
|
||||
border-top-right-radius: 4px;
|
||||
margin-right: 2px;
|
||||
}
|
||||
QTabBar::tab:hover {
|
||||
background-color: #e0e0e0;
|
||||
}
|
||||
QTabBar::tab:selected {
|
||||
background-color: white;
|
||||
color: #4CAF50;
|
||||
font-weight: bold;
|
||||
border-color: #4CAF50;
|
||||
}
|
||||
QTabWidget::pane {
|
||||
border: 1px solid #ddd;
|
||||
border-top: none;
|
||||
border-radius: 0 0 4px 4px;
|
||||
padding: 10px;
|
||||
}
|
||||
""")
|
||||
main_layout.addWidget(self.tab_widget)
|
||||
|
||||
# 创建标签页内容
|
||||
self.create_tabs()
|
||||
|
||||
# 记录日志
|
||||
logger.info("主窗口创建完成")
|
||||
|
||||
def create_tabs(self):
|
||||
"""
|
||||
创建标签页
|
||||
"""
|
||||
# 聚名爬取标签页
|
||||
juming_widget = JumingCrawlerWidget()
|
||||
self.tab_widget.addTab(self.wrap_scrollable_tab(juming_widget), "聚名爬取")
|
||||
|
||||
# 域名筛选标签页
|
||||
filter_widget = DomainFilterWidget()
|
||||
self.tab_widget.addTab(self.wrap_scrollable_tab(filter_widget), "域名筛选")
|
||||
|
||||
# 域名导入标签页
|
||||
import_widget = DomainImportWidget()
|
||||
self.tab_widget.addTab(self.wrap_scrollable_tab(import_widget), "域名导入")
|
||||
|
||||
# 敏感词配置标签页
|
||||
sensitive_widget = SensitiveWordsWidget()
|
||||
self.tab_widget.addTab(self.wrap_scrollable_tab(sensitive_widget), "敏感词配置")
|
||||
|
||||
# 系统设置标签页
|
||||
settings_widget = SystemSettingsWidget()
|
||||
self.tab_widget.addTab(self.wrap_scrollable_tab(settings_widget), "系统设置")
|
||||
|
||||
logger.info("标签页创建完成")
|
||||
|
||||
def wrap_scrollable_tab(self, widget):
|
||||
"""
|
||||
给标签页统一包一层滚动区域,保证右侧滚动条始终可用
|
||||
"""
|
||||
container = QWidget()
|
||||
container_layout = QVBoxLayout(container)
|
||||
container_layout.setContentsMargins(0, 0, 0, 0)
|
||||
container_layout.setSpacing(0)
|
||||
|
||||
scroll_area = QScrollArea()
|
||||
scroll_area.setWidgetResizable(True)
|
||||
scroll_area.setFrameShape(QFrame.NoFrame)
|
||||
scroll_area.setVerticalScrollBarPolicy(Qt.ScrollBarAsNeeded)
|
||||
scroll_area.setHorizontalScrollBarPolicy(Qt.ScrollBarAsNeeded)
|
||||
scroll_area.setWidget(widget)
|
||||
|
||||
container_layout.addWidget(scroll_area)
|
||||
return container
|
||||
284
domainCheck/app/ui/sensitive_words.py
Normal file
284
domainCheck/app/ui/sensitive_words.py
Normal file
@@ -0,0 +1,284 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :sensitive_words.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/8 23:49
|
||||
@explain : 敏感词配置界面
|
||||
'''
|
||||
|
||||
from PySide6.QtWidgets import QWidget, QVBoxLayout, QHBoxLayout, QPushButton, QLabel, QTextEdit, QFileDialog
|
||||
from PySide6.QtCore import Qt, QThread, Signal
|
||||
from loguru import logger
|
||||
from app.utils.database import Database
|
||||
|
||||
|
||||
class SaveWordsThread(QThread):
|
||||
"""
|
||||
保存敏感词线程
|
||||
"""
|
||||
finished = Signal(bool, str, int)
|
||||
|
||||
def __init__(self, words):
|
||||
super().__init__()
|
||||
self.words = words
|
||||
|
||||
def run(self):
|
||||
try:
|
||||
db = Database()
|
||||
|
||||
# 先清空现有敏感词
|
||||
db.execute("DELETE FROM sensitive_words")
|
||||
|
||||
# 批量添加敏感词
|
||||
word_tuples = [(word, 'default', 1) for word in self.words]
|
||||
if word_tuples:
|
||||
db.batch_add_sensitive_words(word_tuples)
|
||||
|
||||
db.close()
|
||||
self.finished.emit(True, "成功保存敏感词", len(self.words))
|
||||
except Exception as e:
|
||||
self.finished.emit(False, str(e), 0)
|
||||
|
||||
|
||||
class LoadWordsThread(QThread):
|
||||
"""
|
||||
加载敏感词线程
|
||||
"""
|
||||
finished = Signal(bool, list, str)
|
||||
|
||||
def run(self):
|
||||
try:
|
||||
db = Database()
|
||||
sensitive_words = db.get_sensitive_words()
|
||||
words = [word['word'] for word in sensitive_words]
|
||||
db.close()
|
||||
self.finished.emit(True, words, f"成功加载 {len(words)} 个敏感词")
|
||||
except Exception as e:
|
||||
self.finished.emit(False, [], str(e))
|
||||
|
||||
|
||||
class SensitiveWordsWidget(QWidget):
|
||||
"""
|
||||
敏感词配置界面
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
初始化敏感词配置界面
|
||||
"""
|
||||
super().__init__()
|
||||
|
||||
# 创建布局
|
||||
layout = QVBoxLayout(self)
|
||||
|
||||
|
||||
|
||||
# 创建文本编辑框
|
||||
self.text_edit = QTextEdit()
|
||||
self.text_edit.setPlaceholderText("请输入敏感词,一行一个")
|
||||
self.text_edit.setStyleSheet("""
|
||||
QTextEdit {
|
||||
font-size: 14px;
|
||||
padding: 10px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
background-color: #f9f9f9;
|
||||
min-height: 300px;
|
||||
}
|
||||
""")
|
||||
layout.addWidget(self.text_edit)
|
||||
|
||||
# 创建按钮布局
|
||||
button_layout = QHBoxLayout()
|
||||
|
||||
# 导入按钮
|
||||
self.import_btn = QPushButton("导入")
|
||||
self.import_btn.clicked.connect(self.import_words)
|
||||
self.import_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #2196F3;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #0b7dda;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.import_btn)
|
||||
|
||||
# 导出按钮
|
||||
self.export_btn = QPushButton("导出")
|
||||
self.export_btn.clicked.connect(self.export_words)
|
||||
self.export_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #ff9800;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #f57c00;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.export_btn)
|
||||
|
||||
# 保存按钮
|
||||
self.save_btn = QPushButton("保存")
|
||||
self.save_btn.clicked.connect(self.save_words)
|
||||
self.save_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #4CAF50;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #45a049;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.save_btn)
|
||||
|
||||
# 加载按钮
|
||||
self.load_btn = QPushButton("加载")
|
||||
self.load_btn.clicked.connect(self.load_words)
|
||||
self.load_btn.setStyleSheet("""
|
||||
QPushButton {
|
||||
font-size: 14px;
|
||||
padding: 8px 16px;
|
||||
background-color: #9c27b0;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
margin-left: 10px;
|
||||
}
|
||||
QPushButton:hover {
|
||||
background-color: #7b1fa2;
|
||||
}
|
||||
""")
|
||||
button_layout.addWidget(self.load_btn)
|
||||
button_layout.setContentsMargins(0, 15, 0, 15)
|
||||
layout.addLayout(button_layout)
|
||||
|
||||
# 创建状态标签
|
||||
self.status_label = QLabel("")
|
||||
self.status_label.setAlignment(Qt.AlignCenter)
|
||||
self.status_label.setStyleSheet("font-size: 14px; color: #333; padding: 10px; background-color: #f0f8ff; border-radius: 4px;")
|
||||
layout.addWidget(self.status_label)
|
||||
|
||||
# 初始化线程
|
||||
self.save_thread = None
|
||||
self.load_thread = None
|
||||
|
||||
# 加载敏感词
|
||||
self.load_words()
|
||||
|
||||
logger.info("敏感词配置界面创建完成")
|
||||
|
||||
def import_words(self):
|
||||
"""
|
||||
导入敏感词
|
||||
"""
|
||||
file_path, _ = QFileDialog.getOpenFileName(self, "选择文件", "", "文本文件 (*.txt)")
|
||||
if file_path:
|
||||
try:
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
words = f.readlines()
|
||||
words = [word.strip() for word in words if word.strip()]
|
||||
self.text_edit.setText('\n'.join(words))
|
||||
self.status_label.setText(f"成功导入 {len(words)} 个敏感词")
|
||||
logger.info(f"成功导入敏感词文件: {file_path}, 共 {len(words)} 个敏感词")
|
||||
except Exception as e:
|
||||
self.status_label.setText(f"导入失败: {str(e)}")
|
||||
logger.error(f"导入敏感词失败: {e}")
|
||||
|
||||
def export_words(self):
|
||||
"""
|
||||
导出敏感词
|
||||
"""
|
||||
words = self.text_edit.toPlainText().split('\n')
|
||||
words = [word.strip() for word in words if word.strip()]
|
||||
|
||||
if not words:
|
||||
self.status_label.setText("没有敏感词可导出")
|
||||
return
|
||||
|
||||
file_path, _ = QFileDialog.getSaveFileName(self, "保存文件", "", "文本文件 (*.txt)")
|
||||
if file_path:
|
||||
try:
|
||||
with open(file_path, 'w', encoding='utf-8') as f:
|
||||
for word in words:
|
||||
f.write(word + '\n')
|
||||
self.status_label.setText(f"成功导出 {len(words)} 个敏感词")
|
||||
logger.info(f"成功导出 {len(words)} 个敏感词到 {file_path}")
|
||||
except Exception as e:
|
||||
self.status_label.setText(f"导出失败: {str(e)}")
|
||||
logger.error(f"导出敏感词失败: {e}")
|
||||
|
||||
def save_words(self):
|
||||
"""
|
||||
保存敏感词
|
||||
"""
|
||||
words = self.text_edit.toPlainText().split('\n')
|
||||
words = [word.strip() for word in words if word.strip()]
|
||||
|
||||
# 禁用按钮,防止重复点击
|
||||
self.save_btn.setEnabled(False)
|
||||
self.status_label.setText("正在保存敏感词...")
|
||||
|
||||
# 创建并启动保存线程
|
||||
self.save_thread = SaveWordsThread(words)
|
||||
self.save_thread.finished.connect(self.on_save_finished)
|
||||
self.save_thread.start()
|
||||
|
||||
def on_save_finished(self, success, message, count):
|
||||
"""
|
||||
保存完成的回调函数
|
||||
"""
|
||||
if success:
|
||||
self.status_label.setText(f"成功保存 {count} 个敏感词")
|
||||
logger.info(f"成功保存 {count} 个敏感词到数据库")
|
||||
else:
|
||||
self.status_label.setText(f"保存失败: {message}")
|
||||
logger.error(f"保存敏感词失败: {message}")
|
||||
|
||||
# 重新启用按钮
|
||||
self.save_btn.setEnabled(True)
|
||||
|
||||
def load_words(self):
|
||||
"""
|
||||
加载敏感词
|
||||
"""
|
||||
# 禁用按钮,防止重复点击
|
||||
self.load_btn.setEnabled(False)
|
||||
self.status_label.setText("正在加载敏感词...")
|
||||
|
||||
# 创建并启动加载线程
|
||||
self.load_thread = LoadWordsThread()
|
||||
self.load_thread.finished.connect(self.on_load_finished)
|
||||
self.load_thread.start()
|
||||
|
||||
def on_load_finished(self, success, words, message):
|
||||
"""
|
||||
加载完成的回调函数
|
||||
"""
|
||||
if success:
|
||||
self.text_edit.setText('\n'.join(words))
|
||||
self.status_label.setText(message)
|
||||
logger.info(message)
|
||||
else:
|
||||
self.status_label.setText(f"加载失败: {message}")
|
||||
logger.error(f"加载敏感词失败: {message}")
|
||||
|
||||
# 重新启用按钮
|
||||
self.load_btn.setEnabled(True)
|
||||
1402
domainCheck/app/ui/system_settings.py
Normal file
1402
domainCheck/app/ui/system_settings.py
Normal file
File diff suppressed because it is too large
Load Diff
4
domainCheck/app/utils/__init__.py
Normal file
4
domainCheck/app/utils/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
工具类模块
|
||||
'''
|
||||
1499
domainCheck/app/utils/database.py
Normal file
1499
domainCheck/app/utils/database.py
Normal file
File diff suppressed because it is too large
Load Diff
195
domainCheck/app/utils/domain_utils.py
Normal file
195
domainCheck/app/utils/domain_utils.py
Normal file
@@ -0,0 +1,195 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :domain_utils.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:05
|
||||
@explain : 域名工具类
|
||||
'''
|
||||
|
||||
import re
|
||||
import tldextract
|
||||
from loguru import logger
|
||||
from app.utils.status_codes import DETECT_STATUS_LABELS, REGISTER_STATUS_LABELS, USE_STATUS_LABELS
|
||||
|
||||
|
||||
def normalize_domain(domain):
|
||||
"""
|
||||
标准化域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: str - 标准化后的域名
|
||||
"""
|
||||
try:
|
||||
# 转换为小写
|
||||
domain = domain.lower()
|
||||
|
||||
# 去除空格
|
||||
domain = domain.strip()
|
||||
|
||||
# 去除协议
|
||||
domain = re.sub(r'^https?://', '', domain)
|
||||
|
||||
# 去除路径和查询参数
|
||||
domain = domain.split('/')[0]
|
||||
domain = domain.split('?')[0]
|
||||
|
||||
# 去除端口
|
||||
domain = domain.split(':')[0]
|
||||
|
||||
# 只保留主域
|
||||
ext = tldextract.extract(domain)
|
||||
if ext.domain and ext.suffix:
|
||||
domain = f"{ext.domain}.{ext.suffix}"
|
||||
|
||||
# 验证域名格式
|
||||
if not is_valid_domain(domain):
|
||||
return None
|
||||
|
||||
return domain
|
||||
except Exception as e:
|
||||
logger.error(f"标准化域名出错: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def is_valid_domain(domain):
|
||||
"""
|
||||
验证域名格式
|
||||
|
||||
:param domain: 域名
|
||||
:return: bool - 是否有效
|
||||
"""
|
||||
try:
|
||||
# 域名格式正则
|
||||
pattern = r'^[a-zA-Z0-9][a-zA-Z0-9-]{0,61}[a-zA-Z0-9]\.[a-zA-Z]{2,}$'
|
||||
return bool(re.match(pattern, domain))
|
||||
except Exception as e:
|
||||
logger.error(f"验证域名格式出错: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def extract_tld(domain):
|
||||
"""
|
||||
提取顶级域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: str - 顶级域名
|
||||
"""
|
||||
try:
|
||||
ext = tldextract.extract(domain)
|
||||
return ext.suffix
|
||||
except Exception as e:
|
||||
logger.error(f"提取顶级域名出错: {e}")
|
||||
return ''
|
||||
|
||||
|
||||
def extract_domain(domain):
|
||||
"""
|
||||
提取主域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: str - 主域名
|
||||
"""
|
||||
try:
|
||||
ext = tldextract.extract(domain)
|
||||
if ext.domain and ext.suffix:
|
||||
return f"{ext.domain}.{ext.suffix}"
|
||||
return domain
|
||||
except Exception as e:
|
||||
logger.error(f"提取主域名出错: {e}")
|
||||
return domain
|
||||
|
||||
|
||||
def is_com_or_net(domain):
|
||||
"""
|
||||
检查是否为 .com 或 .net 域名
|
||||
|
||||
:param domain: 域名
|
||||
:return: bool - 是否为 .com 或 .net 域名
|
||||
"""
|
||||
try:
|
||||
tld = extract_tld(domain)
|
||||
return tld in ['com', 'net']
|
||||
except Exception as e:
|
||||
logger.error(f"检查域名后缀出错: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def generate_domain_variants(domain):
|
||||
"""
|
||||
生成域名变体
|
||||
|
||||
:param domain: 域名
|
||||
:return: list - 域名变体列表
|
||||
"""
|
||||
try:
|
||||
variants = []
|
||||
|
||||
# 原始域名
|
||||
variants.append(domain)
|
||||
|
||||
# 添加 www
|
||||
if not domain.startswith('www.'):
|
||||
variants.append(f"www.{domain}")
|
||||
|
||||
# 移除 www
|
||||
if domain.startswith('www.'):
|
||||
variants.append(domain[4:])
|
||||
|
||||
return variants
|
||||
except Exception as e:
|
||||
logger.error(f"生成域名变体出错: {e}")
|
||||
return [domain]
|
||||
|
||||
|
||||
def parse_domain_status(status_code):
|
||||
"""
|
||||
解析域名状态码
|
||||
|
||||
:param status_code: 状态码
|
||||
:return: str - 状态描述
|
||||
"""
|
||||
return REGISTER_STATUS_LABELS.get(status_code, '未知')
|
||||
|
||||
|
||||
def parse_use_status(status_code):
|
||||
"""
|
||||
解析使用状态码
|
||||
|
||||
:param status_code: 状态码
|
||||
:return: str - 状态描述
|
||||
"""
|
||||
return USE_STATUS_LABELS.get(status_code, '未知')
|
||||
|
||||
|
||||
def parse_detect_status(status_code):
|
||||
"""
|
||||
解析检测状态码
|
||||
|
||||
:param status_code: 状态码
|
||||
:return: str - 状态描述
|
||||
"""
|
||||
return DETECT_STATUS_LABELS.get(status_code, '未知')
|
||||
|
||||
|
||||
def parse_source_type(source_type):
|
||||
"""
|
||||
解析来源类型
|
||||
|
||||
:param source_type: 来源类型
|
||||
:return: str - 来源描述
|
||||
"""
|
||||
source_map = {
|
||||
1: '聚名一口价',
|
||||
2: '聚名过期删除',
|
||||
3: 'zone file',
|
||||
4: '搜索引擎采集',
|
||||
5: '企业目录采集',
|
||||
6: '手工录入',
|
||||
7: 'TXT 导入',
|
||||
8: '第三方接口',
|
||||
9: '其它'
|
||||
}
|
||||
|
||||
return source_map.get(source_type, '未知')
|
||||
157
domainCheck/app/utils/http_utils.py
Normal file
157
domainCheck/app/utils/http_utils.py
Normal file
@@ -0,0 +1,157 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :http_utils.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 0:06
|
||||
@explain : HTTP工具类
|
||||
'''
|
||||
|
||||
import requests
|
||||
from curl_cffi import requests as curl_requests
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class HTTPUtils:
|
||||
"""
|
||||
HTTP工具类
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def get(url, headers=None, params=None, timeout=10, proxies=None, use_curl=False):
|
||||
"""
|
||||
发送GET请求
|
||||
|
||||
:param url: 请求URL
|
||||
:param headers: 请求头
|
||||
:param params: 查询参数
|
||||
:param timeout: 超时时间
|
||||
:param proxies: 代理
|
||||
:param use_curl: 是否使用curl_cffi
|
||||
:return: requests.Response - 响应对象
|
||||
"""
|
||||
try:
|
||||
if use_curl:
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.get(url, headers=headers, params=params, timeout=timeout, proxies=proxies, impersonate='chrome')
|
||||
else:
|
||||
# 使用requests
|
||||
response = requests.get(url, headers=headers, params=params, timeout=timeout, proxies=proxies)
|
||||
|
||||
response.raise_for_status() # 检查状态码
|
||||
return response
|
||||
except Exception as e:
|
||||
logger.error(f"GET请求失败: {url}, 错误: {e}")
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def post(url, headers=None, data=None, json=None, timeout=10, proxies=None, use_curl=False):
|
||||
"""
|
||||
发送POST请求
|
||||
|
||||
:param url: 请求URL
|
||||
:param headers: 请求头
|
||||
:param data: 表单数据
|
||||
:param json: JSON数据
|
||||
:param timeout: 超时时间
|
||||
:param proxies: 代理
|
||||
:param use_curl: 是否使用curl_cffi
|
||||
:return: requests.Response - 响应对象
|
||||
"""
|
||||
try:
|
||||
if use_curl:
|
||||
# 使用curl_cffi模拟浏览器
|
||||
response = curl_requests.post(url, headers=headers, data=data, json=json, timeout=timeout, proxies=proxies, impersonate='chrome')
|
||||
else:
|
||||
# 使用requests
|
||||
response = requests.post(url, headers=headers, data=data, json=json, timeout=timeout, proxies=proxies)
|
||||
|
||||
response.raise_for_status() # 检查状态码
|
||||
return response
|
||||
except Exception as e:
|
||||
logger.error(f"POST请求失败: {url}, 错误: {e}")
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def get_random_user_agent():
|
||||
"""
|
||||
获取随机用户代理
|
||||
|
||||
:return: str - 用户代理
|
||||
"""
|
||||
user_agents = [
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Firefox/138.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Edge/146.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.6 Safari/605.1.15'
|
||||
]
|
||||
|
||||
import random
|
||||
return random.choice(user_agents)
|
||||
|
||||
@staticmethod
|
||||
def get_default_headers():
|
||||
"""
|
||||
获取默认请求头
|
||||
|
||||
:return: dict - 请求头
|
||||
"""
|
||||
return {
|
||||
'User-Agent': HTTPUtils.get_random_user_agent(),
|
||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
|
||||
'Accept-Language': 'zh-CN,zh;q=0.9',
|
||||
'Accept-Encoding': 'gzip, deflate, br',
|
||||
'Connection': 'keep-alive',
|
||||
'Upgrade-Insecure-Requests': '1',
|
||||
'Sec-Fetch-Dest': 'document',
|
||||
'Sec-Fetch-Mode': 'navigate',
|
||||
'Sec-Fetch-Site': 'none',
|
||||
'Sec-Fetch-User': '?1'
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def retry_request(func, max_retries=3, delay=1):
|
||||
"""
|
||||
重试请求
|
||||
|
||||
:param func: 请求函数
|
||||
:param max_retries: 最大重试次数
|
||||
:param delay: 重试延迟
|
||||
:return: 函数返回值
|
||||
"""
|
||||
import time
|
||||
|
||||
for i in range(max_retries):
|
||||
try:
|
||||
result = func()
|
||||
if result:
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.warning(f"请求失败,第 {i+1} 次重试: {e}")
|
||||
|
||||
if i < max_retries - 1:
|
||||
time.sleep(delay)
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def check_proxy(proxy):
|
||||
"""
|
||||
检查代理是否可用
|
||||
|
||||
:param proxy: 代理URL
|
||||
:return: bool - 是否可用
|
||||
"""
|
||||
try:
|
||||
proxies = {
|
||||
'http': proxy,
|
||||
'https': proxy
|
||||
}
|
||||
|
||||
response = requests.get('https://www.baidu.com', proxies=proxies, timeout=5)
|
||||
return response.status_code == 200
|
||||
except Exception as e:
|
||||
logger.error(f"代理检查失败: {proxy}, 错误: {e}")
|
||||
return False
|
||||
66
domainCheck/app/utils/status_codes.py
Normal file
66
domainCheck/app/utils/status_codes.py
Normal file
@@ -0,0 +1,66 @@
|
||||
"""Centralized status codes and labels used across detection, database and UI."""
|
||||
|
||||
REGISTER_STATUS_PENDING = 0
|
||||
REGISTER_STATUS_AVAILABLE = 2
|
||||
REGISTER_STATUS_REGISTERED = 3
|
||||
REGISTER_STATUS_GRACE = 4
|
||||
REGISTER_STATUS_REDEMPTION = 5
|
||||
REGISTER_STATUS_PENDING_DELETE = 6
|
||||
REGISTER_STATUS_CLIENT_HOLD = 7
|
||||
REGISTER_STATUS_SERVER_HOLD = 8
|
||||
REGISTER_STATUS_UNKNOWN = 9
|
||||
REGISTER_STATUS_FAILED = 10
|
||||
|
||||
DETECT_STATUS_PENDING = 0
|
||||
DETECT_STATUS_COMPLETED = 1
|
||||
DETECT_STATUS_RUNNING = 2
|
||||
DETECT_STATUS_BLACKLISTED = 3
|
||||
DETECT_STATUS_FAILED = 4
|
||||
|
||||
USE_STATUS_UNUSED = 0
|
||||
USE_STATUS_USED = 1
|
||||
USE_STATUS_SOLD = 2
|
||||
USE_STATUS_RESERVED = 3
|
||||
|
||||
REVIEW_STATUS_NONE = 0
|
||||
REVIEW_STATUS_PENDING = 1
|
||||
REVIEW_STATUS_APPROVED = 2
|
||||
REVIEW_STATUS_REJECTED = 3
|
||||
|
||||
THIRD_PARTY_STATUS_PENDING = 0
|
||||
THIRD_PARTY_STATUS_DONE = 1
|
||||
|
||||
REGISTER_STATUS_LABELS = {
|
||||
REGISTER_STATUS_PENDING: '待检测',
|
||||
REGISTER_STATUS_AVAILABLE: '可注册',
|
||||
REGISTER_STATUS_REGISTERED: '已注册',
|
||||
REGISTER_STATUS_GRACE: '宽限期',
|
||||
REGISTER_STATUS_REDEMPTION: '赎回期',
|
||||
REGISTER_STATUS_PENDING_DELETE: '删除期',
|
||||
REGISTER_STATUS_CLIENT_HOLD: 'clientHold',
|
||||
REGISTER_STATUS_SERVER_HOLD: 'serverHold',
|
||||
REGISTER_STATUS_UNKNOWN: '状态未知',
|
||||
REGISTER_STATUS_FAILED: '检测失败',
|
||||
}
|
||||
|
||||
DETECT_STATUS_LABELS = {
|
||||
DETECT_STATUS_PENDING: '待检测',
|
||||
DETECT_STATUS_COMPLETED: '检测完成',
|
||||
DETECT_STATUS_RUNNING: '检测中',
|
||||
DETECT_STATUS_BLACKLISTED: '黑名单',
|
||||
DETECT_STATUS_FAILED: '检测失败',
|
||||
}
|
||||
|
||||
USE_STATUS_LABELS = {
|
||||
USE_STATUS_UNUSED: '未使用',
|
||||
USE_STATUS_USED: '已经使用',
|
||||
USE_STATUS_SOLD: '已经卖出',
|
||||
USE_STATUS_RESERVED: '已经预定',
|
||||
}
|
||||
|
||||
REVIEW_STATUS_LABELS = {
|
||||
REVIEW_STATUS_NONE: '无需复核',
|
||||
REVIEW_STATUS_PENDING: '待人工复核',
|
||||
REVIEW_STATUS_APPROVED: '人工通过',
|
||||
REVIEW_STATUS_REJECTED: '人工拒绝',
|
||||
}
|
||||
74
domainCheck/check_database.py
Normal file
74
domainCheck/check_database.py
Normal file
@@ -0,0 +1,74 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :check_database.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 21:40
|
||||
@explain : 检查数据库表结构和注释
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from app.config import config
|
||||
|
||||
|
||||
def check_database():
|
||||
"""
|
||||
检查数据库表结构和注释
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
print("数据库连接成功")
|
||||
|
||||
# 检查所有表
|
||||
print("\n=== 所有表 ===")
|
||||
cur.execute("SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'")
|
||||
tables = cur.fetchall()
|
||||
for table in tables:
|
||||
print(f"表名: {table[0]}")
|
||||
|
||||
# 检查domains表结构
|
||||
print("\n=== domains表结构 ===")
|
||||
cur.execute("\d+ domains.txt")
|
||||
result = cur.fetchall()
|
||||
for row in result:
|
||||
print(row)
|
||||
|
||||
# 检查其他表结构
|
||||
print("\n=== detect_tasks表结构 ===")
|
||||
cur.execute("\d+ detect_tasks")
|
||||
result = cur.fetchall()
|
||||
for row in result:
|
||||
print(row)
|
||||
|
||||
print("\n=== domain_blacklist表结构 ===")
|
||||
cur.execute("\d+ domain_blacklist")
|
||||
result = cur.fetchall()
|
||||
for row in result:
|
||||
print(row)
|
||||
|
||||
print("\n=== domain_detections表结构 ===")
|
||||
cur.execute("\d+ domain_detections")
|
||||
result = cur.fetchall()
|
||||
for row in result:
|
||||
print(row)
|
||||
|
||||
except Exception as e:
|
||||
print(f"检查数据库失败: {e}")
|
||||
finally:
|
||||
if cur:
|
||||
cur.close()
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
check_database()
|
||||
69
domainCheck/check_db_structure.py
Normal file
69
domainCheck/check_db_structure.py
Normal file
@@ -0,0 +1,69 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :check_db_structure.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 21:50
|
||||
@explain : 检查数据库表结构和索引
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from app.config import config
|
||||
|
||||
|
||||
def check_db_structure():
|
||||
"""
|
||||
检查数据库表结构和索引
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
print("数据库连接成功")
|
||||
|
||||
# 检查domains表结构
|
||||
print("\n=== domains表结构 ===")
|
||||
cur.execute("SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns WHERE table_name = 'domains.txt'")
|
||||
for row in cur.fetchall():
|
||||
print(row)
|
||||
|
||||
# 检查domains表索引
|
||||
print("\n=== domains表索引 ===")
|
||||
cur.execute("SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'domains.txt'")
|
||||
for row in cur.fetchall():
|
||||
print(row)
|
||||
|
||||
# 检查其他表结构
|
||||
print("\n=== detect_tasks表结构 ===")
|
||||
cur.execute("SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns WHERE table_name = 'detect_tasks'")
|
||||
for row in cur.fetchall():
|
||||
print(row)
|
||||
|
||||
print("\n=== domain_blacklist表结构 ===")
|
||||
cur.execute("SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns WHERE table_name = 'domain_blacklist'")
|
||||
for row in cur.fetchall():
|
||||
print(row)
|
||||
|
||||
print("\n=== domain_detections表结构 ===")
|
||||
cur.execute("SELECT column_name, data_type, is_nullable, column_default FROM information_schema.columns WHERE table_name = 'domain_detections'")
|
||||
for row in cur.fetchall():
|
||||
print(row)
|
||||
|
||||
except Exception as e:
|
||||
print(f"检查数据库失败: {e}")
|
||||
finally:
|
||||
if cur:
|
||||
cur.close()
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
check_db_structure()
|
||||
128
domainCheck/create_sensitive_words_table.py
Normal file
128
domainCheck/create_sensitive_words_table.py
Normal file
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :create_sensitive_words_table.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/10 03:30
|
||||
@explain : 创建敏感词表
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from app.config import config
|
||||
|
||||
|
||||
def create_sensitive_words_table():
|
||||
"""
|
||||
创建敏感词表
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
print(f"成功连接到数据库: {config.DB_HOST}:{config.DB_PORT}/{config.DB_DATABASE}")
|
||||
|
||||
# 创建游标
|
||||
cur = conn.cursor()
|
||||
|
||||
# 创建敏感词表
|
||||
create_table_sql = """
|
||||
CREATE TABLE IF NOT EXISTS sensitive_words (
|
||||
id SERIAL PRIMARY KEY,
|
||||
word VARCHAR(255) UNIQUE NOT NULL,
|
||||
category VARCHAR(50) DEFAULT 'default',
|
||||
priority INTEGER DEFAULT 1,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
"""
|
||||
cur.execute(create_table_sql)
|
||||
print("创建敏感词表成功")
|
||||
|
||||
# 添加表注释
|
||||
add_table_comment_sql = """
|
||||
COMMENT ON TABLE sensitive_words IS '敏感词表,存储需要过滤的敏感词'
|
||||
"""
|
||||
cur.execute(add_table_comment_sql)
|
||||
print("添加表注释成功")
|
||||
|
||||
# 添加字段注释
|
||||
add_column_comments_sql = """
|
||||
COMMENT ON COLUMN sensitive_words.id IS '主键ID';
|
||||
COMMENT ON COLUMN sensitive_words.word IS '敏感词';
|
||||
COMMENT ON COLUMN sensitive_words.category IS '敏感词分类';
|
||||
COMMENT ON COLUMN sensitive_words.priority IS '优先级,数字越大优先级越高';
|
||||
COMMENT ON COLUMN sensitive_words.created_at IS '创建时间';
|
||||
COMMENT ON COLUMN sensitive_words.updated_at IS '更新时间';
|
||||
"""
|
||||
cur.execute(add_column_comments_sql)
|
||||
print("添加字段注释成功")
|
||||
|
||||
# 创建索引
|
||||
create_index_sql = """
|
||||
CREATE INDEX IF NOT EXISTS idx_sensitive_words_word ON sensitive_words(word);
|
||||
CREATE INDEX IF NOT EXISTS idx_sensitive_words_category ON sensitive_words(category);
|
||||
"""
|
||||
cur.execute(create_index_sql)
|
||||
print("创建索引成功")
|
||||
|
||||
# 创建更新触发器
|
||||
create_function_sql = """
|
||||
CREATE OR REPLACE FUNCTION update_updated_at_column()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.updated_at = CURRENT_TIMESTAMP;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
"""
|
||||
cur.execute(create_function_sql)
|
||||
print("创建更新函数成功")
|
||||
|
||||
# 检查触发器是否存在
|
||||
check_trigger_sql = """
|
||||
SELECT COUNT(*) FROM pg_trigger WHERE tgname = 'update_sensitive_words_updated_at'
|
||||
"""
|
||||
cur.execute(check_trigger_sql)
|
||||
trigger_exists = cur.fetchone()[0] > 0
|
||||
|
||||
if not trigger_exists:
|
||||
create_trigger_sql = """
|
||||
CREATE TRIGGER update_sensitive_words_updated_at
|
||||
BEFORE UPDATE ON sensitive_words
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_updated_at_column();
|
||||
"""
|
||||
cur.execute(create_trigger_sql)
|
||||
print("创建触发器成功")
|
||||
else:
|
||||
print("触发器已存在,跳过创建")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
print("事务提交成功")
|
||||
|
||||
# 关闭游标和连接
|
||||
cur.close()
|
||||
conn.close()
|
||||
print("连接关闭成功")
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"创建敏感词表失败: {e}")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("开始创建敏感词表...")
|
||||
success = create_sensitive_words_table()
|
||||
if success:
|
||||
print("敏感词表创建成功!")
|
||||
else:
|
||||
print("敏感词表创建失败!")
|
||||
27
domainCheck/create_sensitive_words_table.sql
Normal file
27
domainCheck/create_sensitive_words_table.sql
Normal file
@@ -0,0 +1,27 @@
|
||||
-- 创建敏感词表
|
||||
CREATE TABLE IF NOT EXISTS sensitive_words (
|
||||
id SERIAL PRIMARY KEY,
|
||||
word VARCHAR(255) UNIQUE NOT NULL,
|
||||
category VARCHAR(50) DEFAULT 'default',
|
||||
priority INTEGER DEFAULT 1,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- 创建索引
|
||||
CREATE INDEX IF NOT EXISTS idx_sensitive_words_word ON sensitive_words(word);
|
||||
CREATE INDEX IF NOT EXISTS idx_sensitive_words_category ON sensitive_words(category);
|
||||
|
||||
-- 更新updated_at触发器
|
||||
CREATE OR REPLACE FUNCTION update_updated_at_column()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.updated_at = CURRENT_TIMESTAMP;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER update_sensitive_words_updated_at
|
||||
BEFORE UPDATE ON sensitive_words
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_updated_at_column();
|
||||
155
domainCheck/detect/aizhan.py
Normal file
155
domainCheck/detect/aizhan.py
Normal file
@@ -0,0 +1,155 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :aizhan.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/3/29 20:59
|
||||
@explain : 爱站网域名查询工具 - 获取域名的网站标题信息
|
||||
'''
|
||||
|
||||
import re # 正则表达式模块,用于从HTML中提取网站标题
|
||||
|
||||
import requests # HTTP请求库,用于发送网络请求
|
||||
from loguru import logger # 日志记录模块,用于记录程序运行日志
|
||||
|
||||
from typing import List, Optional # 类型提示
|
||||
|
||||
|
||||
def check_aizhan(domain: str, sensitive_words: Optional[List[str]] = None, proxies: dict = None):
|
||||
'''
|
||||
查询域名的网站标题信息是否存在敏感词
|
||||
|
||||
通过爱站网(aizhan.com)查询域名的网站标题
|
||||
|
||||
:param domain: 待查询的域名(如:www.baidu.com,不带协议头)
|
||||
:param proxies: 代理配置(如:{'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'}
|
||||
:return: str - 网站标题字符串,如果查询失败则返回空字符串
|
||||
'''
|
||||
# 初始化敏感词列表
|
||||
if sensitive_words is None:
|
||||
sensitive_words = []
|
||||
|
||||
# 构建爱站网查询URL
|
||||
# 格式:https://www.aizhan.com/cha/{域名}/
|
||||
url = f'https://www.aizhan.com/cha/{domain}/'
|
||||
|
||||
# 构建HTTP请求头(模拟浏览器访问)
|
||||
headers = {
|
||||
'Accept': '*/*', # 接受所有类型的内容
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0',
|
||||
# 用户代理(浏览器标识)
|
||||
'Referer': url, # 来源页面(防盗链)
|
||||
'host': 'www.aizhan.com', # 目标主机地址
|
||||
}
|
||||
|
||||
# 添加重试机制
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
# 发送GET请求获取页面内容
|
||||
# url: 请求URL
|
||||
# headers: HTTP请求头
|
||||
# timeout=10: 设置超时时间10秒
|
||||
# proxies: 代理配置(如需使用代理)
|
||||
response = requests.get(url, headers=headers, timeout=10, proxies=proxies)
|
||||
|
||||
# 判断请求是否成功(HTTP 200表示成功)
|
||||
if response.status_code == 200:
|
||||
# 使用正则表达式提取网站标题
|
||||
# 正则解释:匹配 id="webpage_title"> 开头,</div> 结尾,中间的内容
|
||||
# 匹配模式:id="webpage_title">标题内容</div>
|
||||
# ([^<]+) 表示匹配一个或多个非<字符,作为捕获组
|
||||
match = re.search(r'id="webpage_title">([^<]+)</div>', response.text)
|
||||
|
||||
# 如果找到匹配则返回标题文本,否则返回空字符串
|
||||
title = match.group(1) if match else ''
|
||||
if title:
|
||||
for sensitive_word in sensitive_words:
|
||||
# 使用正则表达式检查是否包含敏感词
|
||||
if re.search(sensitive_word, title, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {title}'
|
||||
return True, title
|
||||
|
||||
# 请求失败,继续重试
|
||||
logger.warning(f"爱站网检测第{retry+1}次失败: {domain}, 状态码: {response.status_code}")
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
# 达到最大重试次数,尝试不使用代理
|
||||
if proxies:
|
||||
logger.info('爱站网检测失败,尝试不使用代理')
|
||||
try:
|
||||
response = requests.get(url, headers=headers, timeout=10, proxies=None)
|
||||
if response.status_code == 200:
|
||||
match = re.search(r'id="webpage_title">([^<]+)</div>', response.text)
|
||||
title = match.group(1) if match else ''
|
||||
if title:
|
||||
for sensitive_word in sensitive_words:
|
||||
if re.search(sensitive_word, title, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {title}'
|
||||
return True, title
|
||||
return True, ''
|
||||
except Exception as e2:
|
||||
logger.error(f"不使用代理的爱站网检测也失败: {domain}, 错误: {e2}")
|
||||
return False, str(e2)
|
||||
return True, ''
|
||||
|
||||
except Exception as e: # 捕获所有异常(网络错误、超时等)
|
||||
logger.warning(f"爱站网检测第{retry+1}次失败: {domain}, 错误: {e}")
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
# 达到最大重试次数,尝试不使用代理
|
||||
if proxies:
|
||||
logger.info('爱站网检测失败,尝试不使用代理')
|
||||
try:
|
||||
response = requests.get(url, headers=headers, timeout=10, proxies=None)
|
||||
if response.status_code == 200:
|
||||
match = re.search(r'id="webpage_title">([^<]+)</div>', response.text)
|
||||
title = match.group(1) if match else ''
|
||||
if title:
|
||||
for sensitive_word in sensitive_words:
|
||||
if re.search(sensitive_word, title, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {title}'
|
||||
return True, title
|
||||
return True, ''
|
||||
except Exception as e2:
|
||||
logger.error(f"不使用代理的爱站网检测也失败: {domain}, 错误: {e2}")
|
||||
return False, str(e2)
|
||||
return False, str(e)
|
||||
|
||||
# # ========================= 【测试代码】 =========================
|
||||
# if __name__ == '__main__':
|
||||
# # 测试用例1:查询常见域名
|
||||
# logger.info("=== 测试爱站网查询 ===")
|
||||
# test_domain_1 = 'baidu.com'
|
||||
# logger.info(f"查询域名: {test_domain_1}")
|
||||
#
|
||||
# result_1 = check_aizhan(test_domain_1)
|
||||
# logger.info(f"网站标题: {result_1}")
|
||||
# logger.info("")
|
||||
#
|
||||
# # 测试用例2:查询另一个域名
|
||||
# logger.info("=== 测试查询QQ域名 ===")
|
||||
# test_domain_2 = 'qq.com'
|
||||
# logger.info(f"查询域名: {test_domain_2}")
|
||||
#
|
||||
# result_2 = check_aizhan(test_domain_2)
|
||||
# logger.info(f"网站标题: {result_2}")
|
||||
# logger.info("")
|
||||
#
|
||||
#
|
||||
#
|
||||
# # 测试用例4:查询可能不存在的域名
|
||||
# logger.info("=== 测试不存在域名 ===")
|
||||
# test_domain_4 = 'nonexistentdomain123456.com'
|
||||
# logger.info(f"查询域名: {test_domain_4}")
|
||||
#
|
||||
# result_4 = check_aizhan(test_domain_4)
|
||||
# logger.info(f"网站标题: {result_4 if result_4 else '(无结果)'}")
|
||||
263
domainCheck/detect/baidu.py
Normal file
263
domainCheck/detect/baidu.py
Normal file
@@ -0,0 +1,263 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :baidu.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/3/29 21:54
|
||||
@explain : 百度域名检测工具 - 包含百度搜索结果查询和百度安全API检测功能
|
||||
'''
|
||||
|
||||
import random # 随机数生成模块,用于随机选择浏览器类型
|
||||
import re # 正则表达式模块,用于从HTML中提取搜索结果
|
||||
import time
|
||||
|
||||
from loguru import logger # 日志记录模块,用于记录程序运行日志
|
||||
from curl_cffi import requests # curl_cffi库,支持浏览器模拟的HTTP请求
|
||||
from typing import List, Optional, Union, Any # 类型提示
|
||||
|
||||
def check_site(domain: str, sensitive_words: Optional[List[str]] = None, proxies: dict = None) -> Union[
|
||||
None, tuple[bool, str], list[Any]]:
|
||||
'''
|
||||
通过百度搜索查询域名的相关信息
|
||||
|
||||
使用curl_cffi模拟浏览器访问百度搜索,获取site:domain的搜索结果标题
|
||||
|
||||
:param domain: 待查询的域名(如:www.baidu.com)
|
||||
:param proxies: 代理配置(如:{'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'})
|
||||
:return: list - 搜索结果标题列表,如果查询失败则返回空列表
|
||||
'''
|
||||
# 初始化敏感词列表
|
||||
if sensitive_words is None:
|
||||
sensitive_words = []
|
||||
|
||||
# 添加重试机制
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try: # try-except块,用于捕获网络请求中的异常
|
||||
# 构建搜索查询参数
|
||||
# wd: 搜索关键词,格式为site:域名
|
||||
params = {
|
||||
'wd': f'site:{domain}' # 搜索site:domain,限制搜索结果为指定域名
|
||||
}
|
||||
|
||||
cookies = {
|
||||
'BIDUPSID': '053DBE4D820C6EFB729DC7B13B1F82B2',
|
||||
'PSTM': '1775657119',
|
||||
'H_PS_PSSID': '63148_67862_67986_68002_68142_68148_68152_68141_68165_68189_68226_68267_68296_68336_68369_68453_68438_68464_68541_68546_68558_68520_68589_68621_68615_68606_68601_68682_68671_68735_68544_68733_68766_68807_68901_68918_68836_68921_68955_68976_68997_69007_69010_69018_69024_69014',
|
||||
'BAIDUID': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1',
|
||||
'BDORZ': 'B490B5EBF6F3CD402E515D22BCDA1598',
|
||||
'BAIDUID_BFESS': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1',
|
||||
'BA_HECTOR': 'ah0l24210la4050l0g0k0g2g0h20071ktcocj27',
|
||||
'ZFY': 'Qxyr75Xm7o8zUxYzuFnYoW7cnS:AnVp:BpnrNVkr3usno:C',
|
||||
'delPer': '0',
|
||||
'BAIDUID_REF': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1',
|
||||
'H_WISE_SIDS': '110085_661771_667681_673676_683089_682564_685373_660925_687556_686285_690306_690303_690478_690576_690334_691545_685595_690883_692754_692777_692118_692910_693221_693364_693403_693656_693941_694124_693392_694171_694236_693996_694316_694324_693886_694409_694379_694696_694780_694783_692006_694884_694918_694996_694991_695032_695118_694840_695140_695193_695110_695262_688610_694933_695278_694865_695214_695374_690651_694577_692378_695452_695457_695480_695603_695642_695631_695668_695824_695843_695866_695888_695892_695897_695975_694177_695900_695939_696145_696150_696153_696069_696078_696066_694359_696288_696308_694985_696297_695715_696128_696313_696457_696431_696473_696492_696590_693385_696610_696682_696679_696662_696651_696655_696658_696673_696643_696110_696729_696733_696772_8000116_8000133_8000138_8000159_8000163_8000167_8000176_8000186_8000190_8000204',
|
||||
'H_WISE_SIDS_BFESS': '110085_661771_667681_673676_683089_682564_685373_660925_687556_686285_690306_690303_690478_690576_690334_691545_685595_690883_692754_692777_692118_692910_693221_693364_693403_693656_693941_694124_693392_694171_694236_693996_694316_694324_693886_694409_694379_694696_694780_694783_692006_694884_694918_694996_694991_695032_695118_694840_695140_695193_695110_695262_688610_694933_695278_694865_695214_695374_690651_694577_692378_695452_695457_695480_695603_695642_695631_695668_695824_695843_695866_695888_695892_695897_695975_694177_695900_695939_696145_696150_696153_696069_696078_696066_694359_696288_696308_694985_696297_695715_696128_696313_696457_696431_696473_696492_696590_693385_696610_696682_696679_696662_696651_696655_696658_696673_696643_696110_696729_696733_696772_8000116_8000133_8000138_8000159_8000163_8000167_8000176_8000186_8000190_8000204',
|
||||
'MSA_PBT': '147',
|
||||
'MSA_ZOOM': '1000',
|
||||
'wpr': '0',
|
||||
'COOKIE_SESSION': '0_0_0_0_0_0_0_0_0_0_0_0_0_1775657396%7C1%230_0_0_0_0_0_0_0_1775657396%7C1',
|
||||
'MSA_PHY_WH': '1440_3440',
|
||||
'POLYFILL': '0',
|
||||
'MSA_WH': '1254_940',
|
||||
'kleck': '7ce2abac229d2e8ba435a8bcc6256f57f53e9d08954443bf',
|
||||
'PSCBD': '16%3A1%3A3',
|
||||
'SE_LAUNCH': '5%3A1775657396_16%3A29594323%3A3',
|
||||
'BDSVRTM': '3',
|
||||
'PSINO': '6',
|
||||
'__bsi': '17976785662214065461_00_7_R_R_6_0303_c02f_Y',
|
||||
}
|
||||
|
||||
headers = {
|
||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
|
||||
'Accept-Language': 'zh-CN,zh;q=0.9',
|
||||
'Connection': 'keep-alive',
|
||||
'Sec-Fetch-Dest': 'document',
|
||||
'Sec-Fetch-Mode': 'navigate',
|
||||
'Sec-Fetch-Site': 'none',
|
||||
'Sec-Fetch-User': '?1',
|
||||
'Upgrade-Insecure-Requests': '1',
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0',
|
||||
'sec-ch-ua': '"Chromium";v="146", "Not-A.Brand";v="24", "Microsoft Edge";v="146"',
|
||||
'sec-ch-ua-mobile': '?0',
|
||||
'sec-ch-ua-platform': '"Windows"',
|
||||
}
|
||||
|
||||
# 百度搜索URL
|
||||
url = 'https://m.baidu.com/s'
|
||||
|
||||
# 发送GET请求获取搜索结果
|
||||
# url: 请求URL
|
||||
# params: 查询参数
|
||||
# headers: HTTP请求头
|
||||
# timeout=15: 超时时间15秒
|
||||
# proxies: 代理配置
|
||||
# impersonate: 模拟的浏览器类型
|
||||
# .content: 获取响应内容(字节流)
|
||||
# .decode('utf-8'): 解码为UTF-8字符串
|
||||
response_text = requests.get(url, params=params, headers=headers, cookies=cookies, timeout=15,
|
||||
proxies=proxies).content.decode('utf-8')
|
||||
# 使用正则表达式提取搜索结果标题
|
||||
# re.findall()返回所有匹配的结果列表
|
||||
# 正则解释:匹配 <!--s-text--> 和 <!--/s-text--> 之间的内容
|
||||
match = re.findall(r'<!--s-text-->([^<]+)<!--/s-text-->', response_text,re.S)
|
||||
|
||||
# 返回匹配结果列表,如果没有匹配则返回空列表
|
||||
search_results= match if match else []
|
||||
|
||||
# 如果没有搜索结果,直接返回通过
|
||||
if not search_results:
|
||||
return True, ''
|
||||
|
||||
# 检查每个搜索结果是否包含敏感词
|
||||
for result in search_results:
|
||||
for sensitive_word in sensitive_words:
|
||||
# 使用正则表达式检查是否包含敏感词
|
||||
if re.search(sensitive_word, result, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {result}'
|
||||
|
||||
# 所有搜索结果都不包含敏感词,返回通过
|
||||
return True, ''
|
||||
except Exception as e: # 捕获所有异常
|
||||
# logger.warning(f"百度site检测第{retry+1}次失败: {domain}, 错误: {e}")
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
proxies=None
|
||||
else:
|
||||
# 达到最大重试次数,返回检测失败
|
||||
logger.error(f"百度site检测失败: {domain}, 已达到最大重试次数")
|
||||
return False, str(e) # 返回错误信息
|
||||
|
||||
|
||||
def baidu(domain: str, proxies: dict = None) -> str:
|
||||
'''
|
||||
通过百度安全API检测域名安全状态
|
||||
|
||||
调用百度移动安全API检测域名是否为风险网站
|
||||
|
||||
:param domain: 待检测的域名(如:www.baidu.com)
|
||||
:param proxies: 代理配置(如:{'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'}
|
||||
:return: str - 安全状态描述("风险网站提示"、"高危网站提示"或"正常"),如果检测失败则返回错误信息
|
||||
'''
|
||||
# 构建HTTP请求头
|
||||
headers = {
|
||||
'Cache-Control': 'no-cache', # 不使用缓存
|
||||
'Content-Type': 'application/json; charset=utf-8', # 内容类型为JSON
|
||||
'Host': 'mobsec-sec.baidu.com', # 目标主机
|
||||
'User-Agent': 'okhttp/3.12.12', # 模拟OKHttp客户端
|
||||
}
|
||||
|
||||
# 构建请求参数
|
||||
params = {
|
||||
'auth_ver': '2', # 认证版本
|
||||
'appkey': '4665fd2c6b0922a551e8ae74', # API应用密钥
|
||||
'nonce': '1751309255340', # 随机数(注意:实际使用时应该动态生成)
|
||||
'lc': '77qHTv4VtmRiXYtd', # 本地配置参数
|
||||
'pkg': 'com.baidu.searchbox', # 应用包名
|
||||
'vc': '-1', # 版本码
|
||||
'cuid': 'CF0E6FCCD824D146605C16AC1C8BAC95%7CVFWAO56TC', # 设备ID(注意:实际使用时应该动态生成)
|
||||
'tk': '', # 令牌(空)
|
||||
'type': '3', # 检测类型
|
||||
'vn': '2.6.0', # 版本号
|
||||
's': 'fdfef2ce96c1c7193f9b80425020f63e', # 签名(注意:实际使用时应该动态生成)
|
||||
}
|
||||
|
||||
# 构建JSON请求数据
|
||||
json_data = {
|
||||
'url': f'https://{domain}/', # 待检测的URL(添加https协议头)
|
||||
}
|
||||
|
||||
# 提取URL作为后续获取结果的键
|
||||
key = json_data['url']
|
||||
|
||||
# 添加重试机制
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try: # try-except块,用于捕获API调用中的异常
|
||||
# 发送POST请求到百度安全API
|
||||
# url: API接口地址
|
||||
# params: 查询参数
|
||||
# headers: HTTP请求头
|
||||
# json: JSON请求数据
|
||||
# proxies: 代理配置
|
||||
# .json(): 解析JSON响应
|
||||
response = requests.post('https://mobsec-sec.baidu.com/3.1/scanurl', params=params, headers=headers,
|
||||
json=json_data, proxies=proxies, timeout=15).json()
|
||||
|
||||
# # 记录API响应(调试用)
|
||||
# logger.info(response)
|
||||
|
||||
# 安全状态码映射表
|
||||
msgs = {
|
||||
2193: '风险网站提示', # 风险网站
|
||||
2243: '高危网站提示', # 高危网站
|
||||
'_': '正常' # 默认正常
|
||||
}
|
||||
|
||||
# 从响应中获取安全等级并映射为状态描述
|
||||
# response['response']['datas'][key].get('grand_level', 'level'): 获取安全等级,默认值为'level'
|
||||
# msgs.get(): 根据安全等级获取对应的状态描述,默认值为'正常'
|
||||
return msgs.get(response['response']['datas'][key].get('grand_level', 'level'), '正常')
|
||||
|
||||
except Exception as e: # 捕获所有异常
|
||||
logger.warning(f"百度网页安全中心检测第{retry+1}次失败: {domain}, 错误: {e}")
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
# 达到最大重试次数,尝试不使用代理
|
||||
if proxies:
|
||||
logger.info('百度网页安全中心检测失败,尝试不使用代理')
|
||||
try:
|
||||
response = requests.post('https://mobsec-sec.baidu.com/3.1/scanurl', params=params, headers=headers,
|
||||
json=json_data, proxies=None, timeout=15).json()
|
||||
msgs = {
|
||||
2193: '风险网站提示', # 风险网站
|
||||
2243: '高危网站提示', # 高危网站
|
||||
'_': '正常' # 默认正常
|
||||
}
|
||||
return msgs.get(response['response']['datas'][key].get('grand_level', 'level'), '正常')
|
||||
except Exception as e2:
|
||||
logger.error(f"不使用代理的百度网页安全中心检测也失败: {domain}, 错误: {e2}")
|
||||
return str(e2) # 返回错误信息
|
||||
return str(e) # 返回错误信息
|
||||
|
||||
|
||||
# # ========================= 【测试代码】 =========================
|
||||
# if __name__ == '__main__': # 程序入口,当直接运行此文件时执行以下代码
|
||||
# # proxies = get_proxies()
|
||||
# proxies = None
|
||||
# # 测试用例1:查询域名搜索结果
|
||||
# logger.info("=== 测试百度搜索结果查询 ===")
|
||||
# test_domain_1 = '920zl.com'
|
||||
# logger.info(f"查询域名: {test_domain_1}")
|
||||
#
|
||||
# result_1 = check_site(test_domain_1, proxies=proxies)
|
||||
# logger.info(result_1)
|
||||
|
||||
#
|
||||
# # 测试用例3:检测域名安全状态
|
||||
# logger.info("=== 测试域名安全检测 ===")
|
||||
# test_domain_3 = 'baidu.com'
|
||||
# logger.info(f"检测域名: {test_domain_3}")
|
||||
#
|
||||
# result_3 = baidu(test_domain_3, proxies=proxies)
|
||||
# logger.info(f"安全状态: {result_3}")
|
||||
# logger.debug(proxies)
|
||||
|
||||
# domains = ['Lqyingye.com', 'tjjinLikeji.com', 'guokangLxs.com', 'zhuoyijz.com', 'jiandanrongyi.com', 'fromhhc.com',
|
||||
# 'djxow.com', '725game.com', 'gcLpchd.cn', '6rdb.com.cn', 'sunmantech.com', 'hroxa.com', 'guLhu65.cn',
|
||||
# 'xxjdch.com', '7e7fm1.cn', 'kaoxueb.com', 'jabas.cn', 'shzhongyi.com', 'figuangze.com', 'eztw9k.cn',
|
||||
# 'foxok.com', 'oLLbmr.com', 'vwpwwkp.cn', 'cbmrs.com', 'zaozhuang56.com', 'hengxiangracing.com',
|
||||
# 'jdyfbpq.cn', 'czca.cn', 'shibingxiongdi.com', 'ubxbm.cn', 'ydbaoshi.com', '2046hd.cn', 'mffmcp.com',
|
||||
# 'hgLdhzz.com', 'okswmi.cn', 'caidaozg.com', 'dkdywx.cn', 'qkjps.com', 'Lwxgk.cn', 'drtechnoLogy.cn',
|
||||
# 'akcie.cn', '588uu.com', 'kuaiyijian.net', 'c5b.cn', 'faka866.cn', 'wiraf03.cn', 'bobjibar.com',
|
||||
# '166233.cn', 'adLfjcjq.cn', 'whrsom.com']
|
||||
# for domain in domains:
|
||||
# result_1 = check_site(domain)
|
||||
# logger.info(f"搜索结果标题数: {len(result_1)}")
|
||||
# for i, title in enumerate(result_1, 1):
|
||||
# logger.info(f" 结果{i}: {title}")
|
||||
# time.sleep(1)
|
||||
191
domainCheck/detect/c360.py
Normal file
191
domainCheck/detect/c360.py
Normal file
@@ -0,0 +1,191 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :c360.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/2 15:57
|
||||
@explain : 360搜索敏感词检测工具 - 检测域名是否包含敏感词
|
||||
'''
|
||||
|
||||
import re # 正则表达式模块
|
||||
import requests # HTTP请求库
|
||||
from loguru import logger # 日志记录
|
||||
from typing import List, Tuple, Optional, Dict # 类型提示
|
||||
|
||||
|
||||
# 常量定义
|
||||
SO_SEARCH_URL = 'https://www.so.com/s' # 360搜索URL
|
||||
SO_REFERER_TEMPLATE = 'https://www.so.com/s?ie=utf-8&q=site%3A{domain}' # Referer模板
|
||||
SEARCH_PATTERN = r'target="_blank">([^<]+)</a></h3>' # 搜索结果匹配模式
|
||||
BLOCKED_CODE = 3 # 拦截状态码
|
||||
|
||||
|
||||
def check_domain(
|
||||
domain: str,
|
||||
sensitive_words: Optional[List[str]] = None,
|
||||
proxies: Optional[Dict] = None
|
||||
) -> Tuple[bool, str]:
|
||||
"""
|
||||
检查域名是否包含敏感词
|
||||
|
||||
通过360搜索获取域名的搜索结果,然后检查搜索结果中是否包含敏感词
|
||||
|
||||
Args:
|
||||
domain: 要检测的域名
|
||||
sensitive_words: 敏感词列表,如果为None则初始化为空列表
|
||||
proxies: 代理配置
|
||||
|
||||
Returns:
|
||||
Tuple[bool, str]: (是否通过, 错误信息)
|
||||
- True, '': 通过检测,不包含敏感词
|
||||
- False, '敏感词:xxx': 包含敏感词,返回具体的敏感词
|
||||
"""
|
||||
# 初始化敏感词列表
|
||||
if sensitive_words is None:
|
||||
sensitive_words = []
|
||||
|
||||
# 构建请求头
|
||||
headers = {
|
||||
'Accept': '*/*', # 接受所有类型
|
||||
'Accept-Language': 'zh-cn', # 中文语言
|
||||
'Referer': SO_REFERER_TEMPLATE.format(domain=domain), # 来源页面
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36', # 用户代理
|
||||
'Host': 'www.so.com', # 主机名
|
||||
}
|
||||
|
||||
# 构建查询参数
|
||||
params = {
|
||||
'ie': 'utf-8', # 编码格式
|
||||
'q': f'site:{domain}', # 搜索查询
|
||||
}
|
||||
|
||||
# 添加重试机制
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
# 发送GET请求获取搜索结果
|
||||
response = requests.get(
|
||||
SO_SEARCH_URL,
|
||||
params=params,
|
||||
headers=headers,
|
||||
proxies=proxies,
|
||||
timeout=15 # 增加超时时间到15秒
|
||||
)
|
||||
|
||||
# 解析响应内容
|
||||
response_html = response.content.decode('utf-8', errors='ignore')
|
||||
|
||||
# 使用正则表达式提取搜索结果
|
||||
search_results = re.findall(SEARCH_PATTERN, response_html)
|
||||
|
||||
# 记录搜索结果数量
|
||||
logger.info(f'360搜索结果数量: {len(search_results)}')
|
||||
|
||||
# 如果没有搜索结果,直接返回通过
|
||||
if not search_results:
|
||||
return True, ''
|
||||
|
||||
# 检查每个搜索结果是否包含敏感词
|
||||
for result in search_results:
|
||||
for sensitive_word in sensitive_words:
|
||||
# 使用正则表达式检查是否包含敏感词
|
||||
if re.search(sensitive_word, result, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {result}'
|
||||
|
||||
# 所有搜索结果都不包含敏感词,返回通过
|
||||
return True, ''
|
||||
|
||||
except requests.exceptions.Timeout:
|
||||
logger.warning(f'360搜索请求第{retry+1}次超时')
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
# 达到最大重试次数,尝试不使用代理
|
||||
if proxies:
|
||||
logger.info('360搜索请求超时,尝试不使用代理')
|
||||
try:
|
||||
response = requests.get(
|
||||
SO_SEARCH_URL,
|
||||
params=params,
|
||||
headers=headers,
|
||||
proxies=None,
|
||||
timeout=15
|
||||
)
|
||||
response_html = response.content.decode('utf-8', errors='ignore')
|
||||
search_results = re.findall(SEARCH_PATTERN, response_html)
|
||||
logger.info(f'不使用代理的360搜索结果数量: {len(search_results)}')
|
||||
if not search_results:
|
||||
return True, ''
|
||||
for result in search_results:
|
||||
for sensitive_word in sensitive_words:
|
||||
if re.search(sensitive_word, result, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {result}'
|
||||
return True, ''
|
||||
except Exception as e:
|
||||
logger.error(f'不使用代理的360搜索请求也失败: {str(e)}')
|
||||
return False, f'360搜索请求超时'
|
||||
return False, '360搜索请求超时'
|
||||
except requests.exceptions.RequestException as e:
|
||||
error_str = str(e)
|
||||
logger.warning(f'360搜索请求第{retry+1}次失败: {error_str}')
|
||||
# 检查是否是"Too many open files"错误
|
||||
if 'Too many open files' in error_str:
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
# 达到最大重试次数,返回failure标记,不拉黑域名
|
||||
logger.error(f'360搜索请求失败,已达到最大重试次数: {error_str}')
|
||||
return True, 'failure'
|
||||
else:
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
# 达到最大重试次数,尝试不使用代理
|
||||
if proxies:
|
||||
logger.info('360搜索请求失败,尝试不使用代理')
|
||||
try:
|
||||
response = requests.get(
|
||||
SO_SEARCH_URL,
|
||||
params=params,
|
||||
headers=headers,
|
||||
proxies=None,
|
||||
timeout=15
|
||||
)
|
||||
response_html = response.content.decode('utf-8', errors='ignore')
|
||||
search_results = re.findall(SEARCH_PATTERN, response_html)
|
||||
logger.info(f'不使用代理的360搜索结果数量: {len(search_results)}')
|
||||
if not search_results:
|
||||
return True, ''
|
||||
for result in search_results:
|
||||
for sensitive_word in sensitive_words:
|
||||
if re.search(sensitive_word, result, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {result}'
|
||||
return True, ''
|
||||
except Exception as e2:
|
||||
logger.error(f'不使用代理的360搜索请求也失败: {str(e2)}')
|
||||
return False, f'360搜索请求失败: {str(e)}'
|
||||
return False, f'360搜索请求失败: {str(e)}'
|
||||
except Exception as e:
|
||||
logger.warning(f'360搜索第{retry+1}次未知错误: {str(e)}')
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
else:
|
||||
return False, f'未知错误: {str(e)}'
|
||||
|
||||
|
||||
# if __name__ == '__main__':
|
||||
# # 测试用例
|
||||
# test_domain = 'niuniushouka.com'
|
||||
# test_sensitive_words = ['赌博', '色情', '暴力','收卡']
|
||||
#
|
||||
# logger.info(f'开始检测域名: {test_domain}')
|
||||
# result = check_domain(test_domain, test_sensitive_words)
|
||||
# logger.info(f'检测结果: {result}')
|
||||
1257
domainCheck/detect/chinaz.js
Normal file
1257
domainCheck/detect/chinaz.js
Normal file
File diff suppressed because one or more lines are too long
242
domainCheck/detect/chinaz.py
Normal file
242
domainCheck/detect/chinaz.py
Normal file
@@ -0,0 +1,242 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :chinaz.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/3/27 23:48
|
||||
@explain : 站长工具SEO查询工具 - 获取域名网站标题、SEO排名等信息
|
||||
'''
|
||||
|
||||
import json # JSON处理模块,用于解析API响应数据
|
||||
import os # 操作系统接口模块,用于获取脚本所在目录路径
|
||||
import re # 正则表达式模块,用于提取网页中的标题和加密密钥
|
||||
import time # 时间处理模块,用于生成时间戳
|
||||
from typing import List, Optional # 类型提示
|
||||
|
||||
import quickjs # JavaScript执行引擎,用于执行JS加密代码生成签名
|
||||
import requests # HTTP请求库,用于发送网络请求
|
||||
from loguru import logger # 日志记录模块,用于记录程序运行日志
|
||||
|
||||
|
||||
def check_title(domain: str,sensitive_words: Optional[List[str]] = None, proxies: dict = None):
|
||||
'''
|
||||
检测域名标题并获取SEO数据
|
||||
|
||||
:param domain: 待检测域名(如:www.baidu.com,不带协议头)
|
||||
:param proxies: 代理配置(如:{'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'}
|
||||
:return: Tuple[bool, str, Tuple[bool, Any]] - (是否成功, 消息, (SEO查询是否成功, SEO数据或错误信息))
|
||||
'''
|
||||
# 初始化敏感词列表
|
||||
if sensitive_words is None:
|
||||
sensitive_words = []
|
||||
|
||||
# 添加重试机制
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
# 构建SEO查询URL(站长工具网站)
|
||||
url = f'https://seo.chinaz.com/{domain}'
|
||||
|
||||
# 发送GET请求获取页面内容,设置超时10秒
|
||||
response = requests.get(url, timeout=10, proxies=proxies)
|
||||
|
||||
# 判断请求是否成功(HTTP 200表示成功)
|
||||
if response.status_code == 200:
|
||||
# 使用正则表达式提取网站标题(从id="site_title"的div标签中提取文本内容)
|
||||
# 正则解释:匹配 id="site_title"> 开头,</div> 结尾,中间的内容
|
||||
match = re.search(r'id="site_title">([^<]+)</div>', response.text)
|
||||
|
||||
# 使用正则表达式提取加密密钥enkey(用于后续API请求的身份验证)
|
||||
# 正则解释:匹配 var enkey = '...' 中的单引号内容
|
||||
pattern = r"var enkey = '([^']+)''"
|
||||
match_enkey = re.search(pattern, response.text)
|
||||
|
||||
# 如果上面的正则匹配失败,尝试匹配正确的单引号版本
|
||||
if not match_enkey:
|
||||
pattern = r"var enkey = '([^']+)" # 匹配 var enkey = '...' 格式
|
||||
match_enkey = re.search(pattern, response.text)
|
||||
|
||||
# 如果找到enkey则提取并去除首尾空格,否则为空字符串
|
||||
enkey = match_enkey.group(1).strip() if match_enkey else ''
|
||||
|
||||
# 提取网站标题,如果找到则返回,否则返回空字符串
|
||||
title = match.group(1).strip() if match else ''
|
||||
|
||||
if title:
|
||||
for sensitive_word in sensitive_words:
|
||||
# 使用正则表达式检查是否包含敏感词
|
||||
if re.search(sensitive_word, title, re.IGNORECASE):
|
||||
logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}')
|
||||
return False, f'检测到敏感词: {sensitive_word} 在结果: {title}', ()
|
||||
|
||||
# 调用get_site_data获取详细SEO数据
|
||||
success, result = get_site_data(domain, enkey, proxies=proxies)
|
||||
|
||||
# 检查网站分类
|
||||
if success and result:
|
||||
# 定义需要拉黑的分类
|
||||
blacklist_categories = ['视频电影', '体育运动', '常用查询', '游戏网站', '游戏', '视频', '体育']
|
||||
|
||||
# 检查分类是否在黑名单中
|
||||
if isinstance(result, dict):
|
||||
# 检查可能的分类字段
|
||||
category_fields = ['Category', 'category', '分类', '网站分类']
|
||||
for field in category_fields:
|
||||
if field in result:
|
||||
category = result[field]
|
||||
if isinstance(category, str):
|
||||
for blacklist_category in blacklist_categories:
|
||||
if blacklist_category in category:
|
||||
logger.warning(f'检测到黑名单分类: {category} 在域名: {domain}')
|
||||
return False, f'网站分类: {category} 属于黑名单分类', ()
|
||||
|
||||
# 检查Result是否为列表
|
||||
elif isinstance(result, list):
|
||||
for item in result:
|
||||
if isinstance(item, dict):
|
||||
category_fields = ['Category', 'category', '分类', '网站分类']
|
||||
for field in category_fields:
|
||||
if field in item:
|
||||
category = item[field]
|
||||
if isinstance(category, str):
|
||||
for blacklist_category in blacklist_categories:
|
||||
if blacklist_category in category:
|
||||
logger.warning(f'检测到黑名单分类: {category} 在域名: {domain}')
|
||||
return False, f'网站分类: {category} 属于黑名单分类', ()
|
||||
|
||||
return True, 'success', (success, result)
|
||||
|
||||
# 请求失败返回空标题和空元组
|
||||
return True, '', ()
|
||||
|
||||
except Exception as e: # 捕获所有异常
|
||||
error_str = str(e)
|
||||
# 检查是否是需要重试的错误类型
|
||||
if any(error_type in error_str for error_type in ['Too many open files', 'EOF occurred in violation of protocol', 'Max retries exceeded', 'Read timed out']):
|
||||
logger.warning(f"站长工具检测第{retry+1}次失败: {domain}, 错误: {e}")
|
||||
if retry < max_retries - 1:
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
continue
|
||||
else:
|
||||
# 达到最大重试次数,返回失败但不拉黑
|
||||
logger.error(f"站长工具检测失败,已达到最大重试次数: {domain}, 错误: {e}")
|
||||
return True, 'failure', (False, str(e))
|
||||
else:
|
||||
# 其他错误直接返回
|
||||
logger.error(f"站长工具检测失败: {domain}, 错误: {e}")
|
||||
return False, str(e), ()
|
||||
|
||||
|
||||
# ========================= 【最终请求】 =========================
|
||||
def get_site_data(domain, enkey: str, proxies: dict = None):
|
||||
'''
|
||||
获取站点的详细SEO数据
|
||||
|
||||
通过站长工具API获取域名的SEO排名、权重等信息
|
||||
|
||||
:param domain: 待检测域名(如:www.baidu.com)
|
||||
:param enkey: 加密密钥(从check_title函数获取)
|
||||
:param proxies: 代理配置(如:{'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'})
|
||||
:return: Tuple[bool, Any] - (是否成功, 结果数据或错误信息)
|
||||
- (True, Result数据): 查询成功
|
||||
- (False, 错误信息): 查询失败
|
||||
'''
|
||||
try:
|
||||
# 获取当前脚本所在目录(用于构建JS文件的完整路径)
|
||||
current_dir = os.path.dirname(__file__)
|
||||
|
||||
# 构建JS加密代码文件的完整路径
|
||||
js_path = os.path.join(current_dir, "chinaz.js")
|
||||
|
||||
# 读取JS加密代码文件(包含生成签名的函数)
|
||||
with open(js_path, "r", encoding="utf-8") as f:
|
||||
js_code = f.read()
|
||||
|
||||
# 创建JavaScript执行上下文(QuickJS引擎)
|
||||
ctx = quickjs.Context()
|
||||
|
||||
# 【关键步骤】加载JS代码到上下文,使其可以被调用
|
||||
# 这一步必须先执行,否则后续的函数调用会失败
|
||||
ctx.eval(js_code)
|
||||
|
||||
# 调用JS函数生成host_key(主机密钥,是生成签名的基础)
|
||||
# 格式:generateHostKey("domain")
|
||||
host_key = ctx.eval(f'generateHostKey("{domain}")')
|
||||
|
||||
# 获取当前时间戳(毫秒),用于防止缓存和重放攻击
|
||||
ts = str(int(time.time() * 1000))
|
||||
|
||||
# 构建请求参数字典(包含API所需的全部参数)
|
||||
params = {
|
||||
# jQuery回调函数名(格式:jQuery + 随机数字 + 时间戳)
|
||||
"callback": f"jQuery11130943805094955342_{int(time.time() * 1000) - 1000}",
|
||||
"action": "GetCategory", # 动作类型:获取分类/排名数据
|
||||
"host": domain, # 主机域名
|
||||
"secretkey": enkey, # 加密密钥(从页面获取)
|
||||
|
||||
# 随机数字(调用JS函数生成,与host_key相关)
|
||||
"rd": ctx.eval(f'getRandomNum("{host_key}")'),
|
||||
"ts": ts, # 时间戳(毫秒)
|
||||
|
||||
# MD5令牌(调用JS函数生成,用于请求签名验证)
|
||||
"token": ctx.eval(f'generateMD5Token("{host_key}","{ts}")'),
|
||||
|
||||
"_": int(time.time() * 1000) # 额外时间戳参数(用于缓存破坏)
|
||||
}
|
||||
|
||||
# 构建HTTP请求头(模拟浏览器访问)
|
||||
headers = {
|
||||
'Accept': '*/*', # 接受所有类型的内容
|
||||
'Accept-Language': 'zh-cn', # 接受的语言:简体中文
|
||||
'Referer': 'https://othertool.chinaz.com/GetTopRanked.ashx', # 来源页面(防盗链)
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.87 Safari/537.36', # 用户代理(浏览器标识)
|
||||
'Host': 'othertool.chinaz.com', # 目标主机地址
|
||||
}
|
||||
|
||||
# API请求URL(站长工具SEO数据查询接口)
|
||||
url = 'https://othertool.chinaz.com/GetTopRanked.ashx'
|
||||
|
||||
# 发送GET请求获取SEO数据
|
||||
resp = requests.get(
|
||||
url, # 请求URL
|
||||
params=params, # 查询参数(自动拼接为URL参数)
|
||||
headers=headers, # 请求头
|
||||
timeout=10, # 超时时间10秒
|
||||
proxies=proxies # 代理配置
|
||||
)
|
||||
|
||||
# 判断请求是否成功
|
||||
if resp.status_code == 200:
|
||||
# 使用正则表达式提取JSON数据(API返回格式:callback(json_data))
|
||||
# 正则解释:匹配括号内的JSON内容
|
||||
match = re.search(r'\((.*?)\)', resp.text)
|
||||
|
||||
# 提取JSON字符串
|
||||
json_str = match.group(1)
|
||||
|
||||
# 解析JSON数据为Python字典
|
||||
data = json.loads(json_str)
|
||||
|
||||
# 返回状态码和结果(StateCode为1表示成功)
|
||||
return data['StateCode'] == 1, data['Result']
|
||||
|
||||
except Exception as e: # 捕获所有异常
|
||||
logger.error(e) # 记录错误日志
|
||||
return False, str(e) # 返回失败状态和错误信息
|
||||
|
||||
|
||||
# # ========================= 【测试代码】 =========================
|
||||
# if __name__ == '__main__':
|
||||
#
|
||||
#
|
||||
# # 测试用例3:使用代理查询
|
||||
# logger.info("=== 测试使用代理 ===")
|
||||
# test_domain_3 = 'tt.com'
|
||||
#
|
||||
# logger.info(f"查询域名: {test_domain_3}")
|
||||
#
|
||||
#
|
||||
# title_3, seo_data_3 = check_title(test_domain_3)
|
||||
# logger.info(f"域名标题: {title_3}")
|
||||
# logger.info(f"SEO数据: {seo_data_3}")
|
||||
253
domainCheck/detect/geetest2.py
Normal file
253
domainCheck/detect/geetest2.py
Normal file
@@ -0,0 +1,253 @@
|
||||
# -*- coding:utf-8 -*-
|
||||
import requests, re, json, base64
|
||||
import io, os, random
|
||||
import time, cv2, json
|
||||
from PIL import Image
|
||||
from functools import partial
|
||||
|
||||
from io import BytesIO
|
||||
|
||||
from urllib3.util import proxy
|
||||
|
||||
from detect.module.use_ua import randomUA
|
||||
from detect.module.gap import quekou
|
||||
import subprocess
|
||||
|
||||
subprocess.Popen = partial(subprocess.Popen, encoding='utf-8')
|
||||
import execjs
|
||||
|
||||
|
||||
class slide():
|
||||
def __init__(self):
|
||||
self.headers = randomUA()
|
||||
self.t = round(time.time() * 1000)
|
||||
|
||||
def __ease_out_expo(self, sep):
|
||||
if sep == 1:
|
||||
return 1
|
||||
else:
|
||||
return 1 - pow(2, -10 * sep)
|
||||
|
||||
def get_slide_track(self, vdict):
|
||||
distance = self.huak(vdict)
|
||||
slide_track = [
|
||||
[random.randint(-50, -10), random.randint(-50, -10), 0],
|
||||
[0, 0, 0],
|
||||
]
|
||||
count = 30 + int(distance / 2)
|
||||
t = random.randint(50, 100)
|
||||
_x = 0
|
||||
_y = 0
|
||||
for i in range(count):
|
||||
x = round(self.__ease_out_expo(i / count) * distance)
|
||||
t += random.randint(10, 20)
|
||||
if x == _x:
|
||||
continue
|
||||
slide_track.append([x, _y, t])
|
||||
_x = x
|
||||
slide_track.append(slide_track[-1])
|
||||
return slide_track
|
||||
|
||||
def tp_huanyuan(self, content):
|
||||
_img = Image.open(BytesIO(content))
|
||||
_Ge = [{"x": -157, "y": -58}, {"x": -145, "y": -58}, {"x": -265, "y": -58}, {"x": -277, "y": -58},
|
||||
{"x": -181, "y": -58}, {"x": -169, "y": -58}, {"x": -241, "y": -58}, {"x": -253, "y": -58},
|
||||
{"x": -109, "y": -58}, {"x": -97, "y": -58}, {"x": -289, "y": -58}, {"x": -301, "y": -58},
|
||||
{"x": -85, "y": -58}, {"x": -73, "y": -58}, {"x": -25, "y": -58}, {"x": -37, "y": -58},
|
||||
{"x": -13, "y": -58}, {"x": -1, "y": -58}, {"x": -121, "y": -58}, {"x": -133, "y": -58},
|
||||
{"x": -61, "y": -58}, {"x": -49, "y": -58}, {"x": -217, "y": -58}, {"x": -229, "y": -58},
|
||||
{"x": -205, "y": -58}, {"x": -193, "y": -58}, {"x": -145, "y": 0}, {"x": -157, "y": 0},
|
||||
{"x": -277, "y": 0}, {"x": -265, "y": 0}, {"x": -169, "y": 0}, {"x": -181, "y": 0},
|
||||
{"x": -253, "y": 0}, {"x": -241, "y": 0}, {"x": -97, "y": 0}, {"x": -109, "y": 0},
|
||||
{"x": -301, "y": 0}, {"x": -289, "y": 0}, {"x": -73, "y": 0}, {"x": -85, "y": 0},
|
||||
{"x": -37, "y": 0}, {"x": -25, "y": 0}, {"x": -1, "y": 0}, {"x": -13, "y": 0},
|
||||
{"x": -133, "y": 0}, {"x": -121, "y": 0}, {"x": -49, "y": 0}, {"x": -61, "y": 0},
|
||||
{"x": -229, "y": 0}, {"x": -217, "y": 0}, {"x": -193, "y": 0}, {"x": -205, "y": 0}]
|
||||
w_sep, h_sep = 10, 58
|
||||
new_img = Image.new('RGB', (260, 116))
|
||||
|
||||
for idx in range(len(_Ge)):
|
||||
x = abs(_Ge[idx]['x'])
|
||||
y = 58 if _Ge[idx]['y'] == -58 else 0
|
||||
img_cut = _img.crop((x, y, x + w_sep, y + h_sep))
|
||||
new_x = idx % 26 * 10
|
||||
new_y = 0 if idx < 26 else 58
|
||||
new_img.paste(img_cut, (new_x, new_y))
|
||||
|
||||
img_byte = BytesIO()
|
||||
new_img.save(img_byte, 'png')
|
||||
return img_byte.getvalue()
|
||||
|
||||
def huak(self, vdict):
|
||||
count = 1
|
||||
bgbase64 = ""
|
||||
tpbase64 = ""
|
||||
for idv, p_url in vdict.items():
|
||||
p_url = 'http://static.geetest.com/' + p_url
|
||||
# print(p_url)
|
||||
vcode = requests.get(p_url, headers=self.headers)
|
||||
text = vcode.content
|
||||
if idv == 'bg':
|
||||
text = self.tp_huanyuan(text)
|
||||
bgbase64 = base64.encodebytes(text).decode()
|
||||
else:
|
||||
tpbase64 = base64.encodebytes(text).decode()
|
||||
|
||||
count += 1
|
||||
if bgbase64 and tpbase64:
|
||||
dis = quekou().get_distance(bgbase64, tpbase64)
|
||||
else:
|
||||
dis = 0
|
||||
return dis
|
||||
|
||||
|
||||
class Geetest2():
|
||||
def __init__(self):
|
||||
self.header = {
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36",
|
||||
}
|
||||
# 获取当前文件的目录
|
||||
current_dir = os.path.dirname(__file__)
|
||||
|
||||
with open(os.path.join(current_dir, 'module', 'crack_geetest2x.js'), "r", encoding="gb2312",
|
||||
errors='ignore') as f:
|
||||
js_encrypt = f.read()
|
||||
self.js = execjs.compile(js_encrypt)
|
||||
|
||||
self.t = round(time.time() * 1000)
|
||||
self.gct_vdi = {}
|
||||
self.gct_volue = ['', ''] # 初始值
|
||||
self.token_vdi = {}
|
||||
self.timeout = (7, 9.05)
|
||||
self.geetest_type = 'slide'
|
||||
self.geetest_path = "/static/js/geetest.6.0.9.js"
|
||||
|
||||
def new_requests(self):
|
||||
self.session = requests.session()
|
||||
self.session.headers = self.header
|
||||
|
||||
def get_vsion(self, http, gt, proxies, pparmas):
|
||||
url = "https://api.geetest.com/gettype.php"
|
||||
params = {
|
||||
"gt": gt,
|
||||
"callback": f"geetest_{self.t}"
|
||||
}
|
||||
try:
|
||||
respe = http.get(url=url, params=params, proxies=proxies).text
|
||||
except Exception as e:
|
||||
if proxy:
|
||||
if pparmas:
|
||||
vdict = {'result': '错误', 'message': f'代理IP超时!,请重新更换代理IP,{e}', 'proxy': proxies,
|
||||
'pparmas': pparmas}
|
||||
else:
|
||||
vdict = {'result': '错误', 'message': f'代理IP超时!,请重新更换代理IP,{e}', 'proxy': proxies}
|
||||
else:
|
||||
vdict = {'result': '错误', 'message': f'请求错误,{e}'}
|
||||
return vdict
|
||||
|
||||
try:
|
||||
data = json.loads(respe[22:-1])["data"]
|
||||
except:
|
||||
vdict = {'result': '错误', 'type': 'auto', 'message': respe}
|
||||
return vdict
|
||||
self.geetest_type = data["type"]
|
||||
self.geetest_path = data['path']
|
||||
self.token_vdi[gt] = 1
|
||||
|
||||
def get_tp(self, gt, challenge, type_, proxies=None, pparmas=None):
|
||||
self.new_requests()
|
||||
self.t = round(time.time() * 1000)
|
||||
if pparmas:
|
||||
h_list = re.findall('([\w-]+):(.+)', pparmas)
|
||||
for idx in h_list:
|
||||
print('新增协议头' + str(idx))
|
||||
self.session.headers[idx[0]] = idx[1].replace('\r', '').replace('\n', '').replace('\r\n', '')
|
||||
print(self.session.headers)
|
||||
|
||||
if not self.token_vdi.get('gt', ''):
|
||||
self.get_vsion(self.session, gt, proxies, pparmas)
|
||||
url = f"https://api.geetest.com/get.php"
|
||||
params = {
|
||||
"gt": gt,
|
||||
"challenge": challenge,
|
||||
"product": "popup",
|
||||
"offline": "false",
|
||||
"protocol": "https://",
|
||||
"type": self.geetest_type,
|
||||
"path": self.geetest_path,
|
||||
"callback": f"geetest_{self.t}"
|
||||
}
|
||||
response = self.session.get(url, params=params, timeout=self.timeout).text
|
||||
geetest_type = self.geetest_type
|
||||
if geetest_type == 'slide':
|
||||
data = json.loads(response[22:-1])
|
||||
else:
|
||||
try:
|
||||
data = json.loads(response[22:-1])["data"]
|
||||
except:
|
||||
vdict = {'result': '错误', 'type': type_, 'message': response}
|
||||
return vdict
|
||||
##print(data)
|
||||
nc, ns = data['c'], data['s']
|
||||
|
||||
if geetest_type == 'slide':
|
||||
data = json.loads(response[22:-1])
|
||||
else:
|
||||
try:
|
||||
data = json.loads(response[22:-1])["data"]
|
||||
except:
|
||||
vdict = {'result': '错误', 'type': type_, 'message': json.loads(response)}
|
||||
return vdict
|
||||
##print(data)
|
||||
nc, ns = data['c'], data['s']
|
||||
gct_url = 'http://static.geetest.com' + data['gct_path']
|
||||
##print('gct地址:', gct_url)
|
||||
if geetest_type == 'click':
|
||||
return {'result': '请求类型不是滑块', 'type': type_, 'gt': gt, 'challenge': challenge,
|
||||
'当前类型': '点选'}
|
||||
if geetest_type == 'slide': # slide
|
||||
|
||||
challenge = data['challenge'] # 这里challenge改变了
|
||||
bg = data['bg'] # 背景图片
|
||||
slice = data['slice'] # 缺口图片
|
||||
vdict = {
|
||||
"bg": bg,
|
||||
"slice": slice,
|
||||
}
|
||||
slide_track = slide().get_slide_track(vdict)
|
||||
imgload = 37
|
||||
w = self.js.call('get_slide_w', nc, ns, gt, challenge, slide_track, imgload, self.gct_volue)
|
||||
params = (
|
||||
('gt', gt),
|
||||
('challenge', challenge),
|
||||
('w', w),
|
||||
('callback', f"geetest_{self.t}"),
|
||||
)
|
||||
response = self.session.get('https://api.geetest.com/ajax.php', params=params, proxies=proxies,
|
||||
timeout=self.timeout).text
|
||||
data = json.loads(response[22:-1])
|
||||
validate = data.get('validate')
|
||||
vdict = {'result': data.get('message'), 'type': geetest_type, 'gt': gt, 'challenge': challenge,
|
||||
'validate': validate}
|
||||
|
||||
return vdict
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
a = Geetest2()
|
||||
headers = {
|
||||
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
|
||||
}
|
||||
url = "https://seo.juziseo.com/class/gtcode/msg/StartMsgCaptchaServlet.php"
|
||||
url = "https://gsxt.hlj.gov.cn/registerValidate.jspx"
|
||||
|
||||
for i in range(1):
|
||||
index = requests.get(url, headers=headers)
|
||||
# print(index.text)
|
||||
html = index.json()
|
||||
challenge = html["challenge"]
|
||||
gt = html["gt"]
|
||||
print('gt', gt, 'challenge', challenge)
|
||||
r = a.get_tp(gt, challenge, 'auto', '')
|
||||
print(r)
|
||||
557
domainCheck/detect/jucha.py
Normal file
557
domainCheck/detect/jucha.py
Normal file
@@ -0,0 +1,557 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :jucha.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/2 22:49
|
||||
@explain : 聚查网API封装类 - 优化版,减少冗余代码,添加详细注释
|
||||
'''
|
||||
|
||||
# 导入标准库
|
||||
import base64 # Base64编码解码
|
||||
import hashlib # 哈希加密
|
||||
import json # JSON数据处理
|
||||
import os # 操作系统接口
|
||||
import pickle # 序列化反序列化
|
||||
import random # 随机数生成
|
||||
import subprocess # 子进程管理
|
||||
import time # 时间处理
|
||||
from functools import partial
|
||||
|
||||
# 导入第三方库
|
||||
import requests # HTTP请求库
|
||||
from loguru import logger # 日志记录
|
||||
from requests.cookies import RequestsCookieJar # Cookie管理
|
||||
|
||||
|
||||
|
||||
# # 移除自定义子进程类的替换,避免影响其他模块
|
||||
# class MySubprocessPopen(subprocess.Popen): # 自定义子进程类
|
||||
# def __init__(self, *args, **kwargs): # 初始化方法
|
||||
# kwargs['encoding'] = "UTF-8" # 设置默认编码为UTF-8
|
||||
# super().__init__(*args, **kwargs) # 调用父类初始化方法
|
||||
#
|
||||
#
|
||||
# subprocess.Popen = MySubprocessPopen # 替换subprocess.Popen为自定义类
|
||||
os.environ["EXECJS_RUNTIME"] = "Node" # 设置JavaScript运行时环境为Node.js
|
||||
startupinfo = subprocess.STARTUPINFO()
|
||||
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
|
||||
subprocess.Popen = partial(subprocess.Popen, encoding='utf-8', startupinfo=startupinfo)
|
||||
|
||||
|
||||
def resolve_node_executable() -> str:
|
||||
current_dir = os.path.dirname(__file__)
|
||||
project_dir = os.path.dirname(current_dir)
|
||||
bundled_node = os.path.join(project_dir, "tools", "node-v20.19.4-win-x64", "node.exe")
|
||||
if os.path.exists(bundled_node):
|
||||
return bundled_node
|
||||
return "node"
|
||||
|
||||
def calculate_seed(t: int) -> str: # 计算验证码种子值
|
||||
now = int(time.time()) # 获取当前时间戳
|
||||
c = now - t # 计算时间差
|
||||
seed = base64.b64encode(str(c).encode()).decode() # Base64编码
|
||||
return seed # 返回种子字符串
|
||||
|
||||
|
||||
def random_fingerprint() -> str: # 生成随机设备指纹
|
||||
data = str(random.random()) + str(time.time()) # 组合随机数和时间戳
|
||||
return hashlib.sha256(data.encode()).hexdigest() # 返回SHA256哈希值
|
||||
|
||||
|
||||
class JC(object): # 聚查网API封装类
|
||||
token: str # 验证码token
|
||||
session_id: str # 会话ID
|
||||
fingerprint: str # 设备指纹
|
||||
captchaId: str # 验证码ID
|
||||
encryptionPublicKey: str # 加密公钥
|
||||
cookie: RequestsCookieJar = {} # 聚查网Cookie
|
||||
juming_cookie: RequestsCookieJar = {} # 聚名网Cookie
|
||||
|
||||
# 通用请求头配置
|
||||
headers = {
|
||||
'accept': 'application/json, text/javascript, */*; q=0.01', # 接受的内容类型
|
||||
'accept-language': 'zh-CN,zh;q=0.9', # 接受的语言
|
||||
'content-type': 'application/x-www-form-urlencoded', # 内容类型
|
||||
'origin': 'https://www.jucha.com', # 请求源
|
||||
'priority': 'u=1, i', # 请求优先级
|
||||
'referer': 'https://www.jucha.com/login', # 来源页面
|
||||
'sec-ch-ua': '"Chromium";v="146", "Not-A.Brand";v="24", "Google Chrome";v="146"', # 浏览器标识
|
||||
'sec-ch-ua-mobile': '?0', # 是否移动端
|
||||
'sec-ch-ua-platform': '"Windows"', # 操作系统平台
|
||||
'sec-fetch-dest': 'empty', # 请求目标
|
||||
'sec-fetch-mode': 'cors', # 请求模式
|
||||
'sec-fetch-site': 'same-origin', # 请求站点
|
||||
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36',
|
||||
# 用户代理
|
||||
'x-requested-with': 'XMLHttpRequest', # AJAX请求标识
|
||||
}
|
||||
|
||||
def __init__(self, proxies: dict = None): # 初始化JC类
|
||||
self.session = requests.Session() # 创建会话对象
|
||||
self.session.timeout = 10 # 设置超时时间
|
||||
self.session.proxies = proxies # 设置代理
|
||||
self.base_url = "https://www.jucha.com" # 设置基础URL
|
||||
self.cookie = requests.cookies.RequestsCookieJar() # 初始化CookieJar
|
||||
self.juming_cookie = requests.cookies.RequestsCookieJar() # 初始化聚名网CookieJar
|
||||
|
||||
def _get_headers(self, referer: str = None) -> dict: # 获取请求头(可自定义referer)
|
||||
headers = self.headers.copy() # 复制默认请求头
|
||||
if referer: # 如果指定了referer
|
||||
headers['referer'] = referer # 更新referer
|
||||
return headers # 返回请求头
|
||||
|
||||
def _get_route_headers(self, route: str) -> dict: # 根据路由获取对应的请求头
|
||||
referer_map = { # 路由到referer的映射
|
||||
'whois': f'{self.base_url}/whois/', # WHOIS查询的referer
|
||||
'beian': f'{self.base_url}/baian/', # 备案查询的referer
|
||||
'safe': f'{self.base_url}/safe/', # 安全检测的referer
|
||||
}
|
||||
return self._get_headers(referer_map.get(route)) # 返回对应的请求头
|
||||
|
||||
def _handle_captcha(self, retry_times: int = 5) -> tuple: # 处理滑块验证码
|
||||
for _ in range(retry_times): # 循环重试
|
||||
init_result = self.captcha_init() # 初始化验证码
|
||||
if init_result[0]: # 初始化成功
|
||||
verify_result = self.captcha_verify() # 验证验证码
|
||||
if verify_result[0]: # 验证成功
|
||||
return True, '验证码验证成功' # 返回成功
|
||||
return False, f'滑块验证码失败,{retry_times}次内未成功' # 返回失败
|
||||
|
||||
def _check_request(self, url: str, data: dict, headers: dict, cookies=None) -> dict: # 检查请求并处理验证码
|
||||
res = self.session.post(url=url, data=data, headers=headers, cookies=cookies) # 发送POST请求
|
||||
try:
|
||||
response = res.json() # 尝试解析JSON
|
||||
except Exception as e: # JSON解析失败
|
||||
logger.error(f"JSON解析失败: {str(e)}") # 记录错误
|
||||
logger.error(f"原始响应内容(前500字符): {res.text[:500]}") # 记录原始响应
|
||||
return {'code': -1, 'msg': f'JSON解析失败: {str(e)}'} # 返回错误
|
||||
return response # 返回响应
|
||||
|
||||
def _check_and_handle_captcha(self, response: dict, domain: str, route: str, xm_codes,
|
||||
data: dict = None) -> tuple: # 检查并处理验证码
|
||||
# 检查response是否为字典
|
||||
if not isinstance(response, dict):
|
||||
return False, None, response # 返回失败
|
||||
|
||||
if response.get('code') == 1001: # 需要验证码
|
||||
captcha_result = self._handle_captcha() # 处理验证码
|
||||
if captcha_result[0]: # 验证码验证成功
|
||||
new_data = data.copy() if data else {} # 复制数据
|
||||
new_data.update({ # 更新验证码参数
|
||||
'_csrf': '', # CSRF令牌
|
||||
'ymlb': domain, # 域名
|
||||
'xm_codes[]': xm_codes, # 检测代码
|
||||
'type': '2' if route in ['beian', 'safe'] else '1', # 类型:备案和安全检测为2,WHOIS为1
|
||||
'route': route, # 路由
|
||||
'captcha_verify_param': self.token, # 验证码token
|
||||
'sessionId': self.session_id, # 会话ID
|
||||
})
|
||||
return True, new_data, None # 返回需要重试
|
||||
return False, None, captcha_result # 返回失败
|
||||
return False, None, response # 不需要验证码
|
||||
|
||||
def captcha_init(self): # 初始化滑块验证码
|
||||
self.fingerprint = random_fingerprint() # 生成随机指纹
|
||||
data = { # 构建请求数据
|
||||
"request_id": self.fingerprint, # 请求ID
|
||||
"scene": "default", # 场景
|
||||
"seed": calculate_seed(286) # 种子值
|
||||
}
|
||||
url = f"{self.base_url}/captcha/init" # 初始化URL
|
||||
response = self.session.post(url, headers=self.headers, data=data, cookies=self.cookie).json() # 发送请求
|
||||
if response['code'] == 1: # 初始化成功
|
||||
self.captchaId = response['data']['captchaId'] # 保存验证码ID
|
||||
self.encryptionPublicKey = response['data']['encryptionPublicKey'] # 保存加密公钥
|
||||
return response['code'] == 1, response['msg'] # 返回结果
|
||||
|
||||
def captcha_verify(self): # 验证滑块验证码
|
||||
data = { # 构建验证数据
|
||||
"offset": 290, # 滑块偏移量
|
||||
"duration": 611, # 滑动持续时间
|
||||
"trail": [ # 滑动轨迹
|
||||
{"x": 0, "y": 0, "time": 0}, {"x": 0, "y": 0, "time": 17}, {"x": 1, "y": 0, "time": 89},
|
||||
{"x": 5, "y": 0, "time": 106}, {"x": 14, "y": 0, "time": 123}, {"x": 28, "y": 0, "time": 139},
|
||||
{"x": 47, "y": 0, "time": 156}, {"x": 74, "y": 0, "time": 173}, {"x": 102, "y": 0, "time": 189},
|
||||
{"x": 131, "y": 0, "time": 206}, {"x": 159, "y": 0, "time": 223}, {"x": 184, "y": 0, "time": 239},
|
||||
{"x": 205, "y": 0, "time": 256}, {"x": 227, "y": 1, "time": 273}, {"x": 248, "y": 0, "time": 289},
|
||||
{"x": 263, "y": 0, "time": 306}, {"x": 276, "y": 0, "time": 323}, {"x": 286, "y": 0, "time": 339},
|
||||
{"x": 290, "y": 0, "time": 356}, {"x": 290, "y": 0, "time": 373}, {"x": 290, "y": 0, "time": 389},
|
||||
{"x": 290, "y": 0, "time": 406}, {"x": 290, "y": 0, "time": 423},
|
||||
{"x": 290, "y": 0, "time": 478}
|
||||
],
|
||||
"fingerprint": self.fingerprint, # 设备指纹
|
||||
"captchaId": self.captchaId, # 验证码ID
|
||||
"serverPublicKey": self.encryptionPublicKey # 服务器公钥
|
||||
}
|
||||
data_str = json.dumps(data, separators=(",", ":")) # 转换为JSON字符串
|
||||
current_dir = os.path.dirname(__file__)
|
||||
|
||||
jsFile_path = current_dir + "/sdk_leg_env.js" # JavaScript脚本路径
|
||||
|
||||
if not os.path.exists(jsFile_path): # 检查脚本文件是否存在
|
||||
logger.error(f"Node.js脚本文件不存在: {jsFile_path}") # 记录错误
|
||||
return False, f"缺少签名脚本: {jsFile_path}" # 返回失败
|
||||
|
||||
try: # 尝试执行Node.js脚本
|
||||
with subprocess.Popen( # 创建子进程
|
||||
[resolve_node_executable(), jsFile_path], # 执行node命令
|
||||
stdin=subprocess.PIPE, # 标准输入管道
|
||||
stdout=subprocess.PIPE, # 标准输出管道
|
||||
stderr=subprocess.PIPE, # 标准错误管道
|
||||
text=True, # 文本模式
|
||||
encoding="utf-8", # UTF-8编码
|
||||
errors="ignore" # 忽略编码错误
|
||||
) as proc: # 进程上下文
|
||||
stdout, stderr = proc.communicate(data_str, timeout=10) # 传入参数并获取输出
|
||||
verify_data = stdout.strip() # 去除空白字符
|
||||
if stderr: # 如果有错误输出
|
||||
logger.error(f'genData_stderr->[{stderr}]') # 记录错误
|
||||
|
||||
if proc.returncode != 0: # 检查退出码
|
||||
return False, f"Node.js脚本执行失败,退出码: {proc.returncode}, 错误信息: {stderr}" # 返回失败
|
||||
|
||||
url = f"{self.base_url}/captcha/verify" # 验证URL
|
||||
verify_headers = self.headers.copy() # 复制请求头
|
||||
verify_headers['Content-Type'] = 'application/json' # 设置内容类型(仅用于验证请求)
|
||||
verify_response = self.session.post(url, headers=verify_headers, data=verify_data,
|
||||
cookies=self.cookie) # 发送验证请求
|
||||
if verify_response.cookies: # 如果响应中有新的Cookie
|
||||
self.cookie.update(verify_response.cookies) # 合并Cookie(而不是替换)
|
||||
response = verify_response.json() # 解析JSON响应
|
||||
if response['code'] == 1: # 验证成功
|
||||
self.token = response['data']['token'] # 保存token
|
||||
self.session_id = response['data']['session_id'] # 保存会话ID
|
||||
return response['code'] == 1, response['msg'] # 返回结果
|
||||
except subprocess.TimeoutExpired: # 超时异常
|
||||
logger.error("Node.js脚本执行超时(超过10秒)") # 记录错误
|
||||
return False, "Node.js脚本执行超时" # 返回失败
|
||||
except Exception as e: # 其他异常
|
||||
logger.error(f"执行Node.js脚本异常: {str(e)}") # 记录错误
|
||||
return False, f"执行Node.js脚本异常: {str(e)}" # 返回失败
|
||||
|
||||
if not verify_data: # 检查输出是否为空
|
||||
return False, "Node.js脚本未输出任何内容" # 返回失败
|
||||
|
||||
def save_cookies(self, filepath="jucha_cookies.pkl"): # 保存Cookie到文件
|
||||
with open(filepath, "wb") as f: # 以二进制写入模式打开文件
|
||||
pickle.dump(self.cookie, f) # 序列化保存Cookie
|
||||
logger.info(f"已保存Cookie到文件: {filepath}")
|
||||
# 保存到Redis
|
||||
try:
|
||||
import redis
|
||||
from app.config import config
|
||||
redis_client = redis.Redis(
|
||||
host=config.REDIS_HOST,
|
||||
port=config.REDIS_PORT,
|
||||
password=config.REDIS_PASSWORD,
|
||||
db=config.REDIS_DB,
|
||||
decode_responses=True
|
||||
)
|
||||
# 将cookie转换为字典
|
||||
cookie_dict = {}
|
||||
# 检查self.cookie的类型
|
||||
if isinstance(self.cookie, dict):
|
||||
# 如果是字典,直接使用
|
||||
cookie_dict = self.cookie
|
||||
logger.info(f"Cookie是字典类型,直接使用")
|
||||
elif hasattr(self.cookie, '__iter__'):
|
||||
# 如果是可迭代对象,遍历处理
|
||||
for cookie in self.cookie:
|
||||
# 检查cookie对象是否有name和value属性
|
||||
if hasattr(cookie, 'name') and hasattr(cookie, 'value'):
|
||||
cookie_dict[cookie.name] = cookie.value
|
||||
logger.info(f"Cookie是可迭代对象,处理后得到: {cookie_dict}")
|
||||
else:
|
||||
logger.warning(f"Cookie类型不支持: {type(self.cookie)}")
|
||||
|
||||
redis_client.set('domain_tool:jucha_cookies', str(cookie_dict))
|
||||
except Exception as e:
|
||||
logger.error(f"保存Cookie到Redis异常: {str(e)}")
|
||||
pass
|
||||
|
||||
def load_cookies(self, filepath="jucha_cookies.pkl"): # 从文件加载Cookie
|
||||
try: # 尝试加载
|
||||
with open(filepath, "rb") as f: # 以二进制读取模式打开文件
|
||||
loaded_cookie = pickle.load(f) # 反序列化加载Cookie
|
||||
# 检查加载的cookie类型
|
||||
if isinstance(loaded_cookie, dict):
|
||||
# 如果是字典,转换为RequestsCookieJar
|
||||
cookie_jar = requests.cookies.RequestsCookieJar()
|
||||
for name, value in loaded_cookie.items():
|
||||
cookie_jar.set(name, value)
|
||||
self.cookie = cookie_jar
|
||||
else:
|
||||
self.cookie = loaded_cookie
|
||||
except Exception as e: # 加载失败
|
||||
logger.error(f"加载Cookie失败: {e}")
|
||||
self.cookie = requests.cookies.RequestsCookieJar() # 创建空的CookieJar
|
||||
|
||||
def load_juming_cookies(self, filepath="juming_cookies.pkl"): # 从文件加载聚名网Cookie
|
||||
try: # 尝试加载
|
||||
with open(filepath, "rb") as f: # 以二进制读取模式打开文件
|
||||
self.juming_cookie = pickle.load(f) # 反序列化加载Cookie
|
||||
except: # 加载失败
|
||||
self.juming_cookie = requests.cookies.RequestsCookieJar() # 创建空的CookieJar
|
||||
|
||||
def auth_login(self): # 使用聚名网账号登录聚查网
|
||||
params = { # 请求参数
|
||||
'platform': 'juming', # 平台标识
|
||||
}
|
||||
res = self.session.get( # 发送GET请求
|
||||
url=f'{self.base_url}/home/login/get_auth_url', # 获取授权URL
|
||||
params=params, # 请求参数
|
||||
headers=self.headers, # 请求头
|
||||
cookies=self.juming_cookie # 聚名网Cookie
|
||||
)
|
||||
if res.cookies: # 如果响应中有Cookie
|
||||
self.cookie.update(res.cookies) # 更新Cookie
|
||||
response = res.json() # 解析JSON响应
|
||||
if response['code'] == 1: # 获取授权URL成功
|
||||
url = f'%s&tiao=1' % response['data']['auth_url'] # 构建跳转URL
|
||||
# 合并聚名网Cookie和聚查网Cookie
|
||||
combined_cookies = requests.cookies.RequestsCookieJar()
|
||||
combined_cookies.update(self.juming_cookie)
|
||||
combined_cookies.update(self.cookie)
|
||||
response = self.session.get( # 发送GET请求
|
||||
url=url, # 跳转URL
|
||||
headers=self.headers, # 请求头
|
||||
cookies=combined_cookies, # 合并后的Cookie
|
||||
allow_redirects=False # 不自动重定向
|
||||
)
|
||||
if response.cookies: # 如果响应中有Cookie
|
||||
# 安全更新Cookie,避免冲突
|
||||
for name, value in response.cookies.items():
|
||||
self.cookie[name] = value # 直接赋值,覆盖同名Cookie
|
||||
if response.status_code == 302: # 重定向状态码
|
||||
url = response.headers['location'] # 获取重定向URL
|
||||
response = self.session.get( # 发送GET请求
|
||||
url=url, # 重定向URL
|
||||
headers=self.headers, # 请求头
|
||||
allow_redirects=False # 不自动重定向
|
||||
)
|
||||
if response.cookies: # 如果响应中有Cookie
|
||||
# 安全更新Cookie,避免冲突
|
||||
for name, value in response.cookies.items():
|
||||
self.cookie[name] = value # 直接赋值,覆盖同名Cookie
|
||||
if response.status_code == 302: # 再次重定向
|
||||
return True, '登录成功' # 返回成功
|
||||
return False, '聚名登录过期' # 返回失败
|
||||
return False, response['msg'] # 返回失败
|
||||
|
||||
def _build_check_data(self, domain: str, route: str, xm_codes, data: dict = None) -> dict: # 构建检测请求数据
|
||||
if data is not None: # 如果data不为空
|
||||
return data # 直接返回data
|
||||
return { # 构建新的请求数据
|
||||
'_csrf': '', # CSRF令牌
|
||||
'ymlb': domain, # 域名
|
||||
'xm_codes[]': xm_codes, # 检测代码
|
||||
'type': '2' if route in ['beian', 'safe'] else '1', # 类型:备案和安全检测为2,WHOIS为1
|
||||
'route': route, # 路由
|
||||
}
|
||||
|
||||
def _make_check_request(self, domain: str, route: str, xm_codes, data: dict = None) -> tuple: # 发起检测请求
|
||||
headers = self._get_route_headers(route) # 获取对应的请求头
|
||||
data = self._build_check_data(domain, route, xm_codes, data) # 构建请求数据
|
||||
url = f'{self.base_url}/home_item/check' # 检测URL
|
||||
response = self._check_request(url, data, headers, self.cookie) # 发送请求
|
||||
return response, data # 返回响应和数据
|
||||
|
||||
def _handle_check_response(self, response: dict, domain: str, route: str, xm_codes, data: dict,
|
||||
callback) -> tuple: # 处理检测响应
|
||||
need_retry, retry_data, final_response = self._check_and_handle_captcha( # 检查并处理验证码
|
||||
response, domain, route, xm_codes, data
|
||||
)
|
||||
if need_retry: # 需要重试
|
||||
return callback(domain=domain, data=retry_data) # 递归调用回调
|
||||
|
||||
# 检查final_response是否为字典
|
||||
if not isinstance(final_response, dict):
|
||||
return False, f'响应格式错误: {final_response}', '' # 返回失败
|
||||
|
||||
if final_response['code'] == 1: # 检测成功
|
||||
return self._get_search_result(final_response, domain, route) # 获取查询结果(传入domain)
|
||||
return False, final_response.get('msg', '未知错误'), '' # 返回失败
|
||||
|
||||
def _get_search_result(self, response: dict, domain: str, route: str) -> tuple: # 获取查询结果
|
||||
data = { # 构建查询数据
|
||||
'_csrf': '', # CSRF令牌
|
||||
'domain': domain, # 域名(注意:使用domain而不是ymlb)
|
||||
'rwid': response['data']['rwid'], # 任务ID
|
||||
'type': '2' if route in ['beian', 'safe'] else '1', # 类型
|
||||
'route': 'baian' if route == 'beian' else route, # 路由
|
||||
}
|
||||
headers = self._get_route_headers(route) # 获取对应的请求头
|
||||
url = f'{self.base_url}/home/item/search' if route != 'whois' else f'{self.base_url}/home/item/search_one' # 查询URL
|
||||
search_response = self._check_request(url, data, headers, self.cookie) # 发送查询请求
|
||||
|
||||
# 检查search_response是否为字典
|
||||
if not isinstance(search_response, dict):
|
||||
return False, f'响应格式错误: {search_response}', '' # 返回失败
|
||||
|
||||
if search_response.get('code', -1) != 1: # 查询失败
|
||||
return False, search_response.get('msg', '查询失败'), '' # 返回失败
|
||||
|
||||
if route == 'whois': # WHOIS查询
|
||||
# 安全获取WHOIS状态
|
||||
whois_zt = ''
|
||||
try:
|
||||
data1 = search_response.get('data', {})
|
||||
if isinstance(data1, dict):
|
||||
data2 = data1.get('data', {})
|
||||
if isinstance(data2, dict):
|
||||
whois = data2.get('whois', {})
|
||||
if isinstance(whois, dict):
|
||||
data3 = whois.get('data', {})
|
||||
if isinstance(data3, dict):
|
||||
data4 = data3.get('data', {})
|
||||
if isinstance(data4, dict):
|
||||
whois_zt = data4.get('zt', '')
|
||||
except Exception as e:
|
||||
logger.error(f"获取WHOIS状态失败: {e}")
|
||||
|
||||
return ( # 返回WHOIS结果
|
||||
True, # 成功
|
||||
search_response['msg'], # 消息
|
||||
whois_zt # WHOIS状态
|
||||
)
|
||||
elif route == 'beian': # 备案查询
|
||||
# 安全获取备案数据
|
||||
beian_data = {}
|
||||
beian_msg = ''
|
||||
try:
|
||||
data1 = search_response.get('data', {})
|
||||
if isinstance(data1, dict):
|
||||
data2 = data1.get('data', {})
|
||||
if isinstance(data2, dict):
|
||||
beian = data2.get('beian', {})
|
||||
if isinstance(beian, dict):
|
||||
data3 = beian.get('data', {})
|
||||
if isinstance(data3, dict):
|
||||
beian_data = data3.get('data', {})
|
||||
beian_msg = data3.get('msg', '')
|
||||
except Exception as e:
|
||||
logger.error(f"获取备案数据失败: {e}")
|
||||
|
||||
is_dict = isinstance(beian_data, dict) # 是否为字典
|
||||
return ( # 返回备案结果
|
||||
True, # 成功
|
||||
search_response['msg'], # 消息
|
||||
( # 备案信息元组
|
||||
beian_data.get('sj', '') if is_dict else '', # 备案时间
|
||||
beian_data.get('lx', '') if is_dict else '', # 备案类型
|
||||
beian_data.get('sy', '') if is_dict else '', # 备案首页地址
|
||||
beian_msg, # 备案状态
|
||||
)
|
||||
)
|
||||
elif route == 'safe': # 安全检测
|
||||
# 安全获取安全检测数据
|
||||
safe_data = {}
|
||||
try:
|
||||
data1 = search_response.get('data', {})
|
||||
if isinstance(data1, dict):
|
||||
safe_data = data1.get('data', {})
|
||||
except Exception as e:
|
||||
logger.error(f"获取安全检测数据失败: {e}")
|
||||
|
||||
check_items = [ # 检测项配置列表
|
||||
('qqjc', 'QQ检测'),
|
||||
('weixin', '微信检测'),
|
||||
('qiang', '被墙检测'),
|
||||
('dyjc', '抖音检测'),
|
||||
('bdjc', '百度检测'),
|
||||
('llqjcgg', '谷歌检测'),
|
||||
('llqjchh', '火狐检测'),
|
||||
]
|
||||
|
||||
results = [] # 结果列表
|
||||
for item_key, _ in check_items: # 遍历检测项
|
||||
try:
|
||||
code = 1
|
||||
msg = ''
|
||||
item_data = safe_data.get(item_key, {})
|
||||
if isinstance(item_data, dict):
|
||||
data1 = item_data.get('data', {})
|
||||
if isinstance(data1, dict):
|
||||
code = int(data1.get('data', 1)) # 获取检测码
|
||||
msg = data1.get('msg', '') # 获取消息
|
||||
if code == 3: # 如果检测码为3
|
||||
msg = '拦截' # 设置为拦截
|
||||
results.append((code, msg)) # 添加到结果列表
|
||||
except Exception as e:
|
||||
logger.error(f"获取安全检测项 {item_key} 失败: {e}")
|
||||
results.append((1, '查询失败')) # 添加失败结果
|
||||
|
||||
return ( # 返回安全检测结果
|
||||
True, # 成功
|
||||
search_response['msg'], # 消息
|
||||
tuple(results) # 安全检测结果元组
|
||||
)
|
||||
return False, '未知路由', '' # 返回失败
|
||||
|
||||
def check_whois_domain(self, domain: str, data=None): # 查询域名WHOIS信息
|
||||
try: # 异常处理
|
||||
response, check_data = self._make_check_request(domain, 'whois', 'whois', data) # 发起检测请求
|
||||
return self._handle_check_response(response, domain, 'whois', 'whois', check_data,
|
||||
self.check_whois_domain) # 处理响应
|
||||
except Exception as e: # 异常处理
|
||||
logger.error(f"check_domain异常: {e}") # 记录错误
|
||||
return False, str(e), '' # 返回失败
|
||||
|
||||
def beian_check_domain(self, domain: str, data=None): # 查询域名备案信息
|
||||
try: # 异常处理
|
||||
response, check_data = self._make_check_request(domain, 'beian', 'beian', data) # 发起检测请求
|
||||
return self._handle_check_response(response, domain, 'beian', 'beian', check_data,
|
||||
self.beian_check_domain) # 处理响应
|
||||
except Exception as e: # 异常处理
|
||||
logger.error(f"beian_check_domain异常: {e}") # 记录错误
|
||||
return False, str(e), [] # 返回失败
|
||||
|
||||
def safe_check_domain(self, domain: str, data=None): # 查询域名安全信息
|
||||
try: # 异常处理
|
||||
xm_codes = [ # 检测代码列表
|
||||
'qqjc', # QQ检测
|
||||
'weixin', # 微信检测
|
||||
'dyjc', # 抖音检测
|
||||
'qiang', # 被墙检测
|
||||
'bdjc', # 百度检测
|
||||
'llqjcgg', # 谷歌检测
|
||||
'llqjchh', # 火狐检测
|
||||
]
|
||||
response, check_data = self._make_check_request(domain, 'safe', xm_codes, data) # 发起检测请求
|
||||
return self._handle_check_response(response, domain, 'safe', xm_codes, check_data,
|
||||
self.safe_check_domain) # 处理响应
|
||||
except Exception as e: # 异常处理
|
||||
logger.error(f"safe_check_domain异常: {e}") # 记录错误
|
||||
return False, str(e), [] # 返回失败
|
||||
|
||||
|
||||
# # 测试代码
|
||||
# if __name__ == '__main__':
|
||||
# j = JC() # 创建JC实例
|
||||
# j.load_cookies()
|
||||
# j.load_juming_cookies() # 加载聚名网Cookie
|
||||
# logger.info("开始登录...")
|
||||
# login_result = j.auth_login() # 使用聚名网账号登录聚查网
|
||||
# logger.info(f"登录结果: {login_result}")
|
||||
# if not login_result[0]:
|
||||
# logger.error("登录失败,无法继续测试")
|
||||
# exit(1)
|
||||
# # 测试WHOIS查询
|
||||
# logger.info("测试WHOIS查询:")
|
||||
# result = j.check_whois_domain('921229.com')
|
||||
# logger.info(f"WHOIS查询结果: {result}")
|
||||
# logger.info(j.cookie)
|
||||
|
||||
# # # 测试备案查询
|
||||
# # logger.info("测试备案查询:")
|
||||
# # result = j.beian_check_domain('baidu.com')
|
||||
# # logger.info(f"备案查询结果: {result}")
|
||||
# #
|
||||
# # 测试安全检测
|
||||
# logger.info("测试安全检测:")
|
||||
# result = j.safe_check_domain('576777.com')
|
||||
# logger.info(f"安全检测结果: {result}")
|
||||
#
|
||||
# # j.save_cookies() # 保存Cookie
|
||||
497
domainCheck/detect/juming.py
Normal file
497
domainCheck/detect/juming.py
Normal file
@@ -0,0 +1,497 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :juming.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/3/19 15:51
|
||||
@explain : 聚名网API封装类 - 提供登录、域名查询、删除域名列表等功能
|
||||
'''
|
||||
|
||||
import base64 # Base64编码模块
|
||||
import hashlib # 哈希算法模块
|
||||
import json # JSON处理模块
|
||||
import os # 操作系统接口模块
|
||||
import pickle # 序列化模块
|
||||
import random # 随机数生成模块
|
||||
import subprocess # 子进程管理模块
|
||||
import time # 时间处理模块
|
||||
from functools import partial
|
||||
from typing import Optional, Tuple, Dict, List, Any # 类型提示
|
||||
|
||||
import requests # HTTP请求库
|
||||
from loguru import logger # 日志记录库
|
||||
from requests.cookies import RequestsCookieJar # Cookie处理
|
||||
|
||||
|
||||
# 常量定义
|
||||
BASE_URL = "https://www.juming.com" # 聚名网基础URL
|
||||
CAPTCHA_MAX_RETRY = 5 # 验证码最大重试次数
|
||||
REQUEST_TIMEOUT = 10 # 请求超时时间(秒)
|
||||
DOWNLOAD_TIMEOUT = 60 # 下载超时时间(秒)
|
||||
SEED_TIME_OFFSET = 286 # 种子时间偏移量
|
||||
SLIDE_OFFSET = 290 # 滑块偏移量
|
||||
SLIDE_DURATION = 611 # 滑动持续时间
|
||||
|
||||
|
||||
class MySubprocessPopen(subprocess.Popen): # 自定义子进程类
|
||||
def __init__(self, *args, **kwargs): # 初始化方法
|
||||
kwargs['encoding'] = "UTF-8" # 设置编码为UTF-8
|
||||
super().__init__(*args, **kwargs) # 调用父类初始化
|
||||
|
||||
|
||||
subprocess.Popen = MySubprocessPopen # 替换默认的Popen类
|
||||
os.environ["EXECJS_RUNTIME"] = "Node" # 设置JavaScript运行时为Node.js
|
||||
startupinfo = subprocess.STARTUPINFO()
|
||||
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
|
||||
subprocess.Popen = partial(subprocess.Popen, encoding='utf-8', startupinfo=startupinfo)
|
||||
|
||||
|
||||
def resolve_node_executable() -> str:
|
||||
current_dir = os.path.dirname(__file__)
|
||||
project_dir = os.path.dirname(current_dir)
|
||||
bundled_node = os.path.join(project_dir, "tools", "node-v20.19.4-win-x64", "node.exe")
|
||||
if os.path.exists(bundled_node):
|
||||
return bundled_node
|
||||
return "node"
|
||||
|
||||
def calculate_seed(t: int) -> str: # 计算验证码种子值
|
||||
now = int(time.time()) # 获取当前时间戳
|
||||
c = now - t # 计算时间差
|
||||
seed = base64.b64encode(str(c).encode()).decode() # Base64编码
|
||||
return seed # 返回种子字符串
|
||||
|
||||
|
||||
def random_fingerprint() -> str: # 生成随机设备指纹
|
||||
data = str(random.random()) + str(time.time()) # 组合随机数和时间戳
|
||||
return hashlib.sha256(data.encode()).hexdigest() # 返回SHA256哈希值
|
||||
|
||||
|
||||
def glwb(s: Optional[str]) -> str: # JS文本过滤函数
|
||||
"""
|
||||
过滤特殊字符,防止XSS攻击
|
||||
Args:
|
||||
s: 待过滤的字符串
|
||||
Returns:
|
||||
过滤后的字符串
|
||||
"""
|
||||
if s is None or not isinstance(s, str): # 检查输入是否有效
|
||||
return '' # 返回空字符串
|
||||
|
||||
a_nr = s # 复制字符串
|
||||
a_nr = a_nr.replace('"', '"') # 替换双引号
|
||||
a_nr = a_nr.replace("'", ''') # 替换单引号
|
||||
a_nr = a_nr.replace('<', '<') # 替换小于号
|
||||
a_nr = a_nr.replace('>', '>') # 替换大于号
|
||||
a_nr = a_nr.replace('\\', '\') # 替换反斜杠
|
||||
|
||||
return a_nr # 返回过滤后的字符串
|
||||
|
||||
|
||||
def md5_19(text: str) -> str: # 计算MD5并返回前19位
|
||||
"""
|
||||
计算MD5哈希值并截取前19位
|
||||
Args:
|
||||
text: 待哈希的文本
|
||||
Returns:
|
||||
MD5哈希值的前19位
|
||||
"""
|
||||
if not text: # 检查输入是否为空
|
||||
return '' # 返回空字符串
|
||||
md5 = hashlib.md5(text.encode('utf-8')).hexdigest() # 生成32位MD5
|
||||
return md5[:19] # 返回前19位
|
||||
|
||||
|
||||
def encrypt_password(loginToken: str, password: str) -> str: # 加密密码
|
||||
"""
|
||||
使用双重MD5加密密码
|
||||
Args:
|
||||
loginToken: 登录令牌
|
||||
password: 明文密码
|
||||
Returns:
|
||||
加密后的密码
|
||||
"""
|
||||
filtered_pwd = glwb(password) # 过滤密码中的特殊字符
|
||||
step1 = f'[jiami{filtered_pwd}mima]' # 拼接固定盐值
|
||||
md5_step1 = md5_19(step1) # 第一次MD5加密
|
||||
step2 = loginToken + md5_step1 # 拼接登录令牌
|
||||
final_result = md5_19(step2) # 第二次MD5加密
|
||||
return final_result # 返回最终加密结果
|
||||
|
||||
|
||||
class JM(object): # 聚名网API封装类
|
||||
token: str # 验证码token
|
||||
loginToken: str = '' # 登录令牌
|
||||
session_id: str # 会话ID
|
||||
fingerprint: str # 设备指纹
|
||||
captchaId: str # 验证码ID
|
||||
encryptionPublicKey: str # 加密公钥
|
||||
cookie: RequestsCookieJar = {} # Cookie存储
|
||||
|
||||
# 默认请求头
|
||||
headers = {
|
||||
'Host': 'www.juming.com', # 主机名
|
||||
'sec-ch-ua-platform': '"Windows"', # 平台标识
|
||||
'x-requested-with': 'XMLHttpRequest', # AJAX请求标识
|
||||
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0', # 用户代理
|
||||
'accept': '*/*', # 接受所有类型
|
||||
'sec-ch-ua': '"Chromium";v="146", "Not-A.Brand";v="24", "Microsoft Edge";v="146"', # 浏览器标识
|
||||
'sec-ch-ua-mobile': '?0', # 是否移动端
|
||||
'origin': 'https://www.juming.com', # 请求源
|
||||
'sec-fetch-site': 'same-origin', # 请求站点
|
||||
'sec-fetch-mode': 'cors', # 请求模式
|
||||
'sec-fetch-dest': 'empty', # 请求目标
|
||||
'referer': 'https://www.juming.com/', # 来源页面
|
||||
'accept-language': 'zh-CN,zh;q=0.9', # 接受的语言
|
||||
'priority': 'u=1, i', # 请求优先级
|
||||
'Content-Type': 'application/x-www-form-urlencoded', # 内容类型
|
||||
}
|
||||
|
||||
def __init__(self, proxies: Optional[Dict] = None): # 初始化方法
|
||||
self.session = requests.Session() # 创建会话对象
|
||||
self.session.timeout = REQUEST_TIMEOUT # 设置超时时间
|
||||
self.session.proxies = proxies # 设置代理
|
||||
self.base_url = BASE_URL # 设置基础URL
|
||||
logger.debug(proxies)
|
||||
|
||||
def captcha_init(self) -> Tuple[bool, str]: # 初始化滑块验证码
|
||||
"""
|
||||
初始化滑块验证码
|
||||
Returns:
|
||||
Tuple[bool, str]: (是否成功, 消息)
|
||||
"""
|
||||
self.fingerprint = random_fingerprint() # 生成随机指纹
|
||||
data = { # 构建请求数据
|
||||
"request_id": self.fingerprint, # 请求ID
|
||||
"scene": "default", # 场景
|
||||
"seed": calculate_seed(SEED_TIME_OFFSET) # 种子值
|
||||
}
|
||||
logger.info(data)
|
||||
self.headers['Content-Type'] = 'application/x-www-form-urlencoded' # 设置内容类型
|
||||
|
||||
url = f"{self.base_url}/captcha/init" # 初始化URL
|
||||
# self.cookie.clear()
|
||||
# self.cookie.update({'acw_sc__v2': '69d948ed49117d89433d4316a79ada6f00d68f26'})
|
||||
# logger.info(self.cookie)
|
||||
response = self.session.post(url, headers=self.headers, data=data, cookies=self.cookie).json() # 发送请求
|
||||
logger.info(response)
|
||||
|
||||
if response['code'] == 1: # 初始化成功
|
||||
self.captchaId = response['data']['captchaId'] # 保存验证码ID
|
||||
self.encryptionPublicKey = response['data']['encryptionPublicKey'] # 保存加密公钥
|
||||
return response['code'] == 1, response['msg'] # 返回结果
|
||||
|
||||
def captcha_verify(self) -> Tuple[bool, str]: # 验证滑块验证码
|
||||
"""
|
||||
验证滑块验证码
|
||||
Returns:
|
||||
Tuple[bool, str]: (是否成功, 消息)
|
||||
"""
|
||||
data = { # 构建验证数据
|
||||
"offset": SLIDE_OFFSET, # 滑块偏移量
|
||||
"duration": SLIDE_DURATION, # 滑动持续时间
|
||||
"trail": [ # 滑动轨迹
|
||||
{"x": 0, "y": 0, "time": 0}, # 起点
|
||||
{"x": 0, "y": 0, "time": 17}, # 第1个点
|
||||
{"x": 1, "y": 0, "time": 89}, # 第2个点
|
||||
{"x": 5, "y": 0, "time": 106}, # 第3个点
|
||||
{"x": 14, "y": 0, "time": 123}, # 第4个点
|
||||
{"x": 28, "y": 0, "time": 139}, # 第5个点
|
||||
{"x": 47, "y": 0, "time": 156}, # 第6个点
|
||||
{"x": 74, "y": 0, "time": 173}, # 第7个点
|
||||
{"x": 102, "y": 0, "time": 189}, # 第8个点
|
||||
{"x": 131, "y": 0, "time": 206}, # 第9个点
|
||||
{"x": 159, "y": 0, "time": 223}, # 第10个点
|
||||
{"x": 184, "y": 0, "time": 239}, # 第11个点
|
||||
{"x": 205, "y": 0, "time": 256}, # 第12个点
|
||||
{"x": 227, "y": 1, "time": 273}, # 第13个点
|
||||
{"x": 248, "y": 0, "time": 289}, # 第14个点
|
||||
{"x": 263, "y": 0, "time": 306}, # 第15个点
|
||||
{"x": 276, "y": 0, "time": 323}, # 第16个点
|
||||
{"x": 286, "y": 0, "time": 339}, # 第17个点
|
||||
{"x": 290, "y": 0, "time": 356}, # 第18个点
|
||||
{"x": 290, "y": 0, "time": 373}, # 第19个点
|
||||
{"x": 290, "y": 0, "time": 389}, # 第20个点
|
||||
{"x": 290, "y": 0, "time": 406}, # 第21个点
|
||||
{"x": 290, "y": 0, "time": 423}, # 第22个点
|
||||
{"x": 290, "y": 0, "time": 478}, # 终点
|
||||
],
|
||||
"fingerprint": self.fingerprint, # 设备指纹
|
||||
"captchaId": self.captchaId, # 验证码ID
|
||||
"serverPublicKey": self.encryptionPublicKey # 服务器公钥
|
||||
}
|
||||
data_str = json.dumps(data, separators=(",", ":")) # 转换为JSON字符串(压缩格式)
|
||||
current_dir = os.path.dirname(__file__)
|
||||
|
||||
jsFile_path = current_dir+"/sdk_leg_env.js" # JavaScript脚本路径
|
||||
|
||||
if not os.path.exists(jsFile_path): # 检查脚本文件是否存在
|
||||
logger.error(f"Node.js脚本文件不存在: {jsFile_path}") # 记录错误
|
||||
return False, f"缺少签名脚本: {jsFile_path}" # 返回失败
|
||||
|
||||
try: # 异常处理
|
||||
with subprocess.Popen( # 执行Node.js脚本
|
||||
[resolve_node_executable(), jsFile_path], # 命令和参数
|
||||
stdin=subprocess.PIPE, # 标准输入
|
||||
stdout=subprocess.PIPE, # 标准输出
|
||||
stderr=subprocess.PIPE, # 标准错误
|
||||
text=True, # 文本模式
|
||||
encoding="utf-8", # UTF-8编码
|
||||
errors="ignore" # 忽略编码错误
|
||||
) as proc: # 进程上下文
|
||||
stdout, stderr = proc.communicate(data_str, timeout=REQUEST_TIMEOUT) # 传入参数并获取输出
|
||||
verify_data = stdout.strip() # 去除空白字符
|
||||
|
||||
if stderr: # 如果有错误输出
|
||||
logger.error(f'genData_stderr->[{stderr}]') # 记录错误
|
||||
|
||||
if proc.returncode != 0: # 检查退出码
|
||||
return False, f"Node.js脚本执行失败,退出码: {proc.returncode}, 错误信息: {stderr}" # 返回失败
|
||||
|
||||
url = f"{self.base_url}/captcha/verify" # 验证URL
|
||||
self.headers['Content-Type'] = 'application/json' # 设置内容类型
|
||||
response = self.session.post(url, headers=self.headers, data=verify_data, cookies=self.cookie).json() # 发送验证请求
|
||||
if response['code'] == 1: # 验证成功
|
||||
self.token = response['data']['token'] # 保存token
|
||||
self.session_id = response['data']['session_id'] # 保存会话ID
|
||||
return response['code'] == 1, response['msg'] # 返回结果
|
||||
except subprocess.TimeoutExpired: # 超时异常
|
||||
logger.error("Node.js脚本执行超时") # 记录错误
|
||||
return False, "Node.js脚本执行超时" # 返回失败
|
||||
except Exception as e: # 其他异常
|
||||
logger.error(f"执行Node.js脚本异常: {str(e)}") # 记录错误
|
||||
return False, f"执行Node.js脚本异常: {str(e)}" # 返回失败
|
||||
|
||||
if not verify_data: # 检查输出是否为空
|
||||
return False, "Node.js脚本未输出任何内容" # 返回失败
|
||||
|
||||
def user_zh_wxdl_ewm(self) -> Tuple[bool, str, Optional[str]]: # 获取微信登录二维码
|
||||
"""
|
||||
获取微信登录二维码
|
||||
Returns:
|
||||
Tuple[bool, str, Optional[str]]: (是否成功, 消息, 二维码URL)
|
||||
"""
|
||||
url = f"{self.base_url}/user_zh/wxdl_ewm" # 二维码URL
|
||||
try: # 异常处理
|
||||
response = self.session.post(url, headers=self.headers).json() # 发送请求
|
||||
self.token = response['data']["token"] # 保存token
|
||||
return response['code'] == 1, response['msg'], response['data']['url'] if response['code'] == 1 else None # 返回结果
|
||||
except Exception as e: # 异常处理
|
||||
return False, f"请求失败: {str(e)}", None # 返回失败
|
||||
|
||||
def user_zh_p_login(self, email: str, password: str, data: Optional[Dict] = None) -> Tuple[bool, str]: # 账号密码登录
|
||||
"""
|
||||
使用账号密码登录
|
||||
Args:
|
||||
email: 邮箱账号
|
||||
password: 明文密码
|
||||
data: 携带的请求数据(用于递归刷新token时传参)
|
||||
Returns:
|
||||
Tuple[bool, str]: (是否成功, 消息)
|
||||
"""
|
||||
try: # 异常处理
|
||||
captcha_success = False # 验证码成功标志
|
||||
captcha_msg = f"滑块验证码失败,{CAPTCHA_MAX_RETRY}次内未成功" # 默认失败消息
|
||||
|
||||
for _ in range(CAPTCHA_MAX_RETRY): # 循环尝试验证滑块
|
||||
if data is None: # 如果没有传入data
|
||||
captcha_success, captcha_msg = self.captcha_init() # 初始化验证码
|
||||
else: # 如果已有data
|
||||
captcha_success = True # 直接视为验证通过
|
||||
|
||||
if captcha_success: # 验证码初始化成功
|
||||
if data is None: # 只有第一次需要验证滑块
|
||||
captcha_success, captcha_msg = self.captcha_verify() # 验证滑块
|
||||
|
||||
if captcha_success: # 滑块验证成功
|
||||
if data is None: # 首次登录
|
||||
data = { # 构建登录数据
|
||||
"re_mm": encrypt_password(self.loginToken, password), # 加密密码
|
||||
"re_yx": email, # 邮箱
|
||||
"fs": "tl", # 登录方式
|
||||
"dltoken": self.loginToken, # 登录令牌
|
||||
"token": json.dumps( # token(压缩格式JSON)
|
||||
{"code": self.token, "sessionId": self.session_id},
|
||||
separators=(",", ":")
|
||||
)
|
||||
}
|
||||
|
||||
login_url = f"{self.base_url}/user_zh/p_login" # 登录URL
|
||||
response = self.session.post( # 发送登录请求
|
||||
url=login_url,
|
||||
headers=self.headers,
|
||||
json=data,
|
||||
cookies=self.cookie,
|
||||
)
|
||||
logger.info(response.text)
|
||||
result = response.json() # 解析响应
|
||||
code = result.get("code") # 获取状态码
|
||||
msg = result.get("msg", "登录接口未返回消息") # 获取消息
|
||||
|
||||
if code == 1: # 登录成功
|
||||
self.cookie = response.cookies # 保存Cookie
|
||||
return True, msg # 返回成功
|
||||
|
||||
elif code == -118: # token过期
|
||||
self.loginToken = result.get("token", "") # 更新登录令牌
|
||||
data["re_mm"] = encrypt_password(self.loginToken, password) # 重新加密密码
|
||||
data["dltoken"] = self.loginToken # 更新令牌
|
||||
return self.user_zh_p_login(email, password, data) # 递归重新登录
|
||||
|
||||
else: # 其他错误
|
||||
return False, msg # 返回失败
|
||||
|
||||
return captcha_success, captcha_msg # 返回验证码结果
|
||||
|
||||
except Exception as e: # 异常处理
|
||||
return False, f"请求失败: {str(e)}" # 返回失败
|
||||
|
||||
def ykj_get_list(self, page: int = 1, page_size: int = 50, data: Optional[Dict] = None) -> Tuple[bool, Any]: # 获取一口价域名列表
|
||||
"""
|
||||
获取一口价域名列表
|
||||
Args:
|
||||
page: 页码
|
||||
page_size: 每页数量
|
||||
data: 额外数据
|
||||
Returns:
|
||||
Tuple[bool, Any]: (是否成功, HTML内容或错误消息)
|
||||
"""
|
||||
data = { # 构建请求数据
|
||||
'psize': page_size, # 每页数量
|
||||
'page': page, # 页码
|
||||
} if data is None else data # 如果有data则使用data
|
||||
url = f"{self.base_url}/ykj/get_list" # 列表URL
|
||||
try: # 异常处理
|
||||
self.headers['Content-Type'] = 'application/x-www-form-urlencoded' # 设置内容类型
|
||||
res = self.session.post(url, headers=self.headers, data=data, cookies=self.cookie) # 发送请求
|
||||
|
||||
response = res.json() # 解析响应
|
||||
if response['code'] == -401: # 需要验证码
|
||||
captcha_response = False, f'滑块验证码失败,{CAPTCHA_MAX_RETRY}次内未成功' # 默认失败
|
||||
for _ in range(CAPTCHA_MAX_RETRY): # 循环尝试验证
|
||||
captcha_response = self.captcha_init() # 初始化验证码
|
||||
if captcha_response[0]: # 初始化成功
|
||||
captcha_response = self.captcha_verify() # 验证验证码
|
||||
if captcha_response[0]: # 验证成功
|
||||
data = {
|
||||
'psize': page_size, # 每页数量
|
||||
'page': page, # 页码
|
||||
# 构建token数据
|
||||
'token': json.dumps({ # token(压缩格式JSON)
|
||||
'code': self.token,
|
||||
'sessionId': self.session_id,
|
||||
}, separators=(",", ":"))
|
||||
}
|
||||
return self.ykj_get_list(page, page_size,data=data) # 递归调用
|
||||
return captcha_response[0], captcha_response[1] # 返回验证码结果
|
||||
|
||||
elif response['code'] == 1 and response.get('data') == 'yzmhuaok': # 验证码通过
|
||||
self.cookie = res.cookies # 更新Cookie
|
||||
return self.ykj_get_list(page, page_size) # 递归调用
|
||||
|
||||
elif response['code'] == 1 and response.get('html'): # 成功获取HTML
|
||||
return response['code'] == 1, response['html'] # 返回HTML
|
||||
|
||||
return False, response['msg'] # 返回失败
|
||||
except Exception as e: # 异常处理
|
||||
return False, f"请求失败: {str(e)}" # 返回失败
|
||||
|
||||
def new_cha_del(self, date: str) -> List[str]: # 获取删除域名列表
|
||||
"""
|
||||
获取指定日期的删除域名列表
|
||||
Args:
|
||||
date: 日期(格式:YYYY-MM-DD)
|
||||
Returns:
|
||||
List[str]: 域名列表
|
||||
"""
|
||||
url = f"{self.base_url}/newcha/del_down?scsj={date}" # 下载URL
|
||||
try: # 异常处理
|
||||
response = self.session.get(url, headers=self.headers, cookies=self.cookie, allow_redirects=False) # 发送请求
|
||||
url = response.headers['Location'] # 获取重定向URL
|
||||
response = self.session.get(url, timeout=DOWNLOAD_TIMEOUT) # 下载文件
|
||||
response_text = response.content.decode('utf-8') # 解码内容
|
||||
lines = response_text.strip().splitlines() # 按行分割
|
||||
return lines # 返回域名列表
|
||||
except Exception as e: # 异常处理
|
||||
logger.error(f"<获取删除域名列表错误>: {str(e)}") # 记录错误
|
||||
return [] # 返回空列表
|
||||
|
||||
def save_cookies(self, filepath: str = "juming_cookies.pkl") -> None: # 保存Cookie到文件
|
||||
"""
|
||||
保存Cookie到文件和Redis
|
||||
Args:
|
||||
filepath: 文件路径
|
||||
"""
|
||||
with open(filepath, "wb") as f: # 以二进制写入模式打开文件
|
||||
pickle.dump(self.cookie, f) # 序列化保存
|
||||
|
||||
# 保存到Redis
|
||||
try:
|
||||
import redis
|
||||
from app.config import config
|
||||
redis_client = redis.Redis(
|
||||
host=config.REDIS_HOST,
|
||||
port=config.REDIS_PORT,
|
||||
password=config.REDIS_PASSWORD,
|
||||
db=config.REDIS_DB,
|
||||
decode_responses=True
|
||||
)
|
||||
# 将cookie转换为字典
|
||||
cookie_dict = {}
|
||||
for cookie in self.cookie:
|
||||
cookie_dict[cookie.name] = cookie.value
|
||||
redis_client.set('domain_tool:juming_cookies', str(cookie_dict))
|
||||
except Exception as e:
|
||||
logger.error(f"保存Cookie到Redis失败: {str(e)}")
|
||||
pass
|
||||
|
||||
def load_cookies(self, filepath: str = "juming_cookies.pkl") -> None: # 从文件加载Cookie
|
||||
"""
|
||||
从文件加载Cookie
|
||||
Args:
|
||||
filepath: 文件路径
|
||||
"""
|
||||
try: # 异常处理
|
||||
with open(filepath, "rb") as f: # 以二进制读取模式打开文件
|
||||
self.cookie = pickle.load(f) # 反序列化加载
|
||||
except: # 异常处理
|
||||
self.cookie = requests.cookies.RequestsCookieJar() # 创建空Cookie
|
||||
|
||||
|
||||
# if __name__ == '__main__': # 主程序入口
|
||||
# m = JM() # 创建JM实例
|
||||
#
|
||||
# # 账号密码登录(请替换为您的账号密码)
|
||||
# email = 'chaofanai1998@gmail.com' # 邮箱账号
|
||||
# password = 'llzz123,./' # 密码
|
||||
#
|
||||
# logger.info(f"开始登录: {email}") # 记录日志
|
||||
# login_result = m.user_zh_p_login(email, password) # 执行登录
|
||||
# logger.info(f"登录结果: {login_result}") # 记录结果
|
||||
#
|
||||
# if not login_result[0]: # 登录失败
|
||||
# logger.error("登录失败,程序退出") # 记录错误
|
||||
# exit(1) # 退出程序
|
||||
#
|
||||
# m.load_cookies() # 加载Cookie
|
||||
#
|
||||
# # 获取一口价域名
|
||||
# import re # 导入正则表达式模块
|
||||
#
|
||||
# for page in range(1, 2): # 循环获取页面
|
||||
# success, html = m.ykj_get_list(page) # 获取列表
|
||||
# if success: # 获取成功
|
||||
# pattern_ym = r"<a class='yda1 ydz' ym='([^']*)'" # 匹配域名
|
||||
# results = re.findall(pattern_ym, html) # 查找所有域名
|
||||
# logger.info(results)
|
||||
# logger.info(f"第{page}页,找到{len(results)}个域名") # 记录结果
|
||||
# if not results: # 如果没有域名
|
||||
# logger.info("没有更多域名,停止获取") # 记录日志
|
||||
# break # 退出循环
|
||||
# time.sleep(5) # 等待5秒
|
||||
#
|
||||
# # 获取删除域名
|
||||
# deleted_domains = m.new_cha_del("2026-03-11") # 获取删除域名
|
||||
# logger.info(f"找到{len(deleted_domains)}个删除域名") # 记录结果
|
||||
#
|
||||
# m.save_cookies() # 保存Cookie
|
||||
236
domainCheck/detect/juziseo.py
Normal file
236
domainCheck/detect/juziseo.py
Normal file
@@ -0,0 +1,236 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :juziseo.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/10 15:00
|
||||
@explain : 桔子SEO API封装类 - 提供登录、域名查询等功能
|
||||
'''
|
||||
|
||||
# 导入标准库
|
||||
import os # 操作系统接口
|
||||
import pickle # 序列化反序列化
|
||||
import re
|
||||
import time # 时间处理
|
||||
from Crypto.Cipher import AES
|
||||
from Crypto.Util.Padding import pad
|
||||
import base64
|
||||
# 导入第三方库
|
||||
import requests # HTTP请求库
|
||||
from loguru import logger # 日志记录
|
||||
from requests.cookies import RequestsCookieJar # Cookie管理
|
||||
|
||||
from detect.geetest2 import Geetest2
|
||||
|
||||
|
||||
|
||||
# AES-CBC 加密
|
||||
def aes_cbc_encrypt(data, key=b'pvjxzjmzwawfscft', iv=b'qvibva1wg0uxwjeu'):
|
||||
cipher = AES.new(key, AES.MODE_CBC, iv)
|
||||
padded_data = pad(data, AES.block_size) # PKCS7填充
|
||||
encrypted = cipher.encrypt(padded_data)
|
||||
return base64.b64encode(encrypted).decode()
|
||||
|
||||
|
||||
class Juziseo:
|
||||
"""
|
||||
桔子SEO API封装类
|
||||
"""
|
||||
cookie: RequestsCookieJar = {} # 桔子SEO Cookie
|
||||
|
||||
# 通用请求头配置
|
||||
headers = {
|
||||
'accept': 'application/json, text/javascript, */*; q=0.01', # 接受的内容类型
|
||||
'accept-language': 'zh-CN,zh;q=0.9', # 接受的语言
|
||||
'content-type': 'application/x-www-form-urlencoded', # 内容类型
|
||||
'origin': 'https://seo.juziseo.com', # 请求源
|
||||
'priority': 'u=1, i', # 请求优先级
|
||||
'referer': 'https://seo.juziseo.com/login', # 来源页面
|
||||
'sec-ch-ua': '"Chromium";v="146", "Not-A.Brand";v="24", "Google Chrome";v="146"', # 浏览器标识
|
||||
'sec-ch-ua-mobile': '?0', # 是否移动端
|
||||
'sec-ch-ua-platform': '"Windows"', # 操作系统平台
|
||||
'sec-fetch-dest': 'empty', # 请求目标
|
||||
'sec-fetch-mode': 'cors', # 请求模式
|
||||
'sec-fetch-site': 'same-origin', # 请求站点
|
||||
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36',
|
||||
# 用户代理
|
||||
'x-requested-with': 'XMLHttpRequest', # AJAX请求标识
|
||||
}
|
||||
|
||||
def __init__(self, proxies: dict = None):
|
||||
"""
|
||||
初始化桔子SEO类
|
||||
|
||||
:param proxies: 代理配置
|
||||
"""
|
||||
self.validate = None
|
||||
self.challenge = None
|
||||
self.gt = None
|
||||
self.session = requests.Session() # 创建会话对象
|
||||
self.session.proxies = proxies # 设置代理
|
||||
self.session.timeout = 10 # 设置超时时间10秒
|
||||
self.base_url = "https://seo.juziseo.com" # 设置基础URL
|
||||
|
||||
def save_cookies(self, filepath="juziseo_cookies.pkl"):
|
||||
"""
|
||||
保存Cookie到文件和Redis
|
||||
|
||||
:param filepath: 文件路径
|
||||
"""
|
||||
with open(filepath, "wb") as f: # 以二进制写入模式打开文件
|
||||
pickle.dump(self.cookie, f) # 序列化保存Cookie
|
||||
logger.info(f"已保存桔子SEO Cookie到 {filepath}")
|
||||
|
||||
# 保存到Redis
|
||||
try:
|
||||
import redis
|
||||
from app.config import config
|
||||
redis_client = redis.Redis(
|
||||
host=config.REDIS_HOST,
|
||||
port=config.REDIS_PORT,
|
||||
password=config.REDIS_PASSWORD,
|
||||
db=config.REDIS_DB,
|
||||
decode_responses=True
|
||||
)
|
||||
# 将cookie转换为字典
|
||||
cookie_dict = {}
|
||||
for cookie in self.cookie:
|
||||
cookie_dict[cookie.name] = cookie.value
|
||||
redis_client.set('domain_tool:juziseo_cookies', str(cookie_dict))
|
||||
except Exception as e:
|
||||
pass
|
||||
|
||||
def load_cookies(self, filepath="juziseo_cookies.pkl"):
|
||||
"""
|
||||
从文件加载Cookie
|
||||
|
||||
:param filepath: 文件路径
|
||||
"""
|
||||
try: # 尝试加载
|
||||
with open(filepath, "rb") as f: # 以二进制读取模式打开文件
|
||||
self.cookie = pickle.load(f) # 反序列化加载Cookie
|
||||
# logger.info(f"已从 {filepath} 加载桔子SEO Cookie")
|
||||
except: # 加载失败
|
||||
self.cookie = requests.cookies.RequestsCookieJar() # 创建空的CookieJar
|
||||
logger.warning(f"加载桔子SEO Cookie失败,创建空Cookie")
|
||||
|
||||
def start_msg_captcha_servlet(self):
|
||||
url = f"{self.base_url}/class/gtcode/msg/StartMsgCaptchaServlet.php"
|
||||
response = self.session.get(url, headers=self.headers, cookies=self.cookie).json()
|
||||
logger.info(response)
|
||||
self.gt = response['gt']
|
||||
self.challenge = response['challenge']
|
||||
return response['success'] == 1
|
||||
|
||||
def get_captcha(self):
|
||||
GETT2 = Geetest2()
|
||||
code = GETT2.get_tp(gt=self.gt, challenge=self.challenge, type_='auto')
|
||||
logger.info(code)
|
||||
if code['result'] == 'success':
|
||||
self.challenge = code['challenge']
|
||||
self.gt = code['gt']
|
||||
self.validate = code['validate']
|
||||
return True
|
||||
else:
|
||||
logger.error(f"获取桔子SEO验证码失败: {code.get('msg', '未知错误')}")
|
||||
return False
|
||||
|
||||
def login(self, username, password):
|
||||
"""
|
||||
登录桔子SEO
|
||||
|
||||
:param username: 账号
|
||||
:param password: 密码
|
||||
:return: tuple - (是否成功, 消息)
|
||||
"""
|
||||
try:
|
||||
for _ in range(3):
|
||||
if self.start_msg_captcha_servlet():
|
||||
if self.get_captcha():
|
||||
break
|
||||
data = {
|
||||
'return_url': '/',
|
||||
'user_name': aes_cbc_encrypt(bytes(username, 'utf-8')),
|
||||
'password': aes_cbc_encrypt(bytes(password, 'utf-8')),
|
||||
'geetest_challenge': self.challenge,
|
||||
'geetest_validate': self.validate,
|
||||
'geetest_seccode': self.validate + '|jordan',
|
||||
'_post_type': 'ajax',
|
||||
}
|
||||
logger.debug(data)
|
||||
# 发送登录请求
|
||||
url = f"{self.base_url}/account/ajax/login_process/"
|
||||
response = self.session.post(url, headers=self.headers, data=data, cookies=self.cookie)
|
||||
logger.info(response.text)
|
||||
# 解析响应
|
||||
result = response.json()
|
||||
|
||||
if result.get('errno') == 1:
|
||||
# 登录成功,保存Cookie
|
||||
self.cookie = response.cookies
|
||||
self.save_cookies()
|
||||
logger.info("桔子SEO登录成功")
|
||||
return True, "登录成功"
|
||||
else:
|
||||
logger.error(f"桔子SEO登录失败: {result.get('msg', '未知错误')}")
|
||||
return False, result.get('msg', '登录失败')
|
||||
except Exception as e:
|
||||
logger.error(f"桔子SEO登录异常: {e}")
|
||||
return False, f"登录失败: {str(e)}"
|
||||
|
||||
def check_history(self, domain: str, sensitive_words: list = None):
|
||||
# 检测域名历史是否存在敏感词
|
||||
|
||||
if sensitive_words is None:
|
||||
sensitive_words = []
|
||||
data = {
|
||||
'qrtypeindex': '1',
|
||||
'domains': domain,
|
||||
'_post_type': 'ajax',
|
||||
}
|
||||
url = f"{self.base_url}/snapshot/save/"
|
||||
response = self.session.post(url, headers=self.headers, data=data, cookies=self.cookie).json()
|
||||
# logger.info(response)
|
||||
if response['errno'] == 1:
|
||||
url = response['rsm']['url']
|
||||
response_html = self.session.get(url, headers=self.headers, cookies=self.cookie).content.decode('utf-8')
|
||||
title_sensitive_words_match = re.search(r'标题敏感词\D*(\d+)', response_html, re.S)
|
||||
title_suspected_sensitive_words_match = re.search(r'标题有疑似敏感词\D*(\d+)', response_html, re.S)
|
||||
content_sensitive_words_match = re.search(r'内容敏感词\D*(\d+)', response_html, re.S)
|
||||
baidu_sensitive_words_match = re.search(r'百度历史收录敏感\D*(\d+)', response_html, re.S)
|
||||
if bool(title_sensitive_words_match or title_suspected_sensitive_words_match or content_sensitive_words_match or baidu_sensitive_words_match):
|
||||
return False, "存在敏感词"
|
||||
subdomain_match = re.search(r'子域名:\D*(\d+)', response_html, re.S)
|
||||
if subdomain_match:
|
||||
return False, f"存在子域名: {subdomain_match.group(1)}"
|
||||
|
||||
for sensitive_word in sensitive_words:
|
||||
if sensitive_word in response_html:
|
||||
return False, f"存在敏感词: {sensitive_word}"
|
||||
return True, 'success'
|
||||
return False, str(response.get('err', '请求失败'))
|
||||
|
||||
def check_external_link(self, domain: str, sensitive_words: list = None):
|
||||
# 外链查询域名是否存在敏感词
|
||||
if sensitive_words is None:
|
||||
sensitive_words = []
|
||||
data = {
|
||||
'qrtypeindex': '1',
|
||||
'domains': domain,
|
||||
'_post_type': 'ajax',
|
||||
}
|
||||
url = f"{self.base_url}/domain_rank/save_domain/"
|
||||
response = self.session.post(url, headers=self.headers, data=data, cookies=self.cookie).json()
|
||||
logger.info(response)
|
||||
if response['errno'] == 1:
|
||||
url = response['rsm']['url']
|
||||
response_html = self.session.get(url, headers=self.headers, cookies=self.cookie).content.decode('utf-8')
|
||||
subdomain_match = re.search(r'子域名:\D*(\d+)', response_html, re.S)
|
||||
if subdomain_match:
|
||||
return False, f"存在子域名: {subdomain_match.group(1)}"
|
||||
for sensitive_word in sensitive_words:
|
||||
if sensitive_word in response_html:
|
||||
return False, f"存在敏感词: {sensitive_word}"
|
||||
return True, 'success'
|
||||
return False, str(response.get('err', '请求失败'))
|
||||
3851
domainCheck/detect/module/crack_geetest2x.js
Normal file
3851
domainCheck/detect/module/crack_geetest2x.js
Normal file
File diff suppressed because one or more lines are too long
53
domainCheck/detect/module/gap.py
Normal file
53
domainCheck/detect/module/gap.py
Normal file
@@ -0,0 +1,53 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
import base64,re
|
||||
import time, os,datetime, sched
|
||||
import cv2
|
||||
from io import BytesIO
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
import requests,re
|
||||
import numpy as np
|
||||
|
||||
|
||||
class quekou():
|
||||
def __init__(self):
|
||||
self.t = int(time.time() * 1000)
|
||||
pass
|
||||
def save_png(self,path,con):
|
||||
with open(path, "wb") as fp:
|
||||
fp.write(con)
|
||||
|
||||
def get_distance2(self,bg,tp):
|
||||
|
||||
bg = base64.b64decode(bg) # base64转二进制
|
||||
|
||||
tp = base64.b64decode(tp) # base64转二进制
|
||||
|
||||
res = self.det.slide_match(tp, bg, simple_target=True)
|
||||
res=res['target'][0]
|
||||
|
||||
return res
|
||||
|
||||
def get_distance(self, bg, tp):
|
||||
|
||||
bg = base64.b64decode(bg) # base64转二进制
|
||||
|
||||
tp = base64.b64decode(tp) # base64转二进制
|
||||
''' bg: 背景图片 tp: 缺口图片 out:输出图片 '''
|
||||
# 读取背景图片和缺口图片
|
||||
bg_img = Image.open(BytesIO(bg)) # 背景图片
|
||||
tp_img = Image.open(BytesIO(tp)) # 缺口图片
|
||||
bg_edge = cv2.Canny(np.array(bg_img), 100, 200)
|
||||
tp_edge = cv2.Canny(np.array(tp_img), 100, 200)
|
||||
# 转换图片格式
|
||||
bg_pic = cv2.cvtColor(bg_edge, cv2.COLOR_GRAY2RGB)
|
||||
tp_pic = cv2.cvtColor(tp_edge, cv2.COLOR_GRAY2RGB)
|
||||
# 缺口匹配
|
||||
res = cv2.matchTemplate(bg_pic, tp_pic, cv2.TM_CCOEFF_NORMED)
|
||||
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res) # 寻找最优匹配
|
||||
# 绘制方框
|
||||
tl = max_loc # 左上角点的坐标
|
||||
return tl[0]
|
||||
|
||||
|
||||
|
||||
|
||||
85
domainCheck/detect/module/use_ua.py
Normal file
85
domainCheck/detect/module/use_ua.py
Normal file
@@ -0,0 +1,85 @@
|
||||
# encoding: utf-8
|
||||
|
||||
import requests
|
||||
|
||||
import random
|
||||
from loguru import logger
|
||||
|
||||
msg="提示:返回出现'ip overtime','forbidden'等信息说明本机IP被目标网站限制,请使用/更换代理IP或更换电脑即可。"
|
||||
warning="仅用于学习交流,请勿用于非法用途,违者后果自负!"
|
||||
logger.warning(warning)
|
||||
logger.info(msg)
|
||||
UA = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/44.0.2403.155 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X; pl-PL; rv:1.0.1) Gecko/20021111 Chimera/0.6",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418.8 (KHTML, like Gecko, Safari) Cheshire/1.0.UNOFFICIAL",
|
||||
"Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.8.1b2) Gecko/20060821 BonEcho/2.0b2 (Debian-1.99+2.0b2+dfsg-1)",
|
||||
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
|
||||
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
|
||||
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0",
|
||||
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
|
||||
"Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
|
||||
"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
|
||||
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
|
||||
"Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
|
||||
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
|
||||
"Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
|
||||
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
|
||||
"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
|
||||
"Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
|
||||
"Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
|
||||
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
|
||||
"Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
|
||||
"Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
|
||||
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
|
||||
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 TaoBrowser/2.0 Safari/536.11",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.71 Safari/537.1 LBBROWSER",
|
||||
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; LBBROWSER)",
|
||||
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E; LBBROWSER)",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 LBBROWSER",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
|
||||
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; QQBrowser/7.0.3698.400)",
|
||||
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SV1; QQDownload 732; .NET4.0C; .NET4.0E; 360SE)",
|
||||
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 732; .NET4.0C; .NET4.0E)",
|
||||
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)",
|
||||
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1",
|
||||
"Mozilla/5.0 (iPad; U; CPU OS 4_2_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5",
|
||||
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13pre",
|
||||
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) Gecko/20100101 Firefox/16.0",
|
||||
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11",
|
||||
"Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10",
|
||||
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1464.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.16 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; CrOS i686 3912.101.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0.6",
|
||||
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1468.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2224.3 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; CrOS i686 3912.101.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36"
|
||||
]
|
||||
|
||||
def randomUA():
|
||||
return {"User-Agent": random.choice(UA)}
|
||||
|
||||
|
||||
# if __name__ == '__main__':
|
||||
#
|
||||
# print(randomUA())
|
||||
75
domainCheck/detect/register.py
Normal file
75
domainCheck/detect/register.py
Normal file
@@ -0,0 +1,75 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :register.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/3/27 22:55
|
||||
@explain : 域名注册状态检测工具
|
||||
'''
|
||||
|
||||
from datetime import datetime, timezone, timedelta # 日期时间处理,用于时区转换
|
||||
|
||||
import requests # HTTP请求库
|
||||
from loguru import logger # 日志记录
|
||||
|
||||
|
||||
def check_register(domain: str, postfix: str = 'com',proxies: dict = None):
|
||||
'''
|
||||
检测注册状态
|
||||
:param domain: 待检测域名(不包含后缀)
|
||||
:param postfix: 域名后缀com、net(默认com)
|
||||
:param proxies: 代理(默认None)
|
||||
:return: 注册状态(2:可注册,3:已注册,-1:检测失败,过期时间(格式:YYYY-MM-DD HH:MM:SS)为空)
|
||||
'''
|
||||
url = f'https://rdap.verisign.com/{postfix}/v1/domain/{domain}' # 构建RDAP查询URL
|
||||
# url = f'https://www.baidu.com' # 构建RDAP查询URL
|
||||
|
||||
# 添加重试机制
|
||||
max_retries = 3
|
||||
for retry in range(max_retries):
|
||||
try:
|
||||
response = requests.get(url, timeout=20, proxies=proxies) # 发送GET请求,设置超时5秒
|
||||
if response.status_code == 200: # HTTP 200表示域名已注册
|
||||
json_data = response.json() # 解析JSON响应
|
||||
for item in json_data['events']: # 遍历事件列表
|
||||
if item['eventAction'] == 'expiration': # 如果是过期时间事件
|
||||
utc_time = datetime.fromisoformat(item['eventDate'].replace('Z', '+00:00')) # 解析UTC时间
|
||||
beijing_tz = timezone(timedelta(hours=8)) # 创建北京时区(UTC+8)
|
||||
beijing_time = utc_time.astimezone(beijing_tz) # 将UTC时间转换为北京时间
|
||||
return 3, beijing_time.strftime("%Y-%m-%d %H:%M:%S") # 返回已注册状态和过期时间
|
||||
|
||||
elif response.status_code == 404: # HTTP 404表示域名不存在,可注册
|
||||
# 可注册状态
|
||||
return 2, '' # 返回可注册状态,过期时间为空
|
||||
logger.error(response.status_code)
|
||||
except requests.exceptions.RequestException as e:
|
||||
# logger.warning(f"注册状态检测第{retry+1}次失败: {domain}, 错误: {e}")
|
||||
if retry < max_retries - 1:
|
||||
import time
|
||||
time.sleep(2) # 等待2秒后重试
|
||||
proxies=None
|
||||
else:
|
||||
# 达到最大重试次数,返回检测失败
|
||||
logger.error(f"注册状态检测失败: {domain}, 已达到最大重试次数")
|
||||
return -1, '' # -1表示检测失败,过期时间为空
|
||||
|
||||
|
||||
|
||||
|
||||
# def get_proxy():
|
||||
#
|
||||
# url='http://api.ch12361.com/getProxy.php?group=A&count=1'
|
||||
# response = requests.get(url).json()
|
||||
# username=response['username']
|
||||
# password=response['password']
|
||||
# ip=response['ip']
|
||||
# port=response['port']
|
||||
#
|
||||
# proxy_url = f"http://{username}:{password}@{ip}:{port}"
|
||||
# logger.info(proxy_url)
|
||||
# return {'http':proxy_url,'https':proxy_url}
|
||||
#
|
||||
# t=datetime.now()
|
||||
# logger.info(check_register('75az.com',proxies=get_proxy())) # 测试查询nnsk.com域名的注册状态
|
||||
# logger.info(datetime.now()-t)
|
||||
15108
domainCheck/detect/sdk_leg.js
Normal file
15108
domainCheck/detect/sdk_leg.js
Normal file
File diff suppressed because one or more lines are too long
897
domainCheck/detect/sdk_leg_env.js
Normal file
897
domainCheck/detect/sdk_leg_env.js
Normal file
@@ -0,0 +1,897 @@
|
||||
process_ = process;
|
||||
require_ = require;
|
||||
delete Buffer;
|
||||
// delete process;
|
||||
delete require;
|
||||
delete global;
|
||||
delete module;
|
||||
delete exports;
|
||||
delete __filename;
|
||||
delete __dirname;
|
||||
delete SharedArrayBuffer;
|
||||
|
||||
AsObj = {
|
||||
// print: console.log,
|
||||
print: function () { },
|
||||
// print_:console.log,
|
||||
}
|
||||
|
||||
no_print = ['Boolean','String','parseFloat','Array','Object','prepareStackTrace_'];
|
||||
function watch(object, WatchName) {
|
||||
const handler = {
|
||||
get(target, property, receiver) {
|
||||
if (
|
||||
property !== 'isNaN' &&
|
||||
property !== 'encodeURI' &&
|
||||
property !== "Uint8Array" &&
|
||||
property !== 'undefined' &&
|
||||
property !== 'JSON' &&
|
||||
property !== 'Number' &&
|
||||
!no_print.includes(property) &&
|
||||
property !== Symbol.for('nodejs.util.inspect.custom') &&
|
||||
typeof property !== 'symbol'
|
||||
) {
|
||||
|
||||
if (property === 'global') {
|
||||
return undefined;
|
||||
}
|
||||
if (property === 'Buffer') {
|
||||
return undefined;
|
||||
}
|
||||
if (property === 'process') {
|
||||
return undefined;
|
||||
}
|
||||
if (WatchName === 'config_data') {
|
||||
debugger
|
||||
}
|
||||
if (WatchName.indexOf('.prototype') != -1 && target[property] != undefined) {
|
||||
return Reflect.get(target, property, receiver);
|
||||
}
|
||||
|
||||
AsObj.print(
|
||||
"方法:", "get",
|
||||
"对象:", WatchName,
|
||||
"属性:", property,
|
||||
"属性类型:", typeof property,
|
||||
"属性值:", typeof target[property] == 'object' ? "object" : target[property],
|
||||
"属性值类型:", typeof target[property]
|
||||
);
|
||||
}
|
||||
|
||||
if (WatchName === 'top') {
|
||||
return window;
|
||||
}
|
||||
|
||||
return Reflect.get(target, property, receiver);
|
||||
},
|
||||
|
||||
set(target, property, value, receiver) {
|
||||
if (WatchName.indexOf('.prototype') != -1 && value != undefined) {
|
||||
return Reflect.set(target, property, value, receiver);
|
||||
}
|
||||
AsObj.print(
|
||||
"方法:", "set",
|
||||
"对象:", WatchName,
|
||||
"属性:", property,
|
||||
"属性类型:", typeof property,
|
||||
"属性值:", typeof value == 'object' ? "object" : value,
|
||||
"属性值类型:", typeof target[property]
|
||||
);
|
||||
return Reflect.set(target, property, value, receiver);
|
||||
},
|
||||
// in操作 检测
|
||||
has(target, property) {
|
||||
AsObj.print(
|
||||
"代理对象:", WatchName,
|
||||
"方法:", "has",
|
||||
"检查属性:", property,
|
||||
"结果:", typeof target[property] == 'object' ? "object" : target[property],
|
||||
);
|
||||
return Reflect.has(target, property);
|
||||
},
|
||||
// Object.key 检测
|
||||
ownKeys(target) {
|
||||
AsObj.print(
|
||||
"方法:", "ownKeys",
|
||||
"对象:", target+''
|
||||
);
|
||||
return Reflect.ownKeys(target);
|
||||
}
|
||||
};
|
||||
|
||||
return new Proxy(object, handler);
|
||||
}
|
||||
// function watch(object, WatchName) {
|
||||
// return object
|
||||
// }
|
||||
|
||||
// 保护函数,toString检测
|
||||
const safeFunction = function safeFunction(func) {
|
||||
//处理安全函数
|
||||
Function.prototype.$call = Function.prototype.call;
|
||||
const $toString = Function.toString;
|
||||
const myFunction_toString_symbol = Symbol('('.concat('', ')'));
|
||||
|
||||
const myToString = function myToString() {
|
||||
return typeof this === 'function' && this[myFunction_toString_symbol] || $toString.$call(this);
|
||||
}
|
||||
|
||||
const set_native = function set_native(func, key, value) {
|
||||
Object.defineProperty(func, key, {
|
||||
"enumerable": false,
|
||||
"configurable": true,
|
||||
"writable": true,
|
||||
"value": value
|
||||
});
|
||||
}
|
||||
|
||||
delete Function.prototype['toString'];
|
||||
set_native(Function.prototype, "toString", myToString);
|
||||
set_native(Function.prototype.toString, myFunction_toString_symbol, "function toString() { [native code] }");
|
||||
|
||||
const safe_Function = function safe_Function(func) {
|
||||
set_native(func, myFunction_toString_symbol, "function" + (func.name ? " " + func.name : "") + "() { [native code] }");
|
||||
}
|
||||
|
||||
return safe_Function(func)
|
||||
}
|
||||
|
||||
//创建函数,并代理上
|
||||
const makeFunction = function makeFunction(name) {
|
||||
v_log = AsObj.print;
|
||||
// 使用 Function 保留函数名
|
||||
func = new Function("v_log", `
|
||||
return function ${name}() {
|
||||
v_log('函数${name}传参-->', arguments);
|
||||
};
|
||||
`)(v_log); // 传递 v_log 到动态函数
|
||||
|
||||
safeFunction(func);
|
||||
func = watch(func,`${name}`);
|
||||
func.prototype = watch(func.prototype, `${name}.prototype`);
|
||||
return func;
|
||||
}
|
||||
|
||||
!(function () {
|
||||
"use strict";
|
||||
const $toString = Function.toString;
|
||||
const myFunction_toString_symbol = Symbol('('.concat('', ')_', (Math.random() + '').toString(36)));
|
||||
const mytoString = function () {
|
||||
return typeof this == 'function' && this[myFunction_toString_symbol] || $toString.call(this);
|
||||
};
|
||||
|
||||
function set_native(func, key, value) {
|
||||
Object.defineProperty(func, key, {
|
||||
"enumerable": false,
|
||||
"configurable": true,
|
||||
"writable": true,
|
||||
"value": value
|
||||
})
|
||||
};
|
||||
delete Function.prototype['toString'];
|
||||
set_native(Function.prototype, "toString", mytoString);
|
||||
set_native(Function.prototype.toString, myFunction_toString_symbol, "function toString() { [native code] }");
|
||||
this.func_set_native = function (func) {
|
||||
set_native(func, myFunction_toString_symbol, `function ${myFunction_toString_symbol, func.name || ''}() { [native code] }`)
|
||||
}
|
||||
}).call(globalThis);
|
||||
|
||||
// 重写全局对象原型链
|
||||
function setTostringAndstringTag(obj) {
|
||||
Object.defineProperties(obj.prototype, {
|
||||
[Symbol.toStringTag]: {
|
||||
configurable: true,
|
||||
value: obj.name
|
||||
}
|
||||
});
|
||||
safeFunction(obj);
|
||||
};
|
||||
|
||||
// 创建标签原型
|
||||
function createTagProto(propObj,portotypeObj) {
|
||||
let res = propObj + ' = ' + 'function ' + propObj + '() { throw new TypeError("Illegal constructor"); };\n';
|
||||
res += 'setTostringAndstringTag(' + propObj + ',null);\n';
|
||||
if (portotypeObj) {
|
||||
for (let key in portotypeObj) {
|
||||
res += propObj + '.prototype.' + portotypeObj[key] + '= function ' + portotypeObj[key] + '() {AsObj.print("'+propObj+'.prototype.' + portotypeObj[key] + '原型方法(需在实例对象上补该方法)::",arguments)};\n';
|
||||
res += 'globalThis.func_set_native(' + propObj + '.prototype.' + portotypeObj[key] + ');\n';
|
||||
}
|
||||
}
|
||||
eval(res);
|
||||
}
|
||||
|
||||
Object.defineProperties(globalThis, {
|
||||
[Symbol.toStringTag]: {
|
||||
configurable: true,
|
||||
value: 'Window'
|
||||
}
|
||||
});
|
||||
|
||||
for (let key in globalThis) {
|
||||
if (typeof globalThis[key] === 'function') {
|
||||
safeFunction(globalThis[key])
|
||||
}
|
||||
}
|
||||
for (let key in console) {
|
||||
if (typeof console[key] === 'function') {
|
||||
safeFunction(console[key])
|
||||
}
|
||||
}
|
||||
|
||||
createTagProto('EventTarget',['addEventListener']);
|
||||
createTagProto('WindowProperties');
|
||||
createTagProto('Window');
|
||||
|
||||
window = globalThis;
|
||||
window.__proto__ = Window.prototype;
|
||||
window.__proto__.__proto__ = WindowProperties.prototype;
|
||||
window.__proto__.__proto__.__proto__ = EventTarget.prototype;
|
||||
Window.__proto__ = EventTarget;
|
||||
|
||||
Object.defineProperty(window, 'WindowProperties', {
|
||||
get: function () {
|
||||
return undefined;
|
||||
}
|
||||
})
|
||||
|
||||
function randoms(min, max) {
|
||||
return Math.floor(Math.random() * (max - min + 1) + min)
|
||||
}
|
||||
|
||||
function getRandomValues(buf) {
|
||||
var min = 0,
|
||||
max = 255;
|
||||
if (buf instanceof Uint16Array) {
|
||||
max = 65535;
|
||||
} else if (buf instanceof Uint32Array) {
|
||||
max = 4294967295;
|
||||
}
|
||||
for (var element in buf) {
|
||||
buf[element] = randoms(min, max);
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
|
||||
self = window.self = window;
|
||||
frames = window.frames = window;
|
||||
top = window.top = window;
|
||||
parent = window.parent = window;
|
||||
global = window.global = window;
|
||||
|
||||
Object.defineProperty(window, "global", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function global(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "top", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function top(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "self", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function self(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "parent", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function parent(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
Object.defineProperty(window, "frames", {
|
||||
configurable:false,
|
||||
enumerable: true,
|
||||
set: undefined,
|
||||
get: function frames(){
|
||||
return window
|
||||
}
|
||||
})
|
||||
|
||||
innerWidth = 1536
|
||||
innerHeight = 715
|
||||
outerWidth = 1536
|
||||
outerHeight = 824
|
||||
devicePixelRatio = 1.25;
|
||||
screenLeft = 0;
|
||||
screenX = 0;
|
||||
screenTop = 0;
|
||||
screenY = 0;
|
||||
opener = null;
|
||||
isSecureContext = true;
|
||||
crypto = {
|
||||
getRandomValues:getRandomValues
|
||||
};
|
||||
|
||||
createTagProto('DOMStringMap')
|
||||
createTagProto('HTMLHeadElement',['insertBefore','removeChild'])
|
||||
createTagProto('HTMLBodyElement',['addEventListener','appendChild','removeChild'])
|
||||
createTagProto('HTMLHtmlElement',['getAttribute'])
|
||||
createTagProto('HTMLDocument')
|
||||
createTagProto('Document',['browsingTopics','appendChild','querySelector','evaluate','querySelectorAll','removeChild','requestStorageAccess','requestStorageAccessFor','hasStorageAccess','getElementsByTagName','hasPrivateToken','createElement','hasRedemptionRecord','hasFocus'])
|
||||
createTagProto('Node')
|
||||
document = {};
|
||||
document.__proto__ = HTMLDocument.prototype;
|
||||
document.__proto__.__proto__ = Document.prototype;
|
||||
document.__proto__.__proto__.__proto__ = Node.prototype;
|
||||
document.__proto__.__proto__.__proto__.__proto__ = EventTarget.prototype;
|
||||
HTMLDocument.__proto__ = Document;
|
||||
HTMLDocument.__proto__.__proto__ = Node;
|
||||
HTMLDocument.__proto__.__proto__.__proto__ = EventTarget;
|
||||
Document.__proto__ = Node;
|
||||
Document.__proto__.__proto__ = EventTarget;
|
||||
Node.__proto__ = EventTarget;
|
||||
|
||||
createTagProto('Plugin');
|
||||
createTagProto('PluginArray');
|
||||
plugins0 = {
|
||||
name: 'PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type: 'application/pdf',
|
||||
},
|
||||
'1':{
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins0['0'].enabledPlugin = plugins0;
|
||||
plugins0['1'].enabledPlugin = plugins0;
|
||||
plugins1 = {
|
||||
name: 'Chrome PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1': {
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins1['0'].enabledPlugin = plugins1;
|
||||
plugins1['1'].enabledPlugin = plugins1;
|
||||
plugins2 = {
|
||||
name: 'Chromium PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1': {
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins2['0'].enabledPlugin = plugins2;
|
||||
plugins2['1'].enabledPlugin = plugins2;
|
||||
plugins3 = {
|
||||
name: 'Microsoft Edge PDF Viewer',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0':{
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1': {
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins3['0'].enabledPlugin = plugins3;
|
||||
plugins3['1'].enabledPlugin = plugins3;
|
||||
plugins4 = {
|
||||
name: 'WebKit built-in PDF',
|
||||
filename: 'internal-pdf-viewer',
|
||||
description:'Portable Document Format',
|
||||
length: 2,
|
||||
'0': {
|
||||
type:'application/pdf'
|
||||
},
|
||||
'1':{
|
||||
type:'text/pdf'
|
||||
}
|
||||
}
|
||||
plugins4['0'].enabledPlugin = plugins4;
|
||||
plugins4['1'].enabledPlugin = plugins4;
|
||||
plugins = {
|
||||
length: 5,
|
||||
'0': plugins0,
|
||||
'1': plugins1,
|
||||
'2': plugins2,
|
||||
'3': plugins3,
|
||||
'4': plugins4,
|
||||
namedItem : function (name) {
|
||||
AsObj.print('Plugin-namedItem:', name)
|
||||
},
|
||||
item: function (index) {
|
||||
AsObj.print('Plugin-item:', index)
|
||||
return watch(plugins0,'item-'+index);
|
||||
},
|
||||
refresh: function () {
|
||||
AsObj.print('Plugin-refresh:',arguments)
|
||||
},
|
||||
}
|
||||
plugins.__proto__ = PluginArray.prototype;
|
||||
|
||||
MimeTypeArray = function MimeTypeArray() {
|
||||
this.length = 2;
|
||||
this['0'] = {
|
||||
suffixes: 'pdf',
|
||||
type: 'application/pdf',
|
||||
description:"Portable Document Format",
|
||||
enabledPlugin: plugins0
|
||||
};
|
||||
this['1'] = {
|
||||
suffixes: 'pdf',
|
||||
type: 'text/pdf',
|
||||
description:"Portable Document Format",
|
||||
enabledPlugin: plugins0
|
||||
};
|
||||
};
|
||||
MimeTypeArray.prototype.toString = function () { return '[object MimeTypeArray]'; }
|
||||
MimeTypeArray.toString = function () { return 'function MimeTypeArray() { [native code] }'; }
|
||||
Object.defineProperties(MimeTypeArray.prototype, { [Symbol.toStringTag]: { value: 'MimeTypeArray' } })
|
||||
MimeTypeArrayc = new MimeTypeArray();
|
||||
MimeTypeArrayc[Symbol.iterator] = function* () {
|
||||
for (let key in this) {
|
||||
yield this[key];
|
||||
}
|
||||
}
|
||||
|
||||
// 创建电池管理器对象原型
|
||||
const BatteryManager = {
|
||||
level: 1,
|
||||
charging: true,
|
||||
chargingTime: 0,
|
||||
dischargingTime: null,
|
||||
onchargingchange: null,
|
||||
onlevelchange: null,
|
||||
toString: function toString() {
|
||||
return `BatteryManager {
|
||||
charging: ${this.charging},
|
||||
level: ${this.level},
|
||||
chargingTime: ${this.chargingTime},
|
||||
dischargingTime: ${this.dischargingTime}
|
||||
}`
|
||||
}
|
||||
}
|
||||
window.BatteryManager = BatteryManager;
|
||||
|
||||
Promise2 = {
|
||||
then: function () {
|
||||
return this;
|
||||
},
|
||||
catch: function (){},
|
||||
};
|
||||
|
||||
createTagProto('Bluetooth');
|
||||
createTagProto('Navigator');
|
||||
Navigator.prototype.hardwareConcurrency = 8;
|
||||
Navigator.prototype.userAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36';
|
||||
Navigator.prototype.appVersion = '5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36'
|
||||
Navigator.prototype.appName = 'Netscape';
|
||||
Navigator.prototype.appCodeName = 'Mozilla';
|
||||
Navigator.prototype.vendor = 'Google Inc.';
|
||||
Navigator.prototype.maxTouchPoints = 10;
|
||||
Navigator.prototype.platform = 'Win32';
|
||||
Navigator.prototype.adAuctionComponents = function adAuctionComponents() {
|
||||
AsObj.print('adAuctionComponents:::', arguments)
|
||||
}
|
||||
safeFunction(Navigator.prototype.adAuctionComponents)
|
||||
Navigator.prototype.runAdAuction = function runAdAuction() {
|
||||
AsObj.print('runAdAuction:::', arguments)
|
||||
}
|
||||
safeFunction(Navigator.prototype.runAdAuction)
|
||||
Navigator.prototype.canLoadAdAuctionFencedFrame = makeFunction('canLoadAdAuctionFencedFrame')
|
||||
Navigator.prototype.deprecatedReplaceInURN = makeFunction('deprecatedReplaceInURN')
|
||||
Navigator.prototype.deprecatedURNToURL = makeFunction('deprecatedURNToURL')
|
||||
Navigator.prototype.joinAdInterestGroup = makeFunction('joinAdInterestGroup')
|
||||
Navigator.prototype.leaveAdInterestGroup = makeFunction('leaveAdInterestGroup')
|
||||
Navigator.prototype.updateAdInterestGroups = makeFunction('updateAdInterestGroups')
|
||||
Navigator.prototype.connection = watch({
|
||||
downlink: 9.1,
|
||||
effectiveType: '4g',
|
||||
rtt: 0,
|
||||
saveData: false,
|
||||
},'connection')
|
||||
Navigator.prototype.language = 'zh-CN';
|
||||
Navigator.prototype.languages = ["zh-CN"];
|
||||
Navigator.prototype.plugins = plugins;
|
||||
Navigator.prototype.webdriver = false;
|
||||
Navigator.prototype.cookieEnabled = true;
|
||||
Navigator.prototype.onLine = true;
|
||||
Navigator.prototype.doNotTrack = null;
|
||||
Navigator.prototype.bluetooth = {};
|
||||
Navigator.prototype.product = 'Gecko'
|
||||
Navigator.prototype.deviceMemory = 8
|
||||
Navigator.prototype.mediaDevices = watch({
|
||||
enumerateDevices: function enumerateDevices() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = [
|
||||
{deviceId: '', kind: 'audioinput', label: '', groupId: ''},
|
||||
{deviceId: '', kind: 'videoinput', label: '', groupId: ''},
|
||||
{deviceId: '', kind: 'audiooutput', label: '', groupId: ''},
|
||||
]
|
||||
resolve(offer);
|
||||
});
|
||||
},
|
||||
getUserMedia: function getUserMedia() {
|
||||
AsObj.print('getUserMedia:::', arguments)
|
||||
}
|
||||
},'mediaDevices')
|
||||
Navigator.prototype.storage = {
|
||||
estimate: function estimate() {
|
||||
AsObj.print('estimate:::', arguments)
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
usage: 0, // 1GB
|
||||
quota: 2147483648, // 1GB,
|
||||
usageDetails: {caches: 512, indexedDB: 2855}
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
}
|
||||
Navigator.prototype.webkitPersistentStorage = watch({},'webkitPersistentStorage')
|
||||
Navigator.prototype.webkitTemporaryStorage = watch({
|
||||
queryUsageAndQuota: function queryUsageAndQuota() {
|
||||
AsObj.print('queryUsageAndQuota:::', arguments)
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
usage: 1024 * 1024 * 1024, // 1GB
|
||||
quota: 1024 * 1024 * 1024, // 1GB
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
},'webkitTemporaryStorage')
|
||||
Navigator.prototype.bluetooth.__proto__ = Bluetooth.prototype;
|
||||
Navigator.prototype.javaEnabled = function javaEnabled() {
|
||||
return false
|
||||
};
|
||||
safeFunction(Navigator.prototype.javaEnabled)
|
||||
Navigator.prototype.getBattery = function getBattery() {
|
||||
AsObj.print('getBattery:::', arguments)
|
||||
return Promise.resolve({
|
||||
__proto__: BatteryManager,
|
||||
// 动态参数配置(示例值)
|
||||
level: 1,
|
||||
charging: true,
|
||||
dischargingTime: null // 2小时放电时间
|
||||
})
|
||||
}
|
||||
safeFunction(Navigator.prototype.getBattery)
|
||||
Navigator.prototype.registerProtocolHandler = function registerProtocolHandler() {
|
||||
AsObj.print('registerProtocolHandler:::',arguments)
|
||||
}
|
||||
safeFunction(Navigator.prototype.registerProtocolHandler)
|
||||
Navigator.prototype.mimeTypes = watch(MimeTypeArrayc,'mimeTypes');
|
||||
Navigator.prototype.geolocation = {
|
||||
getCurrentPosition: function getCurrentPosition() {
|
||||
return Promise2;
|
||||
}
|
||||
}
|
||||
Navigator.prototype.pdfViewerEnabled = true;
|
||||
Navigator.prototype.doNotTrack = null;
|
||||
Navigator.prototype.keyboard = watch({
|
||||
getLayoutMap: function getLayoutMap() {
|
||||
AsObj.print('Navigator.prototype.keyboard:', arguments)
|
||||
return {
|
||||
then: function () {
|
||||
// arguments[0](watch({
|
||||
// size: 48,
|
||||
// values: function () {
|
||||
// return ['k', 'g', '2', '0', 'v', 'a', '`', 'l', '\\', "'", 'w', '8', 'm', 'h', '.', '7', '1', 'p', 'd', 'f', 'o', 'q', 'c', 'n', '[', 'z', 'y', '3', '6', '5', 'x', '/', '\\', ',', '-', '4', 'b', 't', '9', 's', 'i', 'u', '=', 'j', ';', 'r', ']', 'e']
|
||||
// }
|
||||
// }, 'navigator.keyboard.getLayoutMap.then'))
|
||||
return {
|
||||
catch: function () {
|
||||
arguments[0]({
|
||||
message:'getLayoutMap() must be called from a top-level browsing context or allowed by the permission policy.'
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},'navigator.keyboard')
|
||||
Navigator.prototype.permissions = watch({
|
||||
query: function query() {
|
||||
arg_obj = arguments[0];
|
||||
if (arg_obj.name === 'audio_capture') {
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
state: 'prompt',
|
||||
onchange: null,
|
||||
name:arg_obj.name
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
if (arg_obj.name === 'microphone') {
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0](watch({
|
||||
state: 'denied',
|
||||
onchange: null,
|
||||
name: 'audio_capture'
|
||||
},'permissions.query.microphone'));
|
||||
return {catch:function(){}}
|
||||
},
|
||||
catch:function(){}
|
||||
}
|
||||
}
|
||||
if (arg_obj.name === 'camera') {
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0](watch({
|
||||
state: 'prompt',
|
||||
onchange: null,
|
||||
name: 'video_capture'
|
||||
},'permissions.query.camera'));
|
||||
return {catch:function(){}}
|
||||
},
|
||||
catch:function(){}
|
||||
}
|
||||
}
|
||||
AsObj.print('permissions.query:::', arguments)
|
||||
|
||||
}
|
||||
},'permissions')
|
||||
Navigator.prototype.productSub = '20030107'
|
||||
Navigator.prototype.getGamepads = function getGamepads() {
|
||||
AsObj.print('getGamepads:::', arguments)
|
||||
return [null,null,null,null]
|
||||
}
|
||||
safeFunction(Navigator.prototype.getGamepads)
|
||||
|
||||
Navigator.prototype.sendBeacon = makeFunction('sendBeacon')
|
||||
|
||||
Navigator.prototype.deprecatedRunAdAuctionEnforcesKAnonymity = false
|
||||
Navigator.prototype.gpu = watch({
|
||||
getPreferredCanvasFormat: function getPreferredCanvasFormat() {
|
||||
AsObj.print('gpu.getPreferredCanvasFormat:', arguments)
|
||||
return 'bgra8unorm'
|
||||
},
|
||||
wgslLanguageFeatures: watch({
|
||||
size: 7,
|
||||
values: function values() {
|
||||
debugger
|
||||
AsObj.print('wgslLanguageFeatures.values')
|
||||
return ['packed_4x8_integer_dot_product', 'unrestricted_pointer_parameters', 'subgroup_uniformity', 'subgroup_id', 'pointer_composite_access', 'readonly_and_readwrite_storage_textures', 'uniform_buffer_standard_layout']
|
||||
},
|
||||
}, 'gpu.wgslLanguageFeatures'),
|
||||
requestAdapter: function requestAdapter() {
|
||||
AsObj.print('gpu.requestAdapter:', arguments)
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0](watch({
|
||||
features: watch({
|
||||
size: 19,
|
||||
values: function () {
|
||||
return ['depth32float-stencil8', 'rg11b10ufloat-renderable', 'bgra8unorm-storage', 'texture-formats-tier1', 'texture-compression-bc', 'dual-source-blending', 'core-features-and-limits', 'float32-filterable', 'indirect-first-instance', 'float32-blendable', 'depth-clip-control', 'texture-compression-bc-sliced-3d', 'timestamp-query', 'texture-formats-tier2', 'clip-distances', 'shader-f16', 'primitive-index', 'texture-component-swizzle', 'subgroups']
|
||||
}
|
||||
}, 'gpu.requestAdapter.features'),
|
||||
info: watch({ vendor: 'intel', architecture: 'gen-11', device: '', description: '', subgroupMinSize: 16 }, 'gpu.requestAdapter.info'),
|
||||
limits: watch({
|
||||
maxBufferSize: 2147483648,
|
||||
maxStorageBufferBindingSize:2147483644
|
||||
}, 'gpu.requestAdapter.limits'),
|
||||
catch:function(){}
|
||||
}, 'gpu.requestAdapter'));
|
||||
return {
|
||||
catch: function () {
|
||||
return {
|
||||
then: function () {
|
||||
arguments[0]()
|
||||
return {catch:function(){}}
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
},
|
||||
catch: function () {
|
||||
}
|
||||
}
|
||||
}
|
||||
},'navigator.gpu')
|
||||
Navigator.prototype.userAgentData = watch({
|
||||
brands:[
|
||||
{
|
||||
"brand": "Google Chrome",
|
||||
"version": "143"
|
||||
},
|
||||
{
|
||||
"brand": "Chromium",
|
||||
"version": "143"
|
||||
},
|
||||
{
|
||||
"brand": "Not A(Brand",
|
||||
"version": "24"
|
||||
}
|
||||
],
|
||||
mobile: false,
|
||||
platform: "Windows",
|
||||
getHighEntropyValues: function getHighEntropyValues() {
|
||||
if (arguments[0] + '' === 'architecture,bitness,model,platformVersion,uaFullVersion,wow64') {
|
||||
return new Promise((resolve, reject) => {
|
||||
const offer = {
|
||||
"architecture": "x86",
|
||||
"bitness": "64",
|
||||
"brands": [
|
||||
{
|
||||
"brand": "Not:A-Brand",
|
||||
"version": "99"
|
||||
},
|
||||
{
|
||||
"brand": "Google Chrome",
|
||||
"version": "145"
|
||||
},
|
||||
{
|
||||
"brand": "Chromium",
|
||||
"version": "145"
|
||||
}
|
||||
],
|
||||
"mobile": false,
|
||||
"model": "",
|
||||
"platform": "Windows",
|
||||
"platformVersion": "10.0.0",
|
||||
"uaFullVersion": "145.0.7632.117",
|
||||
"wow64": false
|
||||
};
|
||||
resolve(offer);
|
||||
});
|
||||
}
|
||||
AsObj.print('getHighEntropyValues:::', arguments)
|
||||
}
|
||||
},'userAgentData')
|
||||
|
||||
navigator = {};
|
||||
navigator.__proto__ = Navigator.prototype;
|
||||
|
||||
createTagProto('Location');
|
||||
location = {
|
||||
"ancestorOrigins": {},
|
||||
"href": "https://www.neimanmarcus.com/",
|
||||
"origin": "https://www.neimanmarcus.com",
|
||||
"protocol": "https:",
|
||||
"host": "www.neimanmarcus.com",
|
||||
"hostname": "www.neimanmarcus.com",
|
||||
"port": "",
|
||||
"pathname": "/",
|
||||
"search": "",
|
||||
"hash": ""
|
||||
};
|
||||
|
||||
location.__proto__ = Location.prototype;
|
||||
location.toString = function toString() {
|
||||
return this.href;
|
||||
}
|
||||
|
||||
createTagProto('Screen');
|
||||
Screen.prototype = Object.assign(Screen.prototype, {
|
||||
availHeight: 824,
|
||||
availLeft: 0,
|
||||
availTop: 0,
|
||||
availWidth: 1536,
|
||||
colorDepth: 32,
|
||||
height: 864,
|
||||
isExtended: true,
|
||||
onchange: null,
|
||||
pixelDepth: 24,
|
||||
width: 1536,
|
||||
orientation: {
|
||||
angle: 0,
|
||||
type: "landscape-primary",
|
||||
onchange: null
|
||||
}
|
||||
})
|
||||
screen = {};
|
||||
screen.__proto__ = Screen.prototype;
|
||||
|
||||
createTagProto('History',['replaceState']);
|
||||
history = {};
|
||||
history.__proto__ = History.prototype;
|
||||
|
||||
chrome = {
|
||||
loadTimes: function loadTimes() { },
|
||||
csi: function csi() { },
|
||||
app: {
|
||||
InstallState: { DISABLED: 'disabled', INSTALLED: 'installed', NOT_INSTALLED: 'not_installed' },
|
||||
RunningState: { CANNOT_RUN: 'cannot_run', READY_TO_RUN: 'ready_to_run', RUNNING: 'running' },
|
||||
getDetails:function getDetails(){},
|
||||
getIsInstalled:function getIsInstalled(){},
|
||||
installState:function installState(){},
|
||||
isInstalled: false,
|
||||
runningState: function runningState(){}
|
||||
},
|
||||
}
|
||||
|
||||
createTagProto('Storage');
|
||||
local = {
|
||||
};
|
||||
localStorage = {
|
||||
getItem: function getItem(key) {
|
||||
AsObj.print("localStorage.getItem::", arguments);
|
||||
if (!local[key]) {
|
||||
return null;
|
||||
}
|
||||
return local[key];
|
||||
},
|
||||
setItem: function setItem(key, value) {
|
||||
AsObj.print("localStorage.setItem::", arguments);
|
||||
local[key] = value;
|
||||
},
|
||||
clear: function clear() {
|
||||
local = {};
|
||||
},
|
||||
removeItem: function removeItem(key) {
|
||||
AsObj.print("localStorage.removeItem::", arguments);
|
||||
delete local[key];
|
||||
}
|
||||
}
|
||||
localStorage.__proto__ = Storage.prototype;
|
||||
sessionStorage = {
|
||||
getItem: function getItem(key) {
|
||||
AsObj.print("sessionStorage.getItem::", arguments);
|
||||
if (!local[key]) {
|
||||
return null;
|
||||
}
|
||||
return local[key];
|
||||
},
|
||||
setItem: function setItem(key, value) {
|
||||
AsObj.print("sessionStorage.setItem::", arguments);
|
||||
local[key] = value;
|
||||
},
|
||||
clear: function clear() {
|
||||
local = {};
|
||||
},
|
||||
removeItem: function removeItem(key) {
|
||||
AsObj.print("sessionStorage.removeItem::", arguments);
|
||||
delete local[key];
|
||||
}
|
||||
}
|
||||
sessionStorage.__proto__ = Storage.prototype;
|
||||
|
||||
// window = watch(window, 'window');
|
||||
// global = watch(global, 'global');
|
||||
// globalThis = watch(globalThis, 'globalThis');
|
||||
// self = watch(self, 'self');
|
||||
// crypto = watch(crypto, 'crypto');
|
||||
// performance = watch(performance, 'performance');
|
||||
// document = watch(document, 'document');
|
||||
// navigator = watch(navigator, 'navigator');
|
||||
// location = watch(location, 'location');
|
||||
// screen = watch(screen, 'screen');
|
||||
// history = watch(history, 'history');
|
||||
// localStorage = watch(localStorage, 'localStorage');
|
||||
// sessionStorage = watch(sessionStorage, 'sessionStorage');
|
||||
// chrome = watch(chrome, 'chrome');
|
||||
|
||||
require_('./sdk_leg.js');
|
||||
|
||||
let input = '';
|
||||
// 收集数据
|
||||
process.stdin.on('data', chunk => {
|
||||
input += chunk;
|
||||
});
|
||||
process.stdin.on('end', async () => {
|
||||
var config_data = JSON.parse(input);
|
||||
var cryptoManager = await CaptchaSDKCorecc();
|
||||
var encryptData = await buildEncryptedVerifyRequestcc(config_data, cryptoManager);
|
||||
console.log(JSON.stringify(encryptData));
|
||||
process.exit(0);
|
||||
})
|
||||
20
domainCheck/detect_options.json
Normal file
20
domainCheck/detect_options.json
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"detect_juziseo": false,
|
||||
"detect_360_site": true,
|
||||
"detect_aizhan": true,
|
||||
"detect_baidu_site": true,
|
||||
"detect_order": [
|
||||
"detect_register",
|
||||
"detect_baidu_site",
|
||||
"detect_360_site",
|
||||
"detect_chinaz",
|
||||
"detect_aizhan",
|
||||
"detect_wayback",
|
||||
"detect_jucha",
|
||||
"detect_juziseo"
|
||||
],
|
||||
"detect_register": true,
|
||||
"detect_wayback": true,
|
||||
"detect_chinaz": true,
|
||||
"detect_jucha": false
|
||||
}
|
||||
2181
domainCheck/detect_worker.py
Normal file
2181
domainCheck/detect_worker.py
Normal file
File diff suppressed because it is too large
Load Diff
3
domainCheck/domain_suffixes.json
Normal file
3
domainCheck/domain_suffixes.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"suffixes": ".com,.net"
|
||||
}
|
||||
BIN
domainCheck/favicon.ico
Normal file
BIN
domainCheck/favicon.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.2 KiB |
BIN
domainCheck/favicon2.ico
Normal file
BIN
domainCheck/favicon2.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.2 KiB |
270
domainCheck/init_database.py
Normal file
270
domainCheck/init_database.py
Normal file
@@ -0,0 +1,270 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :init_database.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/9 21:20
|
||||
@explain : 数据库初始化脚本
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from loguru import logger
|
||||
from app.config import config
|
||||
|
||||
|
||||
def init_database():
|
||||
"""
|
||||
初始化数据库表结构
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
logger.info("数据库连接成功")
|
||||
|
||||
# 创建domains表
|
||||
create_domains_table = """
|
||||
CREATE TABLE IF NOT EXISTS domains (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain VARCHAR(255) UNIQUE NOT NULL,
|
||||
tld VARCHAR(50) NOT NULL,
|
||||
source_type INTEGER NOT NULL,
|
||||
use_status INTEGER DEFAULT 0,
|
||||
detect_status INTEGER DEFAULT 0,
|
||||
register_status INTEGER DEFAULT 0,
|
||||
has_beian BOOLEAN DEFAULT FALSE,
|
||||
company_type VARCHAR(100),
|
||||
website_url VARCHAR(255),
|
||||
beian_year INTEGER,
|
||||
snapshot_years TEXT,
|
||||
expire_date TIMESTAMP,
|
||||
create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
update_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
"""
|
||||
cur.execute(create_domains_table)
|
||||
logger.info("创建domains表成功")
|
||||
|
||||
# 添加新字段
|
||||
try:
|
||||
# 添加review_status字段
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS review_status INTEGER DEFAULT 0")
|
||||
# 添加detect_time字段
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS detect_time TIMESTAMP")
|
||||
# 添加backlink_count字段
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS backlink_count INTEGER DEFAULT 0")
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS jucha_status INTEGER DEFAULT 0")
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS juziseo_status INTEGER DEFAULT 0")
|
||||
# 尝试修改has_beian字段类型
|
||||
try:
|
||||
# 先检查字段类型
|
||||
cur.execute("SELECT data_type FROM information_schema.columns WHERE table_name = 'domains' AND column_name = 'has_beian'")
|
||||
result = cur.fetchone()
|
||||
if result and result[0] != 'integer':
|
||||
# 先删除默认值
|
||||
cur.execute("ALTER TABLE domains ALTER COLUMN has_beian DROP DEFAULT")
|
||||
# 修改字段类型
|
||||
cur.execute("ALTER TABLE domains ALTER COLUMN has_beian TYPE INTEGER USING CASE WHEN has_beian THEN 2 ELSE 3 END")
|
||||
# 设置新的默认值
|
||||
cur.execute("ALTER TABLE domains ALTER COLUMN has_beian SET DEFAULT 1")
|
||||
except Exception as e:
|
||||
logger.warning(f"修改has_beian字段类型失败: {e}")
|
||||
logger.info("添加新字段成功")
|
||||
except Exception as e:
|
||||
logger.warning(f"添加新字段失败: {e}")
|
||||
# 回滚当前事务
|
||||
conn.rollback()
|
||||
# 只添加新字段,不修改has_beian字段类型
|
||||
try:
|
||||
# 添加review_status字段
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS review_status INTEGER DEFAULT 0")
|
||||
# 添加detect_time字段
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS detect_time TIMESTAMP")
|
||||
# 添加backlink_count字段
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS backlink_count INTEGER DEFAULT 0")
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS jucha_status INTEGER DEFAULT 0")
|
||||
cur.execute("ALTER TABLE domains ADD COLUMN IF NOT EXISTS juziseo_status INTEGER DEFAULT 0")
|
||||
logger.info("添加部分新字段成功")
|
||||
except Exception as e2:
|
||||
logger.warning(f"添加新字段失败: {e2}")
|
||||
|
||||
# 添加字段注释
|
||||
cur.execute("COMMENT ON COLUMN domains.id IS '主键ID'")
|
||||
cur.execute("COMMENT ON COLUMN domains.domain IS '域名'")
|
||||
cur.execute("COMMENT ON COLUMN domains.tld IS '顶级域名'")
|
||||
cur.execute("COMMENT ON COLUMN domains.source_type IS '来源类型'")
|
||||
cur.execute("COMMENT ON COLUMN domains.use_status IS '使用状态'")
|
||||
cur.execute("COMMENT ON COLUMN domains.detect_status IS '检测状态'")
|
||||
cur.execute("COMMENT ON COLUMN domains.register_status IS '注册状态'")
|
||||
cur.execute("COMMENT ON COLUMN domains.review_status IS '人工复核状态'")
|
||||
cur.execute("COMMENT ON COLUMN domains.has_beian IS '是否有备案历史'")
|
||||
cur.execute("COMMENT ON COLUMN domains.company_type IS '单位性质'")
|
||||
cur.execute("COMMENT ON COLUMN domains.website_url IS '网站首页网址'")
|
||||
cur.execute("COMMENT ON COLUMN domains.beian_year IS '备案年份'")
|
||||
cur.execute("COMMENT ON COLUMN domains.snapshot_years IS '快照年份'")
|
||||
cur.execute("COMMENT ON COLUMN domains.expire_date IS '过期时间'")
|
||||
cur.execute("COMMENT ON COLUMN domains.detect_time IS '检测时间'")
|
||||
cur.execute("COMMENT ON COLUMN domains.jucha_status IS '聚查检测状态:0未检测,1已检测'")
|
||||
cur.execute("COMMENT ON COLUMN domains.juziseo_status IS '桔子检测状态:0未检测,1已检测'")
|
||||
cur.execute("COMMENT ON COLUMN domains.create_time IS '创建时间'")
|
||||
cur.execute("COMMENT ON COLUMN domains.update_time IS '更新时间'")
|
||||
logger.info("添加domains表字段注释成功")
|
||||
|
||||
# 创建detect_tasks表
|
||||
create_tasks_table = """
|
||||
CREATE TABLE IF NOT EXISTS detect_tasks (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain_id INTEGER REFERENCES domains(id),
|
||||
task_type INTEGER NOT NULL,
|
||||
status INTEGER DEFAULT 0,
|
||||
priority INTEGER DEFAULT 0,
|
||||
retry_count INTEGER DEFAULT 0,
|
||||
create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
update_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
"""
|
||||
cur.execute(create_tasks_table)
|
||||
logger.info("创建detect_tasks表成功")
|
||||
|
||||
# 添加字段注释
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.id IS '主键ID'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.domain_id IS '域名ID'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.task_type IS '任务类型'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.status IS '任务状态'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.priority IS '优先级'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.retry_count IS '重试次数'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.create_time IS '创建时间'")
|
||||
cur.execute("COMMENT ON COLUMN detect_tasks.update_time IS '更新时间'")
|
||||
logger.info("添加detect_tasks表字段注释成功")
|
||||
|
||||
# 创建domain_blacklist表
|
||||
create_blacklist_table = """
|
||||
CREATE TABLE IF NOT EXISTS domain_blacklist (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain VARCHAR(255) UNIQUE NOT NULL,
|
||||
reason TEXT,
|
||||
create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
"""
|
||||
cur.execute(create_blacklist_table)
|
||||
logger.info("创建domain_blacklist表成功")
|
||||
|
||||
# 添加字段注释
|
||||
cur.execute("COMMENT ON COLUMN domain_blacklist.id IS '主键ID'")
|
||||
cur.execute("COMMENT ON COLUMN domain_blacklist.domain IS '域名'")
|
||||
cur.execute("COMMENT ON COLUMN domain_blacklist.reason IS '黑名单原因'")
|
||||
cur.execute("COMMENT ON COLUMN domain_blacklist.create_time IS '创建时间'")
|
||||
logger.info("添加domain_blacklist表字段注释成功")
|
||||
|
||||
# 创建sensitive_words表
|
||||
create_sensitive_words_table = """
|
||||
CREATE TABLE IF NOT EXISTS sensitive_words (
|
||||
id SERIAL PRIMARY KEY,
|
||||
word VARCHAR(255) UNIQUE NOT NULL,
|
||||
create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
"""
|
||||
cur.execute(create_sensitive_words_table)
|
||||
logger.info("创建sensitive_words表成功")
|
||||
|
||||
# 添加缺失的字段
|
||||
try:
|
||||
# 添加category字段
|
||||
cur.execute("ALTER TABLE sensitive_words ADD COLUMN IF NOT EXISTS category VARCHAR(50) DEFAULT 'default'")
|
||||
# 添加priority字段
|
||||
cur.execute("ALTER TABLE sensitive_words ADD COLUMN IF NOT EXISTS priority INTEGER DEFAULT 1")
|
||||
logger.info("添加sensitive_words表字段成功")
|
||||
except Exception as e:
|
||||
logger.warning(f"添加sensitive_words表字段失败: {e}")
|
||||
|
||||
# 添加字段注释
|
||||
try:
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.id IS '主键ID'")
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.word IS '敏感词'")
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.category IS '分类'")
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.priority IS '优先级'")
|
||||
cur.execute("COMMENT ON COLUMN sensitive_words.create_time IS '创建时间'")
|
||||
logger.info("添加sensitive_words表字段注释成功")
|
||||
except Exception as e:
|
||||
logger.warning(f"添加sensitive_words表字段注释失败: {e}")
|
||||
|
||||
# 创建domain_detections表
|
||||
create_detections_table = """
|
||||
CREATE TABLE IF NOT EXISTS domain_detections (
|
||||
id SERIAL PRIMARY KEY,
|
||||
domain_id INTEGER REFERENCES domains(id),
|
||||
baidu_history JSONB,
|
||||
baidu_site JSONB,
|
||||
qihu360_site JSONB,
|
||||
google_site JSONB,
|
||||
chinaz_info JSONB,
|
||||
aizhan_info JSONB,
|
||||
juziseo_info JSONB,
|
||||
jucha_info JSONB,
|
||||
same_url BOOLEAN,
|
||||
is_chinese_title BOOLEAN,
|
||||
backlink_count_gt_10 BOOLEAN,
|
||||
create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
update_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
"""
|
||||
cur.execute(create_detections_table)
|
||||
logger.info("创建domain_detections表成功")
|
||||
cur.execute("CREATE UNIQUE INDEX IF NOT EXISTS idx_domain_detections_domain_id ON domain_detections(domain_id)")
|
||||
|
||||
# 为已存在的domain_detections表添加新字段
|
||||
try:
|
||||
# 添加same_url字段
|
||||
cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS same_url BOOLEAN")
|
||||
# 添加is_chinese_title字段
|
||||
cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS is_chinese_title BOOLEAN")
|
||||
# 添加backlink_count_gt_10字段
|
||||
cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS backlink_count_gt_10 BOOLEAN")
|
||||
logger.info("为domain_detections表添加新字段成功")
|
||||
except Exception as e:
|
||||
logger.warning(f"为domain_detections表添加新字段失败: {e}")
|
||||
# 回滚当前事务
|
||||
conn.rollback()
|
||||
|
||||
# 添加字段注释
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.id IS '主键ID'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.domain_id IS '域名ID'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.baidu_history IS '百度历史'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.baidu_site IS '百度site'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.qihu360_site IS '360 site'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.google_site IS 'Google site'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.chinaz_info IS '站长之家信息'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.aizhan_info IS '爱站网信息'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.juziseo_info IS '桔子SEO信息'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.jucha_info IS '聚查信息'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.same_url IS '首网址和备案网址是否一样'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.is_chinese_title IS 'title是否简体中文'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.backlink_count_gt_10 IS '快照历史友链数量是否大于10'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.create_time IS '创建时间'")
|
||||
cur.execute("COMMENT ON COLUMN domain_detections.update_time IS '更新时间'")
|
||||
logger.info("添加domain_detections表字段注释成功")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
logger.info("数据库初始化完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"数据库初始化失败: {e}")
|
||||
if conn:
|
||||
conn.rollback()
|
||||
finally:
|
||||
if cur:
|
||||
cur.close()
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
init_database()
|
||||
1
domainCheck/logo.svg
Normal file
1
domainCheck/logo.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 12 KiB |
1
domainCheck/new_logo.svg
Normal file
1
domainCheck/new_logo.svg
Normal file
@@ -0,0 +1 @@
|
||||
<svg t="1776041645024" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2611" width="200" height="200"><path d="M462.9 858.3c-96 0-192.1-35.4-266.9-106.6C41.5 604.5 35.5 359 182.7 204.5 254 129.6 350.2 87 453.5 84.5c103.3-2.4 201.5 35.4 276.3 106.7 74.9 71.3 117.5 167.5 120 270.8s-35.4 201.5-106.7 276.3c-75.8 79.8-177.9 120-280.2 120z m0.3-730.7c-2.8 0-5.7 0-8.6 0.1-91.8 2.2-177.3 40.1-240.6 106.6C83.2 371.5 88.5 589.6 225.8 720.4S581.2 845.9 712 708.6c63.4-66.5 97-153.7 94.8-245.5-2.2-91.8-40.1-177.3-106.6-240.6-64.5-61.4-148.3-94.9-237-94.9z" fill="#1195FE" p-id="2612"></path><path d="M862.1 928.3c-17.7 0-35.5-6.5-49.3-19.7l-150.1-143 19.3-15.4c12.6-10 24.3-20.7 34.8-31.7 10.5-11 20.6-23.2 30-36.3l14.5-20.1 150.1 143c13.8 13.1 21.6 30.9 22.1 50s-6.5 37.2-19.6 51c-14.1 14.8-33 22.2-51.8 22.2zM727.8 768l114.8 109.4c11.3 10.8 29.2 10.3 40-1s10.3-29.2-1-40L766.8 727c-6.1 7.4-12.3 14.5-18.7 21.2-6.5 6.8-13.2 13.4-20.3 19.8zM268.8 551.8c-9.8-23.6-15.2-49-15.9-75.3-1.4-56.1 19.2-109.5 58-150.1 1.2-1.3 2.4-2.5 3.7-3.8 38.3-38.3 89.1-60.1 143.5-61.4 30.3-0.7 59.2 5 85.5 16l32.6-32.6c-35.8-17.9-76.3-27.6-119.1-26.6-67.7 1.6-130.7 29.3-177.3 78.6-46.7 49-71.5 113.3-69.9 180.9 0.9 37.8 10 74.1 26.4 106.9l32.5-32.6zM657 390.1c9.9 23.5 15.7 49.1 16.3 76.1 2.8 115.9-89.2 212.5-205.1 215.3-30.1 0.7-59.4-4.9-86.4-16.2l-32.6 32.6c36.7 18.5 77.7 27.8 120 26.8 67.6-1.6 128.5-29.8 172.9-74.2 47.3-47.3 76-113.2 74.3-185.3-0.9-38.7-10.6-75.1-26.9-107.6L657 390.1z" fill="#1195FE" p-id="2613"></path><path d="M287.283 615.504l188.401-62.199 6.05 18.327-188.4 62.199zM430.89 361.097l188.4-62.199 6.051 18.327-188.4 62.199z" fill="#1195FE" p-id="2614"></path><path d="M472.411 564.972l-27.36-196.516 19.117-2.661 27.36 196.515z" fill="#1195FE" p-id="2615"></path><path d="M623.8 314m-33.8 0a33.8 33.8 0 1 0 67.6 0 33.8 33.8 0 1 0-67.6 0Z" fill="#FFFFFF" p-id="2616"></path><path d="M623.8 360.8c-25.8 0-46.8-21-46.8-46.8 0-25.8 21-46.8 46.8-46.8 25.8 0 46.8 21 46.8 46.8 0 25.8-21 46.8-46.8 46.8z m0-67.5c-11.4 0-20.7 9.3-20.7 20.7 0 11.4 9.3 20.7 20.7 20.7s20.7-9.3 20.7-20.7c0-11.4-9.3-20.7-20.7-20.7z" fill="#1195FE" p-id="2617"></path><path d="M449.7 363.9m-33.8 0a33.8 33.8 0 1 0 67.6 0 33.8 33.8 0 1 0-67.6 0Z" fill="#FFFFFF" p-id="2618"></path><path d="M449.7 410.7c-25.8 0-46.8-21-46.8-46.8 0-25.8 21-46.8 46.8-46.8 25.8 0 46.8 21 46.8 46.8 0 25.8-21 46.8-46.8 46.8z m0-67.5c-11.4 0-20.7 9.3-20.7 20.7 0 11.4 9.3 20.7 20.7 20.7 11.4 0 20.7-9.3 20.7-20.7 0-11.5-9.3-20.7-20.7-20.7z" fill="#1195FE" p-id="2619"></path><path d="M302.4 628.7m-33.8 0a33.8 33.8 0 1 0 67.6 0 33.8 33.8 0 1 0-67.6 0Z" fill="#FFFFFF" p-id="2620"></path><path d="M302.4 675.5c-25.8 0-46.8-21-46.8-46.8 0-25.8 21-46.8 46.8-46.8 25.8 0 46.8 21 46.8 46.8 0 25.8-21 46.8-46.8 46.8z m0-67.5c-11.4 0-20.7 9.3-20.7 20.7 0 11.4 9.3 20.7 20.7 20.7 11.4 0 20.7-9.3 20.7-20.7 0-11.4-9.3-20.7-20.7-20.7z" fill="#1195FE" p-id="2621"></path><path d="M474.7 560.5m-33.8 0a33.8 33.8 0 1 0 67.6 0 33.8 33.8 0 1 0-67.6 0Z" fill="#FFFFFF" p-id="2622"></path><path d="M474.7 607.3c-25.8 0-46.8-21-46.8-46.8s21-46.8 46.8-46.8 46.8 21 46.8 46.8-21 46.8-46.8 46.8z m0-67.5c-11.4 0-20.7 9.3-20.7 20.7 0 11.4 9.3 20.7 20.7 20.7s20.7-9.3 20.7-20.7c0-11.4-9.3-20.7-20.7-20.7z" fill="#1195FE" p-id="2623"></path></svg>
|
||||
|
After Width: | Height: | Size: 3.2 KiB |
406
domainCheck/project_design.md
Normal file
406
domainCheck/project_design.md
Normal file
@@ -0,0 +1,406 @@
|
||||
# 域名库系统设计方案
|
||||
|
||||
## 1. 项目结构设计
|
||||
|
||||
```
|
||||
domainScanDemo/
|
||||
├── app/ # 应用主目录
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py # 主入口
|
||||
│ ├── ui/ # 可视化界面
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── main_window.py # 主窗口
|
||||
│ │ ├── domain_import.py # 域名导入界面
|
||||
│ │ ├── domain_filter.py # 域名筛选界面
|
||||
│ │ └── sensitive_words.py # 敏感词配置界面
|
||||
│ ├── core/ # 核心功能
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── domain_collector.py # 域名收集器
|
||||
│ │ ├── domain_processor.py # 域名处理器
|
||||
│ │ ├── detect_engine.py # 检测引擎
|
||||
│ │ ├── task_scheduler.py # 任务调度器
|
||||
│ │ └── export_manager.py # 导出管理器
|
||||
│ ├── detectors/ # 检测插件
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── base.py # 基础检测类
|
||||
│ │ ├── rdap_detector.py # RDAP检测
|
||||
│ │ ├── wayback_detector.py # 时光机检测
|
||||
│ │ ├── baidu_detector.py # 百度检测
|
||||
│ │ ├── qihu360_detector.py # 360检测
|
||||
│ │ ├── google_detector.py # Google检测
|
||||
│ │ ├── chinaz_detector.py # 站长之家检测
|
||||
│ │ ├── aizhan_detector.py # 爱站网检测
|
||||
│ │ ├── juziseo_detector.py # 桔子SEO检测
|
||||
│ │ └── jucha_detector.py # 聚查检测
|
||||
│ ├── utils/ # 工具类
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── domain_utils.py # 域名工具
|
||||
│ │ ├── http_utils.py # HTTP工具
|
||||
│ │ ├── database.py # 数据库操作
|
||||
│ │ └── config.py # 配置管理
|
||||
│ └── config/ # 配置文件
|
||||
│ ├── __init__.py
|
||||
│ ├── settings.py # 系统设置
|
||||
│ └── sensitive_words.txt # 敏感词列表
|
||||
├── scripts/ # 脚本工具
|
||||
│ ├── __init__.py
|
||||
│ ├── domain_import.py # 域名导入脚本
|
||||
│ ├── detect_runner.py # 检测运行脚本
|
||||
│ └── stats_report.py # 统计报告脚本
|
||||
├── tests/ # 测试代码
|
||||
│ ├── __init__.py
|
||||
│ ├── test_domain_utils.py # 域名工具测试
|
||||
│ ├── test_detectors.py # 检测器测试
|
||||
│ └── test_database.py # 数据库测试
|
||||
├── requirements.txt # 依赖包
|
||||
├── README.md # 项目说明
|
||||
└── .env # 环境变量
|
||||
```
|
||||
|
||||
## 2. 核心功能模块设计
|
||||
|
||||
### 2.1 域名收集模块
|
||||
|
||||
**功能**:从多个来源收集域名数据
|
||||
|
||||
**实现**:
|
||||
- **聚名网爬虫**:抓取一口价和过期删除域名
|
||||
- **搜索引擎采集**:通过关键词搜索收集域名
|
||||
- **企业目录抓取**:从企业黄页等网站抓取域名
|
||||
- **Zone File 解析**:解析域名区域文件
|
||||
- **TXT 导入**:支持从文本文件导入域名
|
||||
- **第三方 API**:集成域名数据 API
|
||||
|
||||
**技术要点**:
|
||||
- 使用 `Scrapy` 框架进行网站爬取
|
||||
- 支持代理池,避免 IP 被封禁
|
||||
- 多线程并行采集,提高效率
|
||||
- 域名标准化处理(小写、去空格、去协议、只留主域)
|
||||
|
||||
### 2.2 域名处理模块
|
||||
|
||||
**功能**:处理收集到的域名,进行标准化和去重
|
||||
|
||||
**实现**:
|
||||
- 域名标准化:转换为小写,去除空格和协议
|
||||
- 后缀过滤:只保留 `.com` 和 `.net`
|
||||
- 去重处理:使用布隆过滤器和数据库唯一索引
|
||||
- 来源标记:记录域名来源和批次信息
|
||||
- 任务创建:为新域名创建检测任务
|
||||
|
||||
**技术要点**:
|
||||
- 使用 Redis 布隆过滤器快速去重
|
||||
- 批量处理,减少数据库操作
|
||||
- 事务处理,确保数据一致性
|
||||
|
||||
### 2.3 检测引擎模块
|
||||
|
||||
**功能**:执行域名检测任务,包括注册状态、风险评估等
|
||||
|
||||
**实现**:
|
||||
- **基础检测**:
|
||||
- 注册状态检测(RDAP优先,其次WHOIS)
|
||||
- 黑名单缓存检查
|
||||
- 时光机快照年份采集
|
||||
- 敏感词匹配(快照内容)
|
||||
|
||||
- **深度检测**:
|
||||
- 百度历史/Site
|
||||
- 360 Site
|
||||
- Google Site
|
||||
- 站长之家/爱站
|
||||
- 桔子SEO(历史记录)
|
||||
- 聚查(WHOIS/备案/拦截)
|
||||
- 桔子SEO(外链查询)
|
||||
|
||||
**技术要点**:
|
||||
- 插件化设计,支持添加新的检测模块
|
||||
- 优先级队列,按成本和重要性排序
|
||||
- 失败重试机制,提高检测成功率
|
||||
- 并发处理,提高检测速度
|
||||
|
||||
### 2.4 任务调度模块
|
||||
|
||||
**功能**:管理检测任务队列,分配任务到检测节点
|
||||
|
||||
**实现**:
|
||||
- 任务队列管理:使用 RabbitMQ 存储任务
|
||||
- 任务分配:根据节点负载分配任务
|
||||
- 状态跟踪:实时监控任务执行状态
|
||||
- 断点续跑:支持系统重启后继续执行任务
|
||||
- 失败重试:自动重试失败的任务
|
||||
|
||||
**技术要点**:
|
||||
- 使用 Celery 作为任务队列
|
||||
- 分布式部署,支持多服务器
|
||||
- 实时监控和告警
|
||||
- 任务优先级管理
|
||||
|
||||
### 2.5 导出管理模块
|
||||
|
||||
**功能**:根据运营筛选条件导出域名
|
||||
|
||||
**实现**:
|
||||
- 多条件组合筛选
|
||||
- 预览功能
|
||||
- TXT 导出
|
||||
- 批量更新使用状态
|
||||
- 批量更新人工审核状态
|
||||
|
||||
**技术要点**:
|
||||
- 高效的数据库查询优化
|
||||
- 支持大文件导出
|
||||
- 导出进度实时显示
|
||||
- 导出历史记录
|
||||
|
||||
### 2.6 可视化界面模块
|
||||
|
||||
**功能**:提供用户友好的操作界面
|
||||
|
||||
**实现**:
|
||||
- **主窗口**:系统概览和功能导航
|
||||
- **域名导入**:支持多种方式导入域名
|
||||
- **域名筛选**:多条件组合筛选
|
||||
- **敏感词配置**:管理敏感词列表
|
||||
- **系统设置**:配置系统参数
|
||||
|
||||
**技术要点**:
|
||||
- 使用 PySide6 构建界面
|
||||
- 响应式设计,支持不同屏幕尺寸
|
||||
- 多线程处理,避免界面卡顿
|
||||
- 实时数据更新
|
||||
|
||||
## 3. 数据库设计
|
||||
|
||||
**数据库**:PostgreSQL 15+
|
||||
|
||||
**核心表结构**:
|
||||
|
||||
| 表名 | 功能 | 关键字段 |
|
||||
|------|------|----------|
|
||||
| `domains` | 域名主表 | domain, tld, use_status, detect_status, register_status, source_type |
|
||||
| `domain_detections` | 检测结果 | domain_id, baidu_history, baidu_site, is_simplified, qihu360_site, google_site |
|
||||
| `domain_sources` | 来源信息 | domain_id, source_type, source_url, batch_id |
|
||||
| `detect_tasks` | 检测任务 | domain_id, task_type, status, priority, retry_count |
|
||||
| `sensitive_words` | 敏感词 | word, category, level |
|
||||
| `domain_blacklist` | 黑名单 | domain, reason, blacklist_time, source |
|
||||
| `system_config` | 系统配置 | config_key, config_value, description |
|
||||
| `detect_nodes` | 检测节点 | node_name, ip_address, status, last_heartbeat |
|
||||
| `task_stats` | 任务统计 | date, total_tasks, success_tasks, failed_tasks |
|
||||
| `domain_stats` | 域名统计 | date, total_domains, available_domains, registered_domains |
|
||||
|
||||
**索引策略**:
|
||||
- 域名字段建立唯一索引
|
||||
- 状态字段建立普通索引
|
||||
- 时间字段建立 BRIN 索引
|
||||
- 复合索引优化多条件查询
|
||||
|
||||
**分区策略**:
|
||||
- `domains` 表按 TLD 分片
|
||||
- `domain_detections` 表按时间分区
|
||||
- `domain_sources` 表按时间分区
|
||||
- `detect_logs` 表按时间分区
|
||||
|
||||
## 4. 技术栈选择
|
||||
|
||||
| 分类 | 技术 | 版本 | 选型理由 |
|
||||
|------|------|------|----------|
|
||||
| **编程语言** | Python | 3.10+ | 丰富的库支持,适合数据处理和爬虫开发 |
|
||||
| **Web 框架** | Scrapy | 2.10+ | 强大的爬虫框架,支持分布式爬取 |
|
||||
| **数据库** | PostgreSQL | 15+ | 支持分区表、JSONB、布隆过滤器,适合亿级数据 |
|
||||
| **缓存** | Redis | 7+ | 高性能缓存,支持布隆过滤器和消息队列 |
|
||||
| **消息队列** | RabbitMQ | 3.10+ | 可靠的任务队列,支持优先级 |
|
||||
| **任务调度** | Celery | 5+ | 分布式任务处理框架 |
|
||||
| **界面库** | PySide6 | 6.5+ | 跨平台 GUI 框架,功能丰富 |
|
||||
| **HTTP 客户端** | Requests | 2.31+ | 简单易用的 HTTP 库 |
|
||||
| **异步处理** | asyncio | 内置 | 高效的异步 I/O 处理 |
|
||||
| **监控** | Prometheus + Grafana | 2.40+ | 强大的监控和可视化 |
|
||||
|
||||
## 5. 系统流程
|
||||
|
||||
### 5.1 域名入库流程
|
||||
|
||||
1. **数据采集**:从多个来源收集域名
|
||||
2. **标准化处理**:转换为小写,去除空格和协议
|
||||
3. **后缀过滤**:只保留 `.com` 和 `.net`
|
||||
4. **去重处理**:使用布隆过滤器和数据库唯一索引
|
||||
5. **来源标记**:记录域名来源和批次信息
|
||||
6. **任务创建**:为新域名创建检测任务
|
||||
|
||||
### 5.2 检测流程
|
||||
|
||||
1. **基础检测**:
|
||||
- 注册状态检测(RDAP优先)
|
||||
- 黑名单缓存检查
|
||||
- 时光机快照年份采集
|
||||
- 敏感词匹配(快照内容)
|
||||
- 命中则直接拉黑并停止后续检测
|
||||
|
||||
2. **深度检测**:
|
||||
- 百度历史/Site
|
||||
- 360 Site
|
||||
- Google Site
|
||||
- 站长之家/爱站
|
||||
- 桔子SEO(历史记录)
|
||||
- 聚查(WHOIS/备案/拦截)
|
||||
- 桔子SEO(外链查询)
|
||||
|
||||
### 5.3 运营筛选流程
|
||||
|
||||
1. **多条件组合筛选**:根据注册状态、使用状态、检测状态等条件
|
||||
2. **预览**:查看筛选结果
|
||||
3. **导出**:导出为 TXT 文件
|
||||
4. **批量更新**:更新使用状态和人工审核状态
|
||||
|
||||
## 6. 关键技术挑战及解决方案
|
||||
|
||||
### 6.1 数据量挑战
|
||||
|
||||
**挑战**:亿级域名数据的存储和查询
|
||||
|
||||
**解决方案**:
|
||||
- 使用 PostgreSQL 分区表
|
||||
- 优化索引策略
|
||||
- 数据分片存储
|
||||
- 缓存热点数据
|
||||
|
||||
### 6.2 检测效率挑战
|
||||
|
||||
**挑战**:大量域名的检测任务处理
|
||||
|
||||
**解决方案**:
|
||||
- 分布式检测架构
|
||||
- 任务优先级队列
|
||||
- 并发处理
|
||||
- 失败重试机制
|
||||
|
||||
### 6.3 反爬机制挑战
|
||||
|
||||
**挑战**:网站反爬措施
|
||||
|
||||
**解决方案**:
|
||||
- 使用代理池
|
||||
- 请求头随机化
|
||||
- 访问频率控制
|
||||
- 模拟浏览器行为
|
||||
|
||||
### 6.4 数据一致性挑战
|
||||
|
||||
**挑战**:分布式环境下的数据一致性
|
||||
|
||||
**解决方案**:
|
||||
- 事务处理
|
||||
- 消息队列确保任务可靠传递
|
||||
- 定期数据同步
|
||||
- 冲突检测和解决
|
||||
|
||||
## 7. 部署方案
|
||||
|
||||
### 7.1 服务器配置
|
||||
|
||||
| 服务器类型 | 配置 | 用途 |
|
||||
|------------|------|------|
|
||||
| **主服务器** | 16核32G,2TB SSD | 数据库、任务调度、Web 界面 |
|
||||
| **检测节点** | 8核16G,500GB SSD | 执行检测任务 |
|
||||
| **存储服务器** | 4核8G,4TB HDD | 数据备份、归档 |
|
||||
|
||||
### 7.2 部署步骤
|
||||
|
||||
1. **环境搭建**:
|
||||
- 安装 PostgreSQL、Redis、RabbitMQ
|
||||
- 配置 Python 环境
|
||||
- 安装依赖包
|
||||
|
||||
2. **数据库初始化**:
|
||||
- 执行 SQL 脚本创建表结构
|
||||
- 配置分区和索引
|
||||
- 导入初始数据
|
||||
|
||||
3. **服务启动**:
|
||||
- 启动数据库服务
|
||||
- 启动消息队列服务
|
||||
- 启动任务调度服务
|
||||
- 启动检测节点
|
||||
|
||||
4. **系统配置**:
|
||||
- 配置敏感词列表
|
||||
- 配置检测参数
|
||||
- 配置监控告警
|
||||
|
||||
### 7.3 监控与维护
|
||||
|
||||
1. **监控系统**:
|
||||
- 数据库性能监控
|
||||
- 任务执行状态监控
|
||||
- 系统负载监控
|
||||
- 网络状态监控
|
||||
|
||||
2. **维护计划**:
|
||||
- 每日:数据备份、任务统计
|
||||
- 每周:索引重建、系统更新
|
||||
- 每月:数据归档、性能优化
|
||||
- 每季度:全量备份、系统评估
|
||||
|
||||
## 8. 预期性能指标
|
||||
|
||||
| 指标 | 预期值 |
|
||||
|------|--------|
|
||||
| 数据导入速度 | > 100万条/分钟 |
|
||||
| 域名去重速度 | > 1000万条/秒 |
|
||||
| 检测任务处理 | > 10000个/分钟 |
|
||||
| 单表查询响应 | < 1秒 |
|
||||
| 系统可用性 | > 99.9% |
|
||||
| 检测准确率 | > 99% |
|
||||
|
||||
## 9. 核心竞争力
|
||||
|
||||
1. **全维度检测**:覆盖注册状态、历史快照、平台风险等多维度
|
||||
2. **智能筛选**:基于规则的自动化筛选,减少人工干预
|
||||
3. **增量建设**:不追求一次性完成,通过持续积累构建完整数据库
|
||||
4. **插件化架构**:检测模块可替换、可扩展
|
||||
5. **数据价值**:为域名投资和企业品牌保护提供数据支持
|
||||
6. **高性能**:分布式架构,支持亿级数据处理
|
||||
7. **可靠性**:完善的错误处理和重试机制
|
||||
8. **用户友好**:直观的可视化界面,操作简单
|
||||
|
||||
## 10. 项目实施计划
|
||||
|
||||
### 第一阶段:基础架构搭建(1个月)
|
||||
- 数据库设计和搭建
|
||||
- 核心模块框架搭建
|
||||
- 数据采集模块实现
|
||||
- 基础检测功能实现
|
||||
|
||||
### 第二阶段:功能完善(2个月)
|
||||
- 深度检测功能实现
|
||||
- 运营管理界面开发
|
||||
- 任务调度系统实现
|
||||
- 多服务器部署方案
|
||||
|
||||
### 第三阶段:优化与测试(1个月)
|
||||
- 性能优化
|
||||
- 稳定性测试
|
||||
- 功能测试
|
||||
- 部署上线
|
||||
|
||||
### 第四阶段:持续迭代
|
||||
- 根据运营反馈优化系统
|
||||
- 增加新的检测模块
|
||||
- 扩展数据源
|
||||
- 提升系统性能
|
||||
|
||||
## 11. 风险评估
|
||||
|
||||
| 风险 | 影响 | 应对措施 |
|
||||
|------|------|----------|
|
||||
| **数据量过大** | 存储和查询性能下降 | 分区表、索引优化、数据分片 |
|
||||
| **API 限制** | 检测速度受限 | 多API源、请求频率控制、缓存 |
|
||||
| **反爬机制** | 爬虫被封禁 | 代理池、请求头随机化、模拟浏览器 |
|
||||
| **系统稳定性** | 服务中断 | 监控告警、自动恢复、冗余部署 |
|
||||
| **成本控制** | 服务器和API成本高 | 优化检测流程、合理使用资源、预算规划 |
|
||||
|
||||
## 12. 结论
|
||||
|
||||
本设计方案基于需求文档,详细说明了系统的架构设计、功能模块和实施计划。系统采用模块化设计,支持增量建设和分布式部署,能够有效应对亿级数据量的挑战。通过多维度的检测和智能筛选,为用户提供高质量的域名资源。
|
||||
|
||||
该方案充分考虑了技术可行性和业务需求,为域名库系统的开发和部署提供了全面的指导。
|
||||
13
domainCheck/proxy_config.json
Normal file
13
domainCheck/proxy_config.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"proxy_url": "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
|
||||
"proxy_enable": true,
|
||||
"allow_direct": true,
|
||||
"proxy_urls": [
|
||||
"http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
|
||||
"http://211.101.244.154:18008/getProxy_batch.php?group=C&count=50",
|
||||
"http://211.101.244.154:18008/getProxy_batch.php?group=D&count=50",
|
||||
"http://211.101.244.154:18008/getProxy_batch.php?group=E&count=50",
|
||||
"http://211.101.244.154:18008/getProxy_batch.php?group=F&count=50",
|
||||
"http://211.101.244.154:18008/getProxy_batch.php?group=G&count=50"
|
||||
]
|
||||
}
|
||||
50
domainCheck/remove_beian_time_field.py
Normal file
50
domainCheck/remove_beian_time_field.py
Normal file
@@ -0,0 +1,50 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
'''
|
||||
@Project :domainScanDemo
|
||||
@File :remove_beian_time_field.py
|
||||
@IDE :PyCharm
|
||||
@Author :梦伴
|
||||
@Date :2026/4/11 22:40
|
||||
@explain : 移除domains表中的beian_time字段
|
||||
'''
|
||||
|
||||
import psycopg2
|
||||
from loguru import logger
|
||||
from app.config import config
|
||||
|
||||
def remove_beian_time_field():
|
||||
"""
|
||||
移除domains表中的beian_time字段
|
||||
"""
|
||||
try:
|
||||
# 连接数据库
|
||||
conn = psycopg2.connect(
|
||||
host=config.DB_HOST,
|
||||
port=config.DB_PORT,
|
||||
database=config.DB_DATABASE,
|
||||
user=config.DB_USER,
|
||||
password=config.DB_PASSWORD
|
||||
)
|
||||
cur = conn.cursor()
|
||||
logger.info("数据库连接成功")
|
||||
|
||||
# 移除beian_time字段
|
||||
try:
|
||||
cur.execute("ALTER TABLE domains DROP COLUMN IF EXISTS beian_time")
|
||||
logger.info("移除beian_time字段成功")
|
||||
except Exception as e:
|
||||
logger.error(f"移除beian_time字段失败: {e}")
|
||||
|
||||
# 提交事务
|
||||
conn.commit()
|
||||
logger.info("字段移除完成")
|
||||
|
||||
# 关闭连接
|
||||
cur.close()
|
||||
conn.close()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"执行失败: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
remove_beian_time_field()
|
||||
BIN
domainCheck/requirements.txt
Normal file
BIN
domainCheck/requirements.txt
Normal file
Binary file not shown.
10
domainCheck/start_app.ps1
Normal file
10
domainCheck/start_app.ps1
Normal file
@@ -0,0 +1,10 @@
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$env:PYTHONUTF8 = '1'
|
||||
|
||||
$python = Join-Path $PSScriptRoot '.venv\Scripts\python.exe'
|
||||
if (-not (Test-Path $python)) {
|
||||
throw "未找到虚拟环境解释器: $python"
|
||||
}
|
||||
|
||||
Set-Location $PSScriptRoot
|
||||
& $python 'app\main.py'
|
||||
10
domainCheck/start_worker.ps1
Normal file
10
domainCheck/start_worker.ps1
Normal file
@@ -0,0 +1,10 @@
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$env:PYTHONUTF8 = '1'
|
||||
|
||||
$python = Join-Path $PSScriptRoot '.venv\Scripts\python.exe'
|
||||
if (-not (Test-Path $python)) {
|
||||
throw "未找到虚拟环境解释器: $python"
|
||||
}
|
||||
|
||||
Set-Location $PSScriptRoot
|
||||
& $python 'detect_worker.py'
|
||||
3
domainCheck/thread_count.json
Normal file
3
domainCheck/thread_count.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"thread_count": "2"
|
||||
}
|
||||
202
domainCheck/需求文档内容_utf8.txt
Normal file
202
domainCheck/需求文档内容_utf8.txt
Normal file
@@ -0,0 +1,202 @@
|
||||
全网域名库需求文档
|
||||
(按原内容整理,未改动业务内容)
|
||||
需求:
|
||||
1.做一个全网域名库(只要com net)
|
||||
2.每天筛查符合条件的过期可以注册的域名出来
|
||||
3.可以对每天一口价域名 精选筛选
|
||||
4.数据库设计好,这个数据量非常大,全网目前已经注册的con net 应该最少3亿,不需要一次性收集完成,每天逐步增加
|
||||
总体流程:
|
||||
一。筛选可以注册域名
|
||||
1.收集域名入库
|
||||
2.先把入库域名做 注册状态检测,域名状态入库更新,
|
||||
后续每天第一时间把可以注册的域名,做检测的各种筛选;
|
||||
(RDAP/注册商/第三方API/whois 可以检测注册状态/或者可以找到更加好的方案)
|
||||
(优先 RDAP,然后 /注册商/第三方APIWHOIS 检测)
|
||||
3.时光机 做基础筛选,主要筛查出快照包含 敏感词的,然后拉黑状态
|
||||
4.各大登录平台做最后筛选检测(看最后人工筛选流程)
|
||||
5.筛选完成 后,把各种状态更新数据库,
|
||||
6.提供可筛选条件,给运营筛选域名导出来
|
||||
二。筛选一口价域名
|
||||
1.从聚名导入一口价域名,去重入库,
|
||||
2.去除黑名单的,然后去时光机做基础筛选
|
||||
3.各大登录平台做最后筛选检测(看最后人工筛选流程)
|
||||
机器流程:
|
||||
A. 入库流水线
|
||||
导入原始域名
|
||||
标准化(小写、去空格、去协议、只留主域)
|
||||
只保留 .com/.net
|
||||
去重
|
||||
写入来源批次
|
||||
创建待检测任务
|
||||
B. 基础检测流水线
|
||||
注册状态检测
|
||||
黑名单缓存检查
|
||||
时光机快照年份采集
|
||||
时光机正文抽样与敏感词匹配
|
||||
命中则直接拉黑并停止后续检测
|
||||
未命中进入深度检测
|
||||
C. 深度检测流水线
|
||||
按成本从低到高跑:
|
||||
百度历史/百度 site
|
||||
360 site
|
||||
Google site
|
||||
站长之家 / 爱站 / Ahrefs
|
||||
桔子(查历史,额度一周50w个域名) /
|
||||
聚查 WHOIS / 备案 / 拦截 /
|
||||
桔子(查外连,这个最后,额度一周只有8000个域名) /
|
||||
D. 导出流水线
|
||||
运营筛选条件组合查询
|
||||
→ 预览
|
||||
→ 导出 TXT
|
||||
→ 批量更新“使用状态”
|
||||
停止规则:
|
||||
1.域名检测状态 = 黑名单
|
||||
→ 后续不再进入任何检测队列
|
||||
2.域名使用状态 ∈ {已使用, 已卖出, 已预定}
|
||||
→ 不进入运营可选池
|
||||
3.域名注册状态 = 未到期不能注册 且 非一口价
|
||||
→ 不进入“可注册域名池”
|
||||
检测优先级
|
||||
1.后缀过滤(只保留 .com/.net)
|
||||
2.格式校验 / 去重
|
||||
3.黑名单缓存命中
|
||||
4.注册状态检测
|
||||
5.时光机基础筛查
|
||||
6.平台风险/标题/外链/收录检测
|
||||
7.备案检测
|
||||
8.导出候选池
|
||||
关键原则:
|
||||
系统采用“增量建设”模式,不要求一次性收集全量 .com/.net,而是通过多来源每日持续入库。
|
||||
所有检测采用流水线 + 队列模式,支持失败重试、断点续跑。
|
||||
黑名单域名一旦命中,立即停止后续检测,避免资源浪费。
|
||||
所有平台检测器必须插件化,后续可替换、可新增。
|
||||
拉黑结果必须更新原因
|
||||
敏感词、风险词必须可配置。
|
||||
数据库设计需支持亿级数据量,
|
||||
注册状态检测优先采用 RDAP,其次第三方接口,最后兼容 WHOIS。
|
||||
Archive 历史快照查询优先采用官方 CDX API,不直接爬页面。
|
||||
域名来源:
|
||||
1、 从聚名下载一口价域名(
|
||||
https://www.juming.com/
|
||||
chaofanai1998@gmail.com
|
||||
llzz123,./
|
||||
)
|
||||
2、 从聚名的过期删除域名列表下载已经过期的域名
|
||||
3. 全网收集全网 com net 域名,特别是 公司注册这些,可以去各大搜索引擎浏览器搜索关键词 收集域名入库
|
||||
4. 手动批量添加域名,给个窗口输入或者导入txt
|
||||
5. 各种企业录网站抓取 所有公司域名
|
||||
6. zone file 收
|
||||
7. 第三方 API / 数据包,问下gpt
|
||||
运营操作窗口:
|
||||
筛选条件:
|
||||
1.域名注册状态: 可以注册的
|
||||
2.是不是一口价域名,这个就不需要筛选条件1了
|
||||
3.域名使用状态: 未使用的
|
||||
4.域名检测状态: 正常的
|
||||
5.是否有备案历史: 是否
|
||||
6.备案年份: 2026 。。
|
||||
7.域名快照所有年份: 可选包含多年份 2026 2025 2024
|
||||
8.快照历史单个页面友情链接数量是否大于10: 是/否
|
||||
9.筛选完成给个导出按钮,都出txt 一行一个;
|
||||
10.给个批量更新使用状态:运营使用后,更新域名使用状态,避免重复检测
|
||||
11.给个批量更新人工审核状态:运营使用后,更新人工复核状态,避免重复检测
|
||||
12.敏感词 录入窗口
|
||||
筛选流程:
|
||||
1.先去时光机(archive.org)做基础筛查,因为时光机查域名历史快照是免费的,先把有黑历史的 一些不需要的条件 域名,拉入黑名单,后续不再检测
|
||||
2.然后再去各大平台筛选
|
||||
时光机筛选: 这个是基础筛选 官网archive.org
|
||||
时光机查询不要写死网页爬取,直接走 CDX API
|
||||
Internet Archive 官方有 Wayback CDX API,可以直接查快照时间轴和版本列表。
|
||||
时光机查询优先用 CDX API
|
||||
先拿快照年份
|
||||
再按年份抽样抓正文
|
||||
正文做敏感词匹配
|
||||
命中后直接黑名单
|
||||
1. 凡是 历史快照信息中存在敏感词的域名全部更新为 黑名单状态,凡是黑名单状态的,后续不做任何检测
|
||||
2. 检测快照年份,更新域名信息,需要把快照所属年份 录入域名库,后面需要作为筛选条件
|
||||
3.快照历史单个页面友情链接数量是否大于10: 是/否 时光机查询快照可以筛选
|
||||
域名表必须要要的字段,其它你可以按需求加
|
||||
域名
|
||||
域名使用状态: 1.未使用,2.已经使用,3.已经卖出,4.已经预定。(这个作用是,后续使用后更新数据库,避免重复检测,浪费资源)
|
||||
域名检测状态: 1:待检测 2 检测中 3 正常 4 黑名单 5 检测失败 6 暂停检测
|
||||
域名注册状态:
|
||||
1 待检测
|
||||
2 可注册
|
||||
3 已注册
|
||||
4 宽限期(autoRenewPeriod)
|
||||
5 赎回期(redemptionPeriod)
|
||||
6 删除期(pendingDelete)
|
||||
7 clientHold
|
||||
8 serverHold
|
||||
9 状态未知
|
||||
10 检测失败
|
||||
域名来源类型
|
||||
1 聚名一口价
|
||||
2 聚名过期删除
|
||||
3 zone file
|
||||
4 搜索引擎采集
|
||||
5 企业目录采集
|
||||
6 手工录入
|
||||
7 TXT 导入
|
||||
8 第三方接口
|
||||
9 其它
|
||||
黑名单原因
|
||||
快照敏感词
|
||||
百度历史过灰
|
||||
标题敏感词
|
||||
子域名异常
|
||||
风险提示
|
||||
备案条件不符
|
||||
拦截检测异常
|
||||
WHOIS/RDAP 状态异常
|
||||
外链锚文本敏感
|
||||
人工拉黑
|
||||
人工复核状态:
|
||||
0 无需复核
|
||||
1 待人工复核
|
||||
2 人工通过
|
||||
3 人工拒绝
|
||||
域名过期时间:
|
||||
是否有备案历史:1:待检测;2:有备案记录;3:没有备案记录
|
||||
备案年份:
|
||||
域名快照所有年份:2026,2025,2024
|
||||
备案单位性质是否企业: 是/否 聚查里面查
|
||||
首网址和备案网址是否一样: 是/否 聚查里面查
|
||||
检测时间
|
||||
百度历史收录状态: 是否有收录 是/否 桔子查询检查
|
||||
百度site收录状态: 是否有收录 是/否 百度site检测
|
||||
title 是否简体中文: 是/否 桔子查询检查有提示
|
||||
360 site收录状态: 是否有收录 是/否 360浏览器 site检测
|
||||
google site收录状态: 是否有收录 是/否 google浏览器 site检测
|
||||
快照历史单个页面友情链接数量是否大于10: 是/否 时光机查询快照可以筛选
|
||||
下面是目前人工筛选流程: 把筛选流程转化脚本筛选
|
||||
第一步:(域名只要.com .net这两种)只负责入库(使用postgresql)做成单独的可视化界面
|
||||
1、从聚名下载一口价合适价格的域名
|
||||
2、从聚名的过期删除域名列表下载已经过期的域名
|
||||
3. 全网收集全网 com net 域名,特别是 公司注册这些
|
||||
4. 手动批量添加域名
|
||||
|
||||
|
||||
从二开始做成单独的可视化界面(检测程序是可以多开到多个服务器上)
|
||||
第二步:检测域名是否注册
|
||||
|
||||
第四步:在站长之家查询,域名的标题是敏感词的都不要,查询中在网站分类是:视频电影、体育运动、常用查询等等都不要 --匹配铭感词 直接拉黑状态
|
||||
第五步:在爱站网查询,域名的标题是敏感词的都不要,查询中在百度网址检测是:低风险、疑似色情博彩风险 严重影响权重等都不要 --匹配铭感词 直接拉黑状态
|
||||
第六步:百度site查询,site查询中发现子域名的不要,域名的标题是敏感词的都不要,顶级域名或www或没收录的可以要 --匹配铭感词/有子域名(www,@,m 除外) 直接拉黑状态
|
||||
第七步:360的site查询,site查询中发现子域名的不要,域名的标题是敏感词的都不要,顶级域名或www或没收录的可以要 --匹配铭感词/有子域名(www,@,m 除外) 直接拉黑状态
|
||||
第八步:爱站网中的网站综合数据,快速批量查询,域名的标题是敏感词的都不要,查询中在百度网址检测是:低风险、疑似色情博彩风险 严重影响权重等都不要 --匹配铭感词/风险提示 直接拉黑状态
|
||||
第九步:百度网址安全中心查询,网址检测结果:{危险}的不要 --风险提示 直接拉黑状态
|
||||
第十步:聚查中的WHOIS 查询:域名状态查询出来是clientHold、serverHold不要 --这2个标识 直接拉黑状态
|
||||
第十一步:聚查中的备案相关查询:查询出来的{审核时间}只要2017-2023年的, ----把备案年份 录入域名里面的字段,给运营筛选
|
||||
{单位性质}最好是企业的, -- 更新域名里面的单位性质 是/否
|
||||
{网站首页网址}要跟{备案域名}一样,对应不上或者是多个网站首页网址的都不要 ---首网址和备案网址是否一样 是/否
|
||||
第十二步:聚查中的拦截检测相关查询:查询出来是正常的才要,只要有一项是拦截的都不要 -- 只要有一项拦截 直接拉黑状态
|
||||
第十三步:登录桔子seo网把下载好的域名复制到桔子查历史中批量查询
|
||||
第十四步:
|
||||
1、批量查询后在历史信息中存在敏感词的域名全部不要 --匹配铭感词 直接拉黑状态
|
||||
2、查询中有些域名是有百度历史收录的,在百度历史收录中做过灰的不要,百度历史收录中发现子域名也不要,
|
||||
有(盾集捡漏清单-汇集全网隐藏折扣,每天一折疯抢!、足球、直播、证券、配资、软件)的不要,在查询后只有显示是简体中文的要,显示是其他语言的都不要包括繁体中文
|
||||
3、有些域名在查询后没有敏感词,但在历史建站记录中是体育、商行、下载,影视、网络、计算、app、HTML SiteMap、模拟器、传媒、二次元、成人、米乐、小说、凯发、人才、华体、娱乐、开户等等这些词的都不要
|
||||
(铭感词,给个地方配置,每个词,分割);--匹配铭感词 直接拉黑状态
|
||||
4、域名在桔子seo网的外链查询緢文本中发现有子域名的不要,在外链查询中存在敏感词的域名全部不要,外链查询緢文本中发现有:内幕、猛料、精料、高手、绝杀、权威、澳门等等都不要 --匹配铭感词 直接拉黑状态
|
||||
使用pyside6做成可视化界面
|
||||
Reference in New Issue
Block a user