网络爬虫基础与Requests库应用
爬虫基础原理
网络爬虫本质是通过HTTP协议模拟客户端请求：
# 核心流程
发送HTTP请求 → 服务器响应 → 数据解析 → 存储结果
# 常用工具
- 请求库：Requests, Selenium
- 解析库：BeautifulSoup, lxml
- 框架：Scrapy合法性问题：网站根目录下的robots.txt定义了爬取规则。
GET请求处理
import requests
response = requests.get('https://www.example.com/article')
print(response.text) # 获取HTML内容请求参数处理
# URL参数拼接
params = {'keyword': '数据分析', 'page': 2}
result = requests.get('https://api.example.com/search', params=params)
# URL编解码
from urllib.parse import quote, unquote
encoded = quote('中文参数')
decoded = unquote('%E4%B8%AD%E6%96%87')请求头设置
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64 x64)',
'Referer': 'https://origin-site.com/'
}
response = requests.get('https://target-site.com', headers=headers)Cookie应用
# 方式1：header携带
auth_headers = {'Cookie': 'session_id=abc123xyz'}
requests.post('https://auth-site.com/vote', headers=auth_headers)
# 方式2：专用参数
requests.get('https://member-site.com', cookies={'token': 'qwerty987'})POST请求示例
login_data = {
'email': 'user@example.com',
'password': 'securePass123',
'remember_me': True
}
session = requests.Session()
session.post('https://login-portal.com/auth', data=login_data)
profile = session.get('https://login-portal.com/profile')响应处理方法
resp = requests.get('https://data-service.com/api')
print(resp.status_code) # 状态码
print(resp.headers) # 响应头
print(resp.json()) # JSON解析
print(resp.cookies) # 获取Cookie二进制数据处理
# 图片下载
img_res = requests.get('https://cdn.com/image.jpg')
with open('local_image.jpg', 'wb') as f:
f.write(img_res.content)
# 视频流处理
video_res = requests.get('https://stream.com/video.mp4', stream=True)
with open('video.mp4', 'wb') as v:
for chunk in video_res.iter_content(chunk_size=1024):
v.write(chunk)代理配置
proxy_config = {
'http': 'http://203.0.113.1:8080',
'https': 'https://203.0.113.2:8443'
}
requests.get('https://check-ip.com', proxies=proxy_config, timeout=5)异常处理机制
from requests.exceptions import Timeout, ConnectionError
try:
requests.get('https://unstable-site.com', timeout=3)
except Timeout:
print("请求超时")
except ConnectionError:
print("网络连接异常")代理池实现
import requests
def get_proxy():
proxy_data = requests.get('http://proxy-pool:5010/get/').json()
scheme = 'https' if proxy_data['https'] else 'http'
return {scheme: f"{scheme}://{proxy_data['proxy']}"}
proxy = get_proxy()
response = requests.get('https://ip-check.com', proxies=proxy)BeautifulSoup解析
from bs4 import BeautifulSoup
html_content = "<html><body><p class='content'>示例文本</p></body></html>"
soup = BeautifulSoup(html_content, 'lxml')
# 元素定位
paragraph = soup.find('p', class_='content')
print(paragraph.text) # 输出: 示例文本文档树遍历
# 获取父元素
parent = paragraph.parent
# 获取兄弟节点
next_sib = paragraph.next_sibling
# 获取所有子元素
children = list(paragraph.children)文档搜索技巧
# 属性搜索
links = soup.find_all(href=re.compile("example.com"))
# 多条件查询
items = soup.find_all(attrs={"class": "item", "data-id": True})