Python爬虫爬取51Job所有职位数据

1、创建项目
scrapy startproject job51_link
2、自动生成Spider
scrapy genspider job51Hlink 51job.com
3、运行爬虫
scrapy crawl job51Hlink
4、使用scrapy shell
scrapy shell crapy shell -s USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" "https://jobs.51job.com/ningbo-yzq/108714907.html?s=01&t=0"

'beijing','tianjin','shanghai','chongqing','hebeisheng',
'shanxisheng','liaoningsheng','jilinsheng','heilongjiangsheng',
'jiangsusheng','zhejiangsheng','anhuisheng','fujiansheng',
'jiangxisheng','shandongsheng','henansheng','hubeisheng',
'hunansheng','guangdongsheng','hainansheng','sichuansheng',
'guizhousheng','yunnansheng','shaanxisheng','gansusheng',
'qinghaisheng','innermongolia','guangxi','tibet','ningxia',
'xinjiang'