利用爬虫抓取新理念英语答案

在此之前,在他人的博客中看到一篇文章新理念外语网络教学平台——获取任意答案,但是写得不是很详细,下面我来详细的说明一下.

原理

首先进入自己学校的平台.打开开发者工具–network,然后登陆

可以看到headers和data.并且data中的tbName和tbPwd都没有加密.这是之后我们需要post的 包括上面两个_

答案从哪来

如果你此前做过测试并且获得过60分以上的成绩,就可以查看那个习题答案.

我们打开开发者工具– network.然后查看答案.

我们发现

答案是通过viewTestTask.apsx获取的.其中data中三个数字就代表返回给viewTestTask.aspx来获取测试的答案.

那么我们就需要 获取这三个数字.

如何获取三个关键数字

首先我们需要进入本周测试的网页.

然后打开开发者工具–network

抓取

通过viewTestTask.aspx传入header和data进行抓取.

由于先人的代码会造成

UnicodeEncodeError: 'latin-1' codec can't encode characters in position 32-37: ordinal not in range(256)错误,我进行了二次修正.仅供参考.适合版本[新理念英语v6.2]

需要修改的地方已经标注. python版本3.7

import random
import requests

root = "http://192.168.9.12/npels/" #网址自行修改
answerurl=root+"/Student/viewTestTask.aspx"  #路径自行修改
ctoken = None

s = requests.session()
ua = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0"}
s.headers.update(ua)
s.get(root)

def login(name,passwd,td):
    data = "__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE="+td+"&tbName="+name+"&tbPwd="+passwd+"&btnLogin=%E7%99%BB+%E5%BD%95"
    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "Accept-Language": "zh-CN,zh;q=0.9", #自行修改
        "Referer": "http://192.168.9.12/npels/",  #自行修改
        "Content-Type": "application/x-www-form-urlencoded",
    }
    t = s.post(root+"/login.aspx", data = data, headers = headers)
    print("登陆成功")
    ctoken = t.text[t.text.find("InitToken('") + 11:t.text.find("InitToken('") + 49]
    c = requests.cookies.RequestsCookieJar()
    c.set('ctoken', ctoken)
    s.cookies.update(c)
    s.get(root+"/studentdefault.aspx")

def randomnocache():
    return str(random.random())


def getanswer(part,ttid,sheetid,sttid):
    data = "action=getPart&partnum="+str(part)+"&ttid="+str(ttid)+"&sheetid="+str(sheetid)+"&sttid="+str(sttid)+"&nocache="+randomnocache()
    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "Accept-Language": "zh-CN,zh;q=0.9",
        "Referer": "http://192.168.9.12/npels/student/viewTestTask.aspx",#自行修改
        "Content-Type": "application/x-www-form-urlencoded",
    }
    ans=s.post(url=answerurl,data=data,headers=headers)
    if ans.text.find("服务器错误")==-1:
        return ans.text

def answer(ttid,sheetid,sttid):
    fo = open("/Users/yi/EnglishAnswer.html","a+")#自行修改
    fo.write(getanswer(1, ttid, sheetid, sttid))
    fo.write(getanswer(2, ttid, sheetid, sttid))
    fo.write(getanswer(3, ttid, sheetid, sttid))
    fo.write(getanswer(4, ttid, sheetid, sttid))
    fo.close()

login("1****","*****","/wE*******lkZDQMqMmW++++CnYG62OwBIGUt") #自行修改,分别为登录名,密码,__VIEWSTATE的参数
answer(3215,1**7,244***)#此处输入 ttid, sheetid, sttid的值

后续

答案是从viewTestTask.aspx传参获取的.并不需要登陆,于是就可以简化成如下代码.

import random
import requests

root = "http://192.168.9.12/npels/"
answerurl=root+"/Student/viewTestTask.aspx"
ctoken = None
s = requests.session()

def randomnocache():
    return str(random.random())

def getanswer(part,ttid,sheetid,sttid):
    data = "action=getPart&partnum="+str(part)+"&ttid="+str(ttid)+"&sheetid="+str(sheetid)+"&sttid="+str(sttid)+"&nocache="+randomnocache()
    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8",
        "Accept-Language": "zh-CN,zh;q=0.9",
        "Referer": "http://192.168.9.12/npels/student/viewTestTask.aspx",
        "Content-Type": "application/x-www-form-urlencoded",
    }
    ans=s.post(url=answerurl,data=data,headers=headers)
    if ans.text.find("服务器错误")==-1:
        return ans.text

def answer(ttid,sheetid,sttid):
    fo = open("/Users/yinys/EnglishAnswer1.html","w+")
    fo.write(getanswer(1, ttid, sheetid, sttid))
    fo.write(getanswer(2, ttid, sheetid, sttid))
   fo.write(getanswer(3, ttid, sheetid, sttid))
    fo.write(getanswer(4, ttid, sheetid, sttid))
    fo.close()

answer(3270,1776,246359)