产品项目开发流程图,seo技术员招聘,网站建设能挣钱吗,数字企业管理系统序言
记录一个近期困扰了一些时间的问题。
我很喜欢在爬虫中遇到问题#xff0c;因为这意味着在这个看似简单的事情里还是有很多值得去探索的新东西。其实本身爬虫也是随着前后端技术的不断更新在进步的。 文章目录 序言Preliminary1 问题缘起1.1 Selenium长文本输入阻塞1.2…序言
记录一个近期困扰了一些时间的问题。
我很喜欢在爬虫中遇到问题因为这意味着在这个看似简单的事情里还是有很多值得去探索的新东西。其实本身爬虫也是随着前后端技术的不断更新在进步的。 文章目录 序言Preliminary1 问题缘起1.1 Selenium长文本输入阻塞1.2 ChromeDriver无法输入非BMP字符如Emoji字符 2 Javascript修改textarea文本内容不生效3 笨方法4 正解附录智谱清言ChatGLMSelenium测试脚本后记 Preliminary
本文假定你已经拥有基于ChromeDriver的Selenium脚本开发经验以及基础的前端知识。
1 问题缘起
1.1 Selenium长文本输入阻塞
长文本输入阻塞的问题是Selenium老大难问题其原因是Selenium的send_keys()函数在输入字符串时会将字符串分解为每个字符进行处理具体可见源码keys_to_typing函数for循环里最后一个else分支
def send_keys(self, *keys_to_send):Sends keys to current focused element.:Args:- keys_to_send: The keys to send. Modifier keys constants can be found in theKeys class.typing keys_to_typing(keys_to_send)if self._driver.w3c:for key in typing:self.key_down(key)self.key_up(key)else:self._actions.append(lambda: self._driver.execute(Command.SEND_KEYS_TO_ACTIVE_ELEMENT, {value: typing}))return selfdef keys_to_typing(value):Processes the values that will be typed in the element.typing []for val in value:if isinstance(val, Keys):typing.append(val)elif isinstance(val, int):val str(val)for i in range(len(val)):typing.append(val[i])else:for i in range(len(val)):typing.append(val[i])return typing
这显然是挺笨重的操作因为我们正常将长文本复制到textarea文本框中并不需要花费很长时间但是用driver.find_element(By.xxx, xxx).send_keys(...)逐字输入就很容易使得页面阻塞这里有个隐藏的问题是输入换行符\n有时会触发文本框内容的提交这个问题相对容易解决因为textarea是闭合标签输入的文本内容本质上是在textarea{文本内容}/textarea中因此将send_keys(...)中的换行符\n替换成br即可避免。
题外话源码为什么要这样写一定会有它的道理。仔细想想也不难理解keys_to_typing函数的参数value并不总是字符串也可能是键盘的某个键位比如常用的我们通过send_keys(Key.CONTROL, v)实现向输入框中粘贴文本因此需要分解成各个字符处理。 图1 智谱清言官网 本文以智谱清言ChatGLM官网的textarea标签的文本框输入为例你也可以在任何一个带有textarea文本框的网站进行测试直接向里面粘贴长文本是非常容易的但是通过浏览器驱动输入长文本则很容易造成阻塞。
下面的代码是基于Chrome浏览器驱动的测试脚本 需要配置用户数据路径initialize_chrome_driver函数中的chrome_user_data_path变量以跳过智谱清言的登录。 目前访问智谱清言官网还有一个问题就是会在页面加载上卡很久有一层Loading…的mask层使得页面元素完全无法交互应该是在验证登录此时页面标签和元素已经加载出来因此不能通过简单的element.is_display来确定页面是否可用。这个状态下可以用Selenium向文本框输入文字但是无法点击提交按钮可能可以通过selenium.webdriver.support.expected_condition.element_to_be_clickable函数判断元素是否可交互来确定页面是否完全加载。 这个问题不展开讨论一般来说很少彻底卡死报错信息是element not interactable。
# -*- coding: utf-8 -*-
# author: caoyang
# email: caoyang163.sufe.edu.cnimport os
import time
import loggingfrom selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.action_chains import ActionChainsdef initialize_chrome_driver(headless, timeout):chrome_user_data_path rC:\Users\caoyang\AppData\Local\Google\Chrome\User Datachrome_options webdriver.ChromeOptions() chrome_options.add_argument(fuser-data-dir{chrome_user_data_path}) # Import user dataif headless:chrome_options.add_argument(--headless)driver webdriver.Chrome(chrome_optionschrome_options)driver.set_page_load_timeout(timeout)if not headless:driver.maximize_window()return driverdef check_element_by_xpath(driver, xpath, timeout):WebDriverWait(driver, timeout).until(lambda _driver: _driver.find_element_by_xpath(xpath).is_displayed())# Regular-used XPaths
layout_xpaths {input-box : //textarea[class\scroll-display-none\], # XPath of the input box for humansend-button-1 : //img[class\enter_icon\], # XPath of the button to send text of input boxsend-button-2 : //div[class\enter\], # XPath of the button to send text of input boxchat-area : //div[id\session-container\], # XPath of the chat area which contains all the talks (consist of several chat boxes)human-box : //div[class\pr\], # XPath of human chat boxai-box : //div[class\answer-content-wrap\], # XPath of AI chat boxai-box-text : //div[class\markdown-body\], # XPath of the text contained in AI chat boxcreate-new-button : //div[class\new-session-button\], # XPath of create new talklike-or-dislike-area : //div[class\interact-operate\], # XPath of div tag contains like and dislike iconsdelete-session : //span[class\btn delete\], # XPath of button to delete old talk}# Initialize
driver initialize_chrome_driver(headlessFalse, timeout60)
driver.get(https://chatglm.cn/main/detail)check_element_by_xpath(driver, xpathlayout_xpaths[input-box], timeout60) # Check if input box is rendered
check_element_by_xpath(driver, xpathlayout_xpaths[send-button-1], timeout60) # Check if send button is rendered
check_element_by_xpath(driver, xpathlayout_xpaths[send-button-2], timeout60) # Check if send button is rendered# Delete old talk
try:driver.find_element_by_xpath(self.layout_xpaths[delete-session]).click()logging.info(Delete old talk ...)
except:logging.info(No old talk found ...)# Request
logging.info(Prompt ...)
prompt 这是一段很长的文本 # e.g. prompt x * 8000## Method 1: Use element.send_keys
driver.find_element_by_xpath(layout_xpaths[input-box]).send_keys(prompt) # Input the given prompt在上述代码中prompt字段过长容易导致send_keys(prompt)造成阻塞以至于影响后续的代码逻辑。 1.2 ChromeDriver无法输入非BMP字符如Emoji字符
ChromeDriver无法输入非BMP字符据说Geckodriver是可行的具体报错信息为
selenium.common.exceptions.WebDriverException: Message: unknown error: ChromeDriver only supports characters in the BMP这个问题就比上面的长文本阻塞要麻烦多了前者只是需要等待一段时间就能输入完但这个问题是真的绕不过去。
好在我们有万能的Javascript
# Request
logging.info(Prompt ...)
prompt # ## Method 1: Use element.send_keys
# driver.find_element_by_xpath(layout_xpaths[input-box]).send_keys(prompt) # Input the given prompt## Method 2: Use Javascript with one auguments (Fail)
js var txt arguments[0]; document.getElementsByTagName(textarea)[0].value txt;
driver.execute_script(js, prompt)
logging.info( - Use Javascript to input ...)注意这里Selenium不能直接用JQuery的语法使用$(textarea)来定位元素因为页面上只有一个textarea标签。 图2 使用Javascript输入Emoji大功告成我们成功地把输入到文本框中啦 问题结束了吗 2 Javascript修改textarea文本内容不生效
然而问题才刚刚开始在图2中只要你点击提交按钮就会发现页面提示发送内容不能为空甚至只要将鼠标移动到文本框中就会消失。
这个问题确实很让人苦恼笔者搜索很多关于Javascript修改textarea内容不生效的文章众说纷纭但是没有一个起作用的。事实上不止是修改textarea标签的value包括innerTextinnerHTMLtextContent都是不起作用的 图3 测试textarea的value, innerText, innerHTML属性的修改 控制台的输出信息会发现单纯修改value确实可以看到修改的文本出现在界面上但是实际HTML中并不会显示点击提交按钮$(.enter_icon).click()也无法发送内容。
如果修改innerText或innerHTMLHTML中确实出现了值修改的文本也出现在界面上但是仍然无法提交且点击提交按钮后界面上不再显示文本但HTML中依然可以看到Hello
某些回答指出需要进行dispatchEvent从最后的正解来看这种回答说对了一半如下面的代码所示通过加入elm.dispatchEvent(new Event(change));似乎很有道理但是实际上依然无法进行提交。
# Request
logging.info(Prompt ...)
prompt Hello!# ## Method 1: Use element.send_keys
# driver.find_element_by_xpath(layout_xpaths[input-box]).send_keys(prompt) # Input the given prompt# ## Method 2: Use Javascript with one auguments (Fail)
# js var txt arguments[0]; document.getElementsByTagName(textarea)[0].value txt;
# driver.execute_script(js, prompt)
# logging.info( - Use Javascript to input ...)# Method 3: Use Javascript with event dispatch (Fail)
js var elm arguments[0], txt arguments[1]; elm.value txt; elm.dispatchEvent(new Event(change));
element driver.find_element_by_xpath(self.layout_xpaths[input-box])
driver.execute_script(js, element, prompt)
logging.info( - Use Javascript to input ...)真是糟透了难道真的没有人遇到跟我一样的问题吗 3 笨方法
在控制台测了一遍又一遍后被这离奇消失的textarea文本折磨得实在是无能为力我决定还是用最笨得方法来解决
先把需要输入的文本复制到剪贴板上使用Pyperclip包使用pip install pyperclip进行安装使用send_keys方法在textarea文本框中进行粘贴文本
效果拔群事实证明能抓到猫的就是好老鼠笨方法一下子就解决了这个该死的问题
# Request
logging.info(Prompt ...)
prompt Hello!# ## Method 1: Use element.send_keys
# driver.find_element_by_xpath(layout_xpaths[input-box]).send_keys(prompt) # Input the given prompt# ## Method 2: Use Javascript with one auguments (Fail)
# js var txt arguments[0]; document.getElementsByTagName(textarea)[0].value txt;
# driver.execute_script(js, prompt)
# logging.info( - Use Javascript to input ...)# # Method 3: Use Javascript with event dispatch (Fail)
# js var elm arguments[0], txt arguments[1]; elm.value txt; elm.dispatchEvent(new Event(change));
# element driver.find_element_by_xpath(self.layout_xpaths[input-box])
# driver.execute_script(js, element, prompt)
# logging.info( - Use Javascript to input ...)# Method 4: Use keyboard operation (Success)
import pyperclip
pyperclip.copy(prompt)
time.sleep(1)
driver.find_element_by_xpath(self.layout_xpaths[input-box]).send_keys(Keys.CONTROL, v)
logging.info( - Use keyboard to input ...)4 正解
但是我们总归还是要解决这个问题的不能因为走了歪门邪道就自鸣得意。其实很容易能想到之所以无法提交Javascript修改的textarea文本框内容肯定是提交按钮没有绑定到修改的内容仍然是文本框原本的默认值这当然需要定义事件进行发送。这里我也搜索到一些人发现无法使用Selenium清空textarea文本框的内容但是他们最后还是向用send_keys(Keys.BACK_SPACE)的键盘操作方法进行妥协。
最后是在StackFlow上找到了这个问题的正解https://stackoverflow.com/questions/23892547/what-is-the-best-way-to-trigger-change-or-input-event-in-react-js For React 16 and React 15.6 Setter .value is not working as we wanted because React library overrides input value setter but we can call the function directly on the input as context. var nativeInputValueSetter Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, value).set;
nativeInputValueSetter.call(input, react 16 value);
var ev2 new Event(input, { bubbles: true});
input.dispatchEvent(ev2);For textarea element you should use prototype of HTMLTextAreaElement class. 这是涉及React框架的知识可是我再也不会回去学习前端了。
最后的正解代码应该是
# Request
logging.info(Prompt ...)
prompt Hello!# ## Method 1: Use element.send_keys
# driver.find_element_by_xpath(layout_xpaths[input-box]).send_keys(prompt) # Input the given prompt# ## Method 2: Use Javascript with one auguments (Fail)
# js var txt arguments[0]; document.getElementsByTagName(textarea)[0].value txt;
# driver.execute_script(js, prompt)
# logging.info( - Use Javascript to input ...)# # Method 3: Use Javascript with event dispatch (Fail)
# js var elm arguments[0], txt arguments[1]; elm.value txt; elm.dispatchEvent(new Event(change));
# element driver.find_element_by_xpath(self.layout_xpaths[input-box])
# driver.execute_script(js, element, prompt)
# logging.info( - Use Javascript to input ...)# # Method 4: Use keyboard operation (Success)
# import pyperclip
# pyperclip.copy(prompt)
# time.sleep(1)
# driver.find_element_by_xpath(self.layout_xpaths[input-box]).send_keys(Keys.CONTROL, v)
# logging.info( - Use keyboard to input ...)# Method 5: Use Javascript with DispatchEvent (Success)
js var txt arguments[0];
const textarea $(textarea);
var nativeTextAreaValueSetter Object.getOwnPropertyDescriptor(window.HTMLTextAreaElement.prototype, value).set;
nativeTextAreaValueSetter.call(textarea, txt);
const event new Event(input, {bubbles: true});
textarea.dispatchEvent(event);
driver.execute_script(js, prompt)
logging.info( - Use Javascript to input ...)附录智谱清言ChatGLMSelenium测试脚本
感谢您阅读到这里作为本文的结束附上完整的代码
# -*- coding: utf-8 -*-
# author: caoyang
# email: caoyang163.sufe.edu.cnimport os
import re
import time
import logging
import requestsfrom bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditionsclass BaseCrawler:tag_regex re.compile(r[^]|\n|\t)global_timeout 60global_interval 300chrome_user_data_path rC:\Users\caoyang\AppData\Local\Google\Chrome\User Data# Convert request headers copied from Firefox to dictionaryclassmethoddef headers_to_dict(cls, headers: str) - dict:lines headers.splitlines()headers_dict {}for line in lines:key, value line.strip().split(:, 1)headers_dict[key.strip()] value.strip()return headers_dict# Easy use of WebDriverWaitclassmethoddef check_element_by_xpath(cls, driver, xpath, timeout30):WebDriverWait(driver, timeout).until(lambda _driver: _driver.find_element_by_xpath(xpath).is_displayed())# param method: e.g. GET, POSTdef easy_requests(self, method, url, **kwargs):while True:try:response requests.request(method, url, **kwargs)breakexcept Exception as e:logging.warning(fError {method} {url}, exception information: {e})logging.warning(fWait for {self.global_interval} seconds ...)time.sleep(self.global_interval)return response# Initialize driverdef initialize_driver(self, browserchrome, headlessTrue, timeout60, **kwargs):browser browser.lower()assert browser in [chrome, firefox], fUnknown browser name: {browser}return eval(fself._initialize_{browser}_driver)(headless, timeout, **kwargs)# Initialize Google Chrome driverdef _initialize_chrome_driver(self, headless, timeout, **kwargs):chrome_options webdriver.ChromeOptions() chrome_options.add_argument(fuser-data-dir{self.chrome_user_data_path}) # Import user dataif headless:chrome_options.add_argument(--headless)driver webdriver.Chrome(chrome_optionschrome_options)driver.set_page_load_timeout(timeout)if not headless:driver.maximize_window()return driver# Initialize Mozilla Firefox driverdef _initialize_firefox_driver(self, headless, timeout, **kwargs):options webdriver.FirefoxOptions()if headless:options.add_argument(--headless)driver webdriver.Firefox(optionsoptions)driver.set_page_load_timeout(timeout)if not headless:driver.maximize_window()return driver# Get cookies by driverdef get_cookies(self, url, driverNone, browserchrome):quit_flag Falseif driver is None:# If there is no driver passedquit_flag Truedriver self.initialize_driver(browserbrowser, headlessTrue, timeout30)driver.get(url)cookies driver.get_cookies()def _cookie_to_string(_cookies):_string str()for _cookie in _cookies:_name _cookie[name]_value _cookie[value].replace( , %20) # %20 refers to space char in HTML_string f{_name}{_value};return _string.strip()if quit_flag:driver.quit()return _cookie_to_string(cookies)class ChatGLMCrawler(BaseCrawler):urls {home: https://chatglm.cn/main/detail, # Home URL}layout_xpaths {input-box : //textarea[class\scroll-display-none\], # XPath of the input box for human# input-box : //div[class\input-box-inner\], # XPath of the input box for human (div tag cannot be interacted)send-button-1 : //img[class\enter_icon\], # XPath of the button to send text of input boxsend-button-2 : //div[class\enter\], # XPath of the button to send text of input boxchat-area : //div[id\session-container\], # XPath of the chat area which contains all the talks (consist of several chat boxes)human-box : //div[class\pr\], # XPath of human chat boxai-box : //div[class\answer-content-wrap\], # XPath of AI chat boxai-box-text : //div[class\markdown-body\], # XPath of the text contained in AI chat boxcreate-new-button : //div[class\new-session-button\], # XPath of create new talklike-or-dislike-area : //div[class\interact-operate\], # XPath of div tag contains like and dislike iconsdelete-session : //span[class\btn delete\], # XPath of button to delete old talk}forbidden_strings []def __init__(self):super(ChatGLMCrawler, self).__init__()# param driver : WebDriver object# param prompt : The question you would like to ask AI# param model_name : One of the key in model_card_xpath, e.g. chatgpt3.5(16k)def request(self, driver, prompt, first_trialTrue):prompt prompt.replace(\n, \\n)if first_trial:driver.get(self.urls[home])self.check_element_by_xpath(driver, xpathself.layout_xpaths[input-box], timeout60) # Check if input box is renderedself.check_element_by_xpath(driver, xpathself.layout_xpaths[send-button-1], timeout60) # Check if send button is renderedself.check_element_by_xpath(driver, xpathself.layout_xpaths[send-button-2], timeout60) # Check if send button is rendered# Delete old talktry:driver.find_element_by_xpath(self.layout_xpaths[delete-session]).click()logging.info(Delete old talk ...)except:logging.info(No old talk found ...)# Requestlogging.info(Prompt ...)try:## Method 1: Use element.send_keysdriver.find_element_by_xpath(self.layout_xpaths[input-box]).send_keys(prompt) # Input the given promptlogging.info( - ok!)except:# ## Method 2: Use Javascript with one auguments (Fail)# js var txt arguments[0]; document.getElementsByTagName(textarea)[0].value txt; # driver.execute_script(js, prompt)# logging.info( - Use Javascript to input ...)# # Method 3: Use Javascript with event dispatch (Fail)# js var elm arguments[0], txt arguments[1]; elm.value txt; elm.dispatchEvent(new Event(change));# element driver.find_element_by_xpath(self.layout_xpaths[input-box])# driver.execute_script(js, element, prompt)# logging.info( - Use Javascript to input ...)# # Method 4: Use keyboard operation (Success)# import pyperclip# pyperclip.copy(prompt)# time.sleep(1)# driver.find_element_by_xpath(self.layout_xpaths[input-box]).send_keys(Keys.CONTROL, v)# logging.info( - Use keyboard to input ...)# Method 5: Use Javascript with DispatchEvent (Success)js var txt arguments[0];const textarea $(textarea);var nativeTextAreaValueSetter Object.getOwnPropertyDescriptor(window.HTMLTextAreaElement.prototype, value).set;nativeTextAreaValueSetter.call(textarea, txt);const event new Event(input, {bubbles: true});textarea.dispatchEvent(event);driver.execute_script(js, prompt)logging.info( - Use Javascript to input ...)while True:# The button is dynamic and sometimes fail to click ontry:driver.find_element_by_xpath(self.layout_xpaths[send-button-1]).click() # Click on the button to send the promptlogging.info(Use send button 1 ...)breakexcept:try:driver.find_element_by_xpath(self.layout_xpaths[send-button-2]).click() # Click on the button to send the promptlogging.info(Use send button 2 ...)breakexcept:logging.info(Use send button error ...)raise Exception(Use send button error ...)# Wait for responseself.check_element_by_xpath(driver, xpathself.layout_xpaths[chat-area], timeout30) # Check if chat area is renderedself.check_element_by_xpath(driver, xpathself.layout_xpaths[human-box], timeout30) # Check if human chat box is renderedself.check_element_by_xpath(driver, xpathself.layout_xpaths[ai-box], timeout30) # Check if AI chat box is renderedfinish_flag True # Indicating if AI generation is finishedwhile finish_flag:try:# If like or dislike appear, then stopdriver.find_element_by_xpath(self.layout_xpaths[like-or-dislike-area])finish_flag Falseexcept:ai_box_text driver.find_element_by_xpath(self.layout_xpaths[ai-box-text]) # Find AI response text element# ai_box_text driver.find_element_by_xpath(self.layout_xpaths[ai-box]) # Find AI response text elementai_box_text_inner_html ai_box_text.get_attribute(innerHTML) # Get inner HTML of the elementresponse self.tag_regex.sub(str(), ai_box_text_inner_html).strip(\n\t ).replace(\n, \\n) # Process response textforbidden_flags [forbidden_string in response for forbidden_string in self.forbidden_strings]if sum(forbidden_flags) 0:# It indicates that a forbidden string occursfinish_flag False# Extract AI response textai_box_text driver.find_element_by_xpath(self.layout_xpaths[ai-box-text]) # Find AI response text elementai_box_text_inner_html ai_box_text.get_attribute(innerHTML) # Get inner HTML of the elementresponse self.tag_regex.sub(str(), ai_box_text_inner_html).strip(\n\t ) # Process response textreturn response# param data_path: EXCEL file of job descriptions# param save_path: file path for storing AI response# param model_name: defined in model_card_xpathsdef demo(self, model_namechatgpt3.5(16k)):driver self.initialize_driver(browserchrome, headlessFalse, timeout60)driver.implicitly_wait(15) # prompt 给我讲述一下《基督山伯爵》的故事500字左右。response self.request(driver, prompt)with open(fd:/answer-chatglm.txt, w, encodingutf8) as f:f.write(response)time.sleep(5)driver.quit()if __name__ __main__:crawler ChatGLMCrawler()crawler.demo()图4 脚本测试截图 后记
昨天S消防演习人在18楼走到10楼就堵得下不去了宝玺姐根本就没有下楼一直呆在工位上我回来后说如果S真的失火应该会有直升机来营救我们应该往上面天台跑而不是往楼下逃生。宝玺姐非常认真地跟我说S如果失火我们肯定都是死路一条有一种高知女性特有的决绝感。我不知道SXY是否也是如此其实我现在离TA很近可是又离TA很远。
在这次高校百英里接力赛摸到10km路跑40分钟的大门之后我坚定了一个信念人可以办成任何事情人也不要办成每一件事情。十年前甚至五年、三年前我都不敢想象有一天能把10km跑进40分钟但现在确确实实做到了即便是在这样奔波的生活中。未来我也一定有机会全马破三、完成铁三越野。
现在我觉得人生是一个等待和追求的过程迷茫的时候可以静静地等待明确的时候应当无畏地追求一切安顿时就该好好休息。我看到不同的人、不同的事、不同的生活、不同的态度我觉得平等的交流是最难能可贵的事让每个认知水平不同的人即便是在一个相对小的群体里都能够发声也敢于发声让每一种声音都有它的容身之处。
我们总是有一种莫名的主观的客观视角总会觉得有一些自己认为理所当然的事情别人都应当如此然而并不其然。当我看到35岁的吉米和乔总吃饭时还在讨论《咒术回战》的五条悟、《进击的巨人》的地鸣听到儿子都已经10多岁的慧悦姐称自己在工作中很闷骚时觉得很不可思议。跟宝玺姐一起工作的三个月里也看到她身上很多看似矛盾、但细想又很合理的事情。人类就是这样多态的实例化对象也注定是多变的非常量实体。
每次写到这里都会想起很多遗憾。岁数越长越觉得人生是很奇特的历程你永远不知道盒子里下一块巧克力是什么颜色你甚至无法知晓盒子里到底是不是一块巧克力或许是根棒棒糖也说不定就像曾经错过的人和事在将来的某个节点是否会再次相遇
罢了。
