[英]How to install phantomjs and selenium with docker-compose on linux server?
[英]How do I load balance phantomjs using docker-compose and haproxy?
我有一個使用Selenium WebDriver與PhantomJS交互的應用程序。 為了擴大規模,我想運行多個PhantomJS實例並使用haproxy對其進行負載平衡。 這是針對本地應用程序的,因此我不關心部署到生產環境或類似環境。
這是我docker-compose.yml
文件:
version: '2'
services:
app:
build: .
volumes:
- .:/code
links:
- mongo
- haproxy
mongo:
image: mongo
phantomjs1:
image: wernight/phantomjs:latest
ports:
- 8910
entrypoint:
- phantomjs
- --webdriver=8910
- --ignore-ssl-errors=true
- --load-images=false
phantomjs2:
image: wernight/phantomjs:latest
ports:
- 8910
entrypoint:
- phantomjs
- --webdriver=8910
- --ignore-ssl-errors=true
- --load-images=false
phantomjs3:
image: wernight/phantomjs:latest
ports:
- 8910
entrypoint:
- phantomjs
- --webdriver=8910
- --ignore-ssl-errors=true
- --load-images=false
phantomjs4:
image: wernight/phantomjs:latest
ports:
- 8910
entrypoint:
- phantomjs
- --webdriver=8910
- --ignore-ssl-errors=true
- --load-images=false
haproxy:
image: haproxy
volumes:
- ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
ports:
- 8910:8910
links:
- phantomjs1
- phantomjs2
- phantomjs3
- phantomjs4
如您所見,我有四個phantomjs實例,一個haproxy實例和一個應用程序(用python編寫)。
這是我的haproxy.cfg
:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
frontend phantomjs_front
bind *:8910
stats uri /haproxy?stats
default_backend phantomjs_back
backend phantomjs_back
balance roundrobin
server phantomjs1 phantomjs1:8910 check
server phantomjs2 phantomjs2:8910 check
server phantomjs3 phantomjs3:8910 check
server phantomjs4 phantomjs4:8910 check
我知道我需要使用粘性會話或haproxy中的某些東西才能使它正常工作,但是我不知道該怎么做。
這是我連接到該服務的python應用程序代碼的相關代碼段:
def get_page(url):
driver = webdriver.Remote(
command_executor='http://haproxy:8910',
desired_capabilities=DesiredCapabilities.PHANTOMJS
)
driver.get(url)
source = driver.page_source
driver.close()
return source
我嘗試運行此代碼時遇到的錯誤是:
phantomjs2_1 | [ERROR - 2016-07-12T23:35:25.454Z] RouterReqHand - _handle.error - {"name":"Variable Resource Not Found","message":"{\"headers\":{\"Accept\":\"application/json\",\"Accept-Encoding\":\"identity\",\"Connection\":\"close\",\"Content-Length\":\"96\",\"Content-Type\":\"application/json;charset=UTF-8\",\"Host\":\"172.19.0.7:8910\",\"User-Agent\":\"Python-urllib/3.5\"},\"httpVersion\":\"1.1\",\"method\":\"POST\",\"post\":\"{\\\"url\\\": \\\"\\\\\\\"http://www.REDACTED.com\\\\\\\"\\\", \\\"sessionId\\\": \\\"4eff6a60-4889-11e6-b4ad-095b9e1284ce\\\"}\",\"url\":\"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url\",\"urlParsed\":{\"anchor\":\"\",\"query\":\"\",\"file\":\"url\",\"directory\":\"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/\",\"path\":\"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url\",\"relative\":\"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url\",\"port\":\"\",\"host\":\"\",\"password\":\"\",\"user\":\"\",\"userInfo\":\"\",\"authority\":\"\",\"protocol\":\"\",\"source\":\"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url\",\"queryKey\":{},\"chunks\":[\"session\",\"4eff6a60-4889-11e6-b4ad-095b9e1284ce\",\"url\"]}}","line":80,"sourceURL":"phantomjs://code/router_request_handler.js","stack":"_handle@phantomjs://code/router_request_handler.js:80:82"}
phantomjs2_1 |
phantomjs2_1 | phantomjs://platform/console++.js:263 in error
app_1 | Traceback (most recent call last):
app_1 | File "selenium_process.py", line 69, in <module>
app_1 | main()
app_1 | File "selenium_process.py", line 61, in main
app_1 | source = get_page(args.url)
app_1 | File "selenium_process.py", line 52, in get_page
app_1 | driver.get(url)
app_1 | File "/usr/local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 248, in get
app_1 | self.execute(Command.GET, {'url': url})
app_1 | File "/usr/local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
app_1 | self.error_handler.check_response(response)
app_1 | File "/usr/local/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 163, in check_response
app_1 | raise exception_class(value)
app_1 | selenium.common.exceptions.WebDriverException: Message: Variable Resource Not Found - {"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"96","Content-Type":"application/json;charset=UTF-8","Host":"172.19.0.7:8910","User-Agent":"Python-urllib/3.5"},"httpVersion":"1.1","method":"POST","post":"{\"url\": \"\\\"http://www.REDACTED.com\\\"\", \"sessionId\": \"4eff6a60-4889-11e6-b4ad-095b9e1284ce\"}","url":"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url","urlParsed":{"anchor":"","query":"","file":"url","directory":"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/","path":"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url","relative":"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/session/4eff6a60-4889-11e6-b4ad-095b9e1284ce/url","queryKey":{},"chunks":["session","4eff6a60-4889-11e6-b4ad-095b9e1284ce","url"]}}
app_1 |
那么,如何使負載平衡起作用? 我想念什么?
我發現在haproxy中需要某種會話管理。 硒webdriver和phantomjs通過會話進行通信。 客戶端發送一個POST /session
並在正文中收到一個帶有會話ID的回復。 該回復看起來像這樣:
{"sessionId":"5a27f2b0-48a5-11e6-97d7-7f5820fc7aa6","status":0,"value":{"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"linux-unknown-64bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}}
然后,隨着會話的進行,會話ID在后續請求中作為URI的一部分發送到服務器,例如GET /session/5a27f2b0-48a5-11e6-97d7-7f5820fc7aa6/source
。 我該如何獲取這些東西以用於haproxy的粘性會話?
您應該能夠在haproxy配置本身中添加cookie。
cookie SERVERID insert indirect nocache
server httpd1 10.0.0.19:9443 cookie httpd1 check
server httpd2 10.0.0.18:9443 cookie httpd2 check
然后,會話將通過haproxy自身進行堅持。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.