[英]Ruby Net::HTTP::Post submit form
I'm working on creating a website scraper. 我正在创建网站抓取工具。 There is a form used to change the current page.
有一种用于更改当前页面的表格。
This is the way that I am submitting the form for the POST request, but it seems to be fetching the same page over and over again. 这是我为POST请求提交表单的方式,但似乎要一次又一次地获取同一页面。
Here is some sample code: 这是一些示例代码:
pages = {
"total_pages" => 19,
"p1" => '1234/1456/78990/123324345/12143343214345/231432143/12432412/435435/',
"p2" => '1432424/123421421/345/435/6/65/5/34/3/2/21/1243',
..
..
..
}
idx = 1
p_count = pages["total_pages"]
#set up the HTTP request to change pages to get all the auction results
uri = URI.parse("http://somerandomwebsite.com?listings")
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Post.new(uri.request_uri)
p_count.times do
puts "On loop sequence: #{idx}"
pg_num = "p#{idx}"
pg_content = pages["#{pg_num}"]
req.set_form_data({"page" => "#{pg_num}", "#{pg_num}" => "#{pg_content}"})
response = http.request(req)
page = Nokogiri::HTML(response.body)
idx = idx + 1
end
It looks like page
never changes. 看起来
page
永远不会改变。 Is there a way to see what the full request looks like each time I am looking to make sure that the proper params are getting passed? 每当我想要确保通过正确的参数时,是否可以查看完整请求的外观? It seems like it's virtually impossible to determine anything about
req
. 似乎几乎不可能确定有关
req
任何信息。
A great way to debug HTTP is to take advantage of http://httpbin.org : 调试HTTP的一种好方法是利用http://httpbin.org :
require 'net/http'
uri = URI('http://httpbin.org/post')
res = Net::HTTP.post_form(uri, 'q' => 'ruby', 'max' => '50')
puts res.body
Which returns: 哪个返回:
# >> {
# >> "args": {},
# >> "data": "",
# >> "files": {},
# >> "form": {
# >> "max": "50",
# >> "q": "ruby"
# >> },
# >> "headers": {
# >> "Accept": "*/*",
# >> "Accept-Encoding": "gzip;q=1.0,deflate;q=0.6,identity;q=0.3",
# >> "Content-Length": "13",
# >> "Content-Type": "application/x-www-form-urlencoded",
# >> "Host": "httpbin.org",
# >> "User-Agent": "Ruby"
# >> },
# >> "json": null,
# >> "origin": "216.69.191.1",
# >> "url": "http://httpbin.org/post"
# >> }
That said, I'd recommend not using Net::HTTP. 就是说,我建议不要使用Net :: HTTP。 There are plenty of great HTTP clients for Ruby that will make it easier to write your code.
Ruby有许多出色的HTTP客户端,它们可以使编写代码变得更加容易。 For instance here's the same thing using HTTPClient :
例如,这是使用HTTPClient的相同内容 :
require 'httpclient'
clnt = HTTPClient.new
res = clnt.post('http://httpbin.org/post', 'q' => 'ruby', 'max' => '50')
puts res.body
# >> {
# >> "args": {},
# >> "data": "",
# >> "files": {},
# >> "form": {
# >> "max": "50",
# >> "q": "ruby"
# >> },
# >> "headers": {
# >> "Accept": "*/*",
# >> "Content-Length": "13",
# >> "Content-Type": "application/x-www-form-urlencoded",
# >> "Date": "Thu, 09 Feb 2017 20:03:57 GMT",
# >> "Host": "httpbin.org",
# >> "User-Agent": "HTTPClient/1.0 (2.8.3, ruby 2.4.0 (2016-12-24))"
# >> },
# >> "json": null,
# >> "origin": "216.69.191.1",
# >> "url": "http://httpbin.org/post"
# >> }
This is untested code because you didn't tell us nearly enough, but it's where I'd start doing what you're doing: 这是未经测试的代码,因为您几乎没有告诉我们,但这是我开始做您正在做的事情的地方:
require 'httpclient'
BASE_URL = 'http://somerandomwebsite.com?listings'
PAGES = [
'1234/1456/78990/123324345/12143343214345/231432143/12432412/435435/',
'1432424/123421421/345/435/6/65/5/34/3/2/21/1243',
]
clnt = HTTPClient.new
PAGES.each.with_index(1) do |page, idx|
puts "On loop sequence: #{idx}"
response = clnt.post(BASE_URL, 'page' => idx, idx => page)
doc = Nokogiri::HTML(response.body)
# ...
end
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.