无法从Ruby on Rails应用程序中的外部URL提取完整的标头

Question

Background:
The rails app I'm working on opens up articles from other websites in an iframe. But some publisher websites (like pitchfork.com, vox.com, medium.com) prevent themselves from opening in iframes by setting "X-Frame-Options: SAMEORIGIN" in their header. So given the URL for an article, I'm trying to examine the header and either open it in the iframe (default) or to open the original site in a new tab (when I detect X-Frame-Options in the header).

---

Problem:
The header I pull in Rails is sometimes incomplete when I pull it (and print to the console) with the following code:

puts y['site'] # example: "vox.com"
puts y['head'] # example: "/2016/1/25/10829662/obama-on-clinton-media"
require 'net/http'
http = Net::HTTP.start(y['site'])
resp = http.head(y['head'])
resp.each { |k, v| puts "#{k}: #{v}" }
http.finish

Example: the header that rails pulls for the this vox.com article ( http://www.vox.com/2016/1/25/10829662/obama-on-clinton-media ) is as follows:

server: nginx/1.6.2
date: Fri, 29 Jan 2016 22:05:17 GMT
content-type: text/html
content-length: 184
connection: keep-alive
location: http://www.vox.com/2016/1/25/10829662/obama-on-clinton-media

But when I try to open it an iframe, the chrome console tells me that it cannot because X-Frame-Options is set to SAMEORIGIN. Upon further investigation in the Network tab, I am able to examine to examine the full header and it is as follows:

HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; charset=utf-8
Status: 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: max-age=0, must-revalidate
X-Request-Id: 693f75c9be4dde491ba3cd78232ac4870c4f82e2
X-Runtime: 0.404545
Content-Encoding: gzip
Via: 1.1 varnish-v4
Content-Length: 26450
Accept-Ranges: bytes
Date: Fri, 29 Jan 2016 22:10:47 GMT
Via: 1.1 varnish
Age: 106
Connection: keep-alive
X-Served-By: cache-jfk1034-JFK
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1454105446.991771,VS0,VE12
Vary: Accept-Encoding, Origin, X-Forwarded-Proto

This problem doesn't occur for all sites. For example, the header I pull from pitchfork.com clearly indicates that it has x-frame-options set. But with sites like vox.com and medium.com the header that I pull does not show x-frame-options (as well as many other items that are being left out).

How can I pull the correct/complete header in my Rails controller in a way that will always detect whether a URL has X-Frame-Options in its header?

Answer 1

I tried here in IRB console, and I noticed that the request to vox.com website is returning 301 Moved Permanently, and it sent the new location in the header.

irb(main):001:0> y = {}
=> {}
irb(main):002:0> y['site'] = "vox.com"
=> "vox.com"
irb(main):003:0> y['head'] = "/2016/1/25/10829662/obama-on-clinton-media"
=> "/2016/1/25/10829662/obama-on-clinton-media"
irb(main):004:0> require 'net/http'
=> true
irb(main):005:0> http = Net::HTTP.start(y['site'])
=> #<Net::HTTP vox.com:80 open=true>
irb(main):006:0> resp = http.head(y['head'])
=> #<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true> (HERE)
irb(main):007:0> resp.each { |k, v| puts "#{k}: #{v}" }
server: nginx/1.6.2
date: Fri, 29 Jan 2016 22:40:07 GMT
content-type: text/html
content-length: 184
connection: keep-alive
location: http://www.vox.com/2016/1/25/10829662/obama-on-clinton-media
=> {"server"=>["nginx/1.6.2"], "date"=>["Fri, 29 Jan 2016 22:40:07 GMT"], "content-type"=>["text/html"], "content-length"=>["184"], "connection"=>["keep-alive"], "location"=>["http://www.vox.com/2016/1/25/10829662/obama-on-clinton-media"]}
irb(main):008:0> http.finish
=> nil

The only difference between the URL you used and the location that the server sent for redirect is the 'www'. Try to use with the 'www' and see if it works.

You can improve your code to read the response code, and if it's 301, try again with the URL the server sent.