使用請求導入數據時出錯 - python

Question

我正在嘗試使用“船舶移動”選項卡從該網頁訪問數據： https://qships.tmr.qld.gov.au/webx/# 。 我嘗試使用 url 服務請求，但不斷出現錯誤。

目前，我已經嘗試過：

import requests
import pandas as pd
import json

url = 'https://qships.tmr.qld.gov.au/webx/services/wxdata.svc/GetDataX'

payload = {'timezone': 'Australia/Sydney','is_first_request': '0','league_id': '4','sport_id': '2','period_id': '1'}

jsonData = requests.post(url, data = payload).json()

返回以下錯誤：

{'ExceptionDetail': {'HelpLink': None, 'InnerException': None, 'Message': "The incoming message has an unexpected message format 'Raw'. The expected message formats for the operation are 'Xml', 'Json'. This can be because a WebContentTypeMapper has not been configured on the binding. See the documentation of WebContentTypeMapper for more details.", 'StackTrace': '   at System.ServiceModel.Dispatcher.DemultiplexingDispatchMessageFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.UriTemplateDispatchFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.DeserializeInputs(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)', 'Type': 'System.InvalidOperationException'}, 'ExceptionType': 'System.InvalidOperationException', 'Message': "The incoming message has an unexpected message format 'Raw'. The expected message formats for the operation are 'Xml', 'Json'. This can be because a WebContentTypeMapper has not been configured on the binding. See the documentation of WebContentTypeMapper for more details.", 'StackTrace': '   at System.ServiceModel.Dispatcher.DemultiplexingDispatchMessageFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.UriTemplateDispatchFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.DeserializeInputs(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)'}

當我更改 json 字典時，我收到以下錯誤：

jsonData = requests.post(url, json={'key':'value'}).json()

{'ExceptionDetail': None, 'ExceptionType': None, 'Message': 'WebX.Security.EAuthError: Request does not belong to an authenticated session.', 'StackTrace': None}

可以通過請求訪問數據還是我必須抓取它？

Answer 1

按照鏈接的 web UI 發出的請求，此腳本似乎能夠訪問一些數據。

用於數據查詢的 JSON 有效負載與原始帖子不同，因此可能需要在此處進行一些調整。

無論如何，最重要的是您需要使用requests.Session() ，並首先向 web UI 發出請求以獲取 session cookie。

from pprint import pprint

import requests

get_session_url = "https://qships.tmr.qld.gov.au/webx/"
get_data_url = "https://qships.tmr.qld.gov.au/webx/services/wxdata.svc/GetDataX"
get_data_query = {
    "token": None,
    "reportCode": "MSQ-WEB-0001",
    "dataSource": None,
    "filterName": "Next 7 days",
    "parameters": [
        {
            "__type": "ParameterValueDTO:#WebX.Core.DTO",
            "sName": "DOMAIN_ID",
            "iValueType": 0,
            "aoValues": [{"Value": -1}],
        }
    ],
    "metaVersion": 0,
}

sess = requests.session()

sess.get(get_session_url).raise_for_status()

json_data = sess.post(
    get_data_url,
    json=get_data_query,
).json()

pprint(json_data)

打印出來（例如）

{'d': {'BuildVersion': '7.0.0.12590',
       'ReportCode': 'MSQ-WEB-0001',
       'Tables': [{'AsOfDate': '16:33 on Jan 17',
                   'BuildVersion': '7.0.0.12590',
                   'Data': [[132058,
                             334359,
                             'EXT',
                             'STOLT MOMIJI',
                             'TANKER',
                             121.52,
...