堆棧內存溢出
  簡體   English   中英

流插入中python客戶端的非確定性錯誤

[英]Non-deterministic errors with python client on streaming inserts

 原文  2017-07-04 20:35:33  1   1    python/ google-bigquery

問題描述

我有一個具有以下架構的BigQuery表： 

    [
    {
        "mode": "REQUIRED",
        "name": "LOTRANS",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "FIRMWARE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "MBATTCHG",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "BATTV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "MINTIMEL",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "STARTTIME",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "NOMBATTV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "DATE",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "MODEL",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "HITRANS",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "TIMELEFT",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "BCHARGE",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "XONBATT",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "TONBATT",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "LINEV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "XOFFBATT",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "ALARMDEL",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "STATUS",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "CUMONBATT",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "CABLE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "UPSMODE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "LASTXFER",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "DRIVER",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "VERSION",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "LOADPCT",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "NOMINV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "ENDAPC",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "SERIALNO",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "HOSTNAME",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "UPSNAME",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "NUMXFERS",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "SENSE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "APC",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "MAXTIME",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "BATTDATE",
        "type": "DATE"
    },
    {
        "mode": "REQUIRED",
        "name": "STATFLAG",
        "type": "INTEGER"
    }
]

數據行的示例如下： 

{
    "ALARMDEL": "No alarm",
    "APC": "001,035,0928",
    "BATTDATE": "2016-12-02",
    "BATTV": "13.7",
    "BCHARGE": "100.0",
    "CABLE": "USB Cable",
    "CUMONBATT": "2758",
    "DATE": "2017-07-04T20:20:26",
    "DRIVER": "USB UPS Driver",
    "ENDAPC": "2017-07-04T20:20:35",
    "FIRMWARE": "904.W1 .D USB FW:W1",
    "HITRANS": "139.0",
    "HOSTNAME": "raspberrypi-master",
    "LASTXFER": "Unacceptable line voltage changes",
    "LINEV": "120.0",
    "LOADPCT": "0.0",
    "LOTRANS": "92.0",
    "MAXTIME": "0",
    "MBATTCHG": "5",
    "MINTIMEL": "3",
    "MODEL": "Back-UPS ES 550G",
    "NOMBATTV": "12.0",
    "NOMINV": "120",
    "NUMXFERS": "188",
    "SENSE": "Medium",
    "SERIALNO": "REDACTED",
    "STARTTIME": "2017-05-23T13:27:07",
    "STATFLAG": "0x05000008",
    "STATUS": "ONLINE",
    "TIMELEFT": "335.8",
    "TONBATT": "0",
    "UPSMODE": "Stand Alone",
    "UPSNAME": "raspberrypi-master",
    "VERSION": "3.14.12 (29 March 2014) debian",
    "XOFFBATT": "2017-07-04T03:55:55",
    "XONBATT": "2017-07-04T03:55:53"
}

我嘗試按照示例代碼進行操作 ，但是每當嘗試使用python客戶端使用上述字典對insert_data進行操作時，都會出現諸如以下的不確定性錯誤： 

[{'index': 0, 'errors': [{u'debugInfo': u'generic::invalid_argument: Cannot convert value to integer (bad value):MBATTCHG', u'reason': u'invalid', u'message': u'Cannot convert value to integer (bad value):MBATTCHG', u'location': u'mbattchg'}]}]

但是錯誤會在每次嘗試插入時發生變化。 有什么方法可以幫助調試為什么錯誤不是確定性的，為什么轉換后的值往往是字典的鍵而不是值？

編輯

將事件修改為以下內容，但仍然出現錯誤 

{
    "ALARMDEL": "No alarm",
    "APC": "001,035,0928",
    "BATTDATE": "2016-12-02",
    "BATTV": 13.7,
    "BCHARGE": 100.0,
    "CABLE": "USB Cable",
    "CUMONBATT": 2758,
    "DATE": "2017-07-05T02:41:00",
    "DRIVER": "USB UPS Driver",
    "ENDAPC": "2017-07-05T02:41:07",
    "FIRMWARE": "904.W1 .D USB FW:W1",
    "HITRANS": 139.0,
    "HOSTNAME": "raspberrypi-master",
    "LASTXFER": "Unacceptable line voltage changes",
    "LINEV": 123.0,
    "LOADPCT": 0.0,
    "LOTRANS": 92.0,
    "MAXTIME": 0,
    "MBATTCHG": 5,
    "MINTIMEL": 3.0,
    "MODEL": "Back-UPS ES 550G",
    "NOMBATTV": 12.0,
    "NOMINV": 120.0,
    "NUMXFERS": "188",
    "SENSE": "Medium",
    "SERIALNO": "REDACTED",
    "STARTTIME": "2017-05-23T13:27:07",
    "STATFLAG": 83886088,
    "STATUS": "ONLINE",
    "TIMELEFT": 335.8,
    "TONBATT": 0,
    "UPSMODE": "Stand Alone",
    "UPSNAME": "raspberrypi-master",
    "VERSION": "3.14.12 (29 March 2014) debian",
    "XOFFBATT": "2017-07-04T03:55:55",
    "XONBATT": "2017-07-04T03:55:53"
}

我嘗試將同一對象發布2次不同的時間，並最終收到以下錯誤消息（每次不同） 

[{'index': 0, 'errors': [{u'debugInfo': u'generic::invalid_argument: Cannot convert value to integer (bad value):CUMONBATT', u'reason': u'invalid', u'message': u'Cannot convert value to integer (bad value):CUMONBATT', u'location': u'cumonbatt'}]}]
[{'index': 0, 'errors': [{u'debugInfo': u'generic::out_of_range: Invalid datetime string "LOADPCT"', u'reason': u'invalid', u'message': u'Invalid datetime string "LOADPCT"', u'location': u'endapc'}]}]

編輯2：

通過使用表的返回模式來構造數組而不是傳入對象來修復 

e = []
for schema_item in table._schema:
    e.append(d[schema_item.name])

rows = [e]

1 個解決方案

解決方案1
 4 已采納  2017-07-04 20:47:58

您用作示例的行確實與您的架構不一致。 應該是這樣的： 

{
    "ALARMDEL": "No alarm",
    "APC": "001,035,0928",
    "BATTDATE": "2016-12-02",
    "BATTV": 13.7,
    "BCHARGE": 100.0,
    "CABLE": "USB Cable",
    "CUMONBATT": 2758,
    "DATE": "2017-07-04T20:20:26",
    "DRIVER": "USB UPS Driver",
    "ENDAPC": "2017-07-04T20:20:35",
    "FIRMWARE": "904.W1 .D USB FW:W1",
    "HITRANS": 139.0,
    "HOSTNAME": "raspberrypi-master",
    "LASTXFER": "Unacceptable line voltage changes",
    "LINEV": 120.0,
    "LOADPCT": 0.0,
    "LOTRANS": 92.0,
    "MAXTIME": 0,
    "MBATTCHG": 5,
    "MINTIMEL": 3,
    "MODEL": "Back-UPS ES 550G",
    "NOMBATTV": 12.0,
    "NOMINV": 120,
    "NUMXFERS": 188,
    "SENSE": "Medium",
    "SERIALNO": "REDACTED",
    "STARTTIME": "2017-05-23T13:27:07",
    "STATFLAG": "0x05000008",
    "STATUS": "ONLINE",
    "TIMELEFT": 335.8,
    "TONBATT": 0,
    "UPSMODE": "Stand Alone",
    "UPSNAME": "raspberrypi-master",
    "VERSION": "3.14.12 (29 March 2014) debian",
    "XOFFBATT": "2017-07-04T03:55:55",
    "XONBATT": "2017-07-04T03:55:53"
}

請注意， FLOATS應該表示為FLOAT號，而不是STRING （例如，以19.2代替“ 19.2”）。

剩下的一個問題是字段STATFLAG ，您將其定義為INTEGER，但它具有字符“ x”，也許您必須將其類型更改為STRING 。

[編輯]：

至於python代碼，只需記住您應該使用Table類中的insert_data方法，對於輸入的rows ，應該有一個包含要插入的每一行的元組的列表，例如： 

rows=[(92.0, '904.W1 .D USB FW:W1', 5, 13.7, 3, '2017-05-23T13:27:07', 12.0, '2017-07-04T20:20:26', 'Back-UPS ES 550G', 139.0, 335.8, 100.0, '2017-07-4T03:55:53', 0, 120.0, '2017-07-04T03:55:55', 'No alarm', 'ONLINE', 758, 'USB Cable', 'Stand Alone', 'Unacceptable line voltage changes', 'USB UPS river', '3.14.12 (29 March 2014) debian', 0.0, 120, '2017-07-4T20:20:35', 'REDACTED', 'raspberrypi-master', 'raspberrypi-master', 88, 'Medium', '001,035,0928', 0, '2016-12-02', 5000008)]
table.insert_data(rows)

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 自動檢測 Python 中的非確定性行為 測試python函數的非確定性行為 為什么Python2中的某些代碼確定性和Python 3中的非確定性？ 確定性python腳本以非確定性方式運行 使用集合顯示非確定性行為的種子 Python RNG Python 中 {ab, abc}* 的非確定性有限自動機 PyTorch 非確定性輟學 scipy linalg確定性/不確定性代碼 為什么字典排序是不確定的？ SimpleParse非確定性語法，直到運行
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM