堆栈内存溢出
  繁体   English   中英

流插入中python客户端的非确定性错误

[英]Non-deterministic errors with python client on streaming inserts

 原文  2017-07-04 20:35:33  3   1    python/ google-bigquery

问题描述

我有一个具有以下架构的BigQuery表： 

    [
    {
        "mode": "REQUIRED",
        "name": "LOTRANS",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "FIRMWARE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "MBATTCHG",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "BATTV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "MINTIMEL",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "STARTTIME",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "NOMBATTV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "DATE",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "MODEL",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "HITRANS",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "TIMELEFT",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "BCHARGE",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "XONBATT",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "TONBATT",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "LINEV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "XOFFBATT",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "ALARMDEL",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "STATUS",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "CUMONBATT",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "CABLE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "UPSMODE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "LASTXFER",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "DRIVER",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "VERSION",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "LOADPCT",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "NOMINV",
        "type": "FLOAT"
    },
    {
        "mode": "REQUIRED",
        "name": "ENDAPC",
        "type": "DATETIME"
    },
    {
        "mode": "REQUIRED",
        "name": "SERIALNO",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "HOSTNAME",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "UPSNAME",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "NUMXFERS",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "SENSE",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "APC",
        "type": "STRING"
    },
    {
        "mode": "REQUIRED",
        "name": "MAXTIME",
        "type": "INTEGER"
    },
    {
        "mode": "REQUIRED",
        "name": "BATTDATE",
        "type": "DATE"
    },
    {
        "mode": "REQUIRED",
        "name": "STATFLAG",
        "type": "INTEGER"
    }
]

数据行的示例如下： 

{
    "ALARMDEL": "No alarm",
    "APC": "001,035,0928",
    "BATTDATE": "2016-12-02",
    "BATTV": "13.7",
    "BCHARGE": "100.0",
    "CABLE": "USB Cable",
    "CUMONBATT": "2758",
    "DATE": "2017-07-04T20:20:26",
    "DRIVER": "USB UPS Driver",
    "ENDAPC": "2017-07-04T20:20:35",
    "FIRMWARE": "904.W1 .D USB FW:W1",
    "HITRANS": "139.0",
    "HOSTNAME": "raspberrypi-master",
    "LASTXFER": "Unacceptable line voltage changes",
    "LINEV": "120.0",
    "LOADPCT": "0.0",
    "LOTRANS": "92.0",
    "MAXTIME": "0",
    "MBATTCHG": "5",
    "MINTIMEL": "3",
    "MODEL": "Back-UPS ES 550G",
    "NOMBATTV": "12.0",
    "NOMINV": "120",
    "NUMXFERS": "188",
    "SENSE": "Medium",
    "SERIALNO": "REDACTED",
    "STARTTIME": "2017-05-23T13:27:07",
    "STATFLAG": "0x05000008",
    "STATUS": "ONLINE",
    "TIMELEFT": "335.8",
    "TONBATT": "0",
    "UPSMODE": "Stand Alone",
    "UPSNAME": "raspberrypi-master",
    "VERSION": "3.14.12 (29 March 2014) debian",
    "XOFFBATT": "2017-07-04T03:55:55",
    "XONBATT": "2017-07-04T03:55:53"
}

我尝试按照示例代码进行操作 ，但是每当尝试使用python客户端使用上述字典对insert_data进行操作时，都会出现诸如以下的不确定性错误： 

[{'index': 0, 'errors': [{u'debugInfo': u'generic::invalid_argument: Cannot convert value to integer (bad value):MBATTCHG', u'reason': u'invalid', u'message': u'Cannot convert value to integer (bad value):MBATTCHG', u'location': u'mbattchg'}]}]

但是错误会在每次尝试插入时发生变化。 有什么方法可以帮助调试为什么错误不是确定性的，为什么转换后的值往往是字典的键而不是值？

编辑

将事件修改为以下内容，但仍然出现错误 

{
    "ALARMDEL": "No alarm",
    "APC": "001,035,0928",
    "BATTDATE": "2016-12-02",
    "BATTV": 13.7,
    "BCHARGE": 100.0,
    "CABLE": "USB Cable",
    "CUMONBATT": 2758,
    "DATE": "2017-07-05T02:41:00",
    "DRIVER": "USB UPS Driver",
    "ENDAPC": "2017-07-05T02:41:07",
    "FIRMWARE": "904.W1 .D USB FW:W1",
    "HITRANS": 139.0,
    "HOSTNAME": "raspberrypi-master",
    "LASTXFER": "Unacceptable line voltage changes",
    "LINEV": 123.0,
    "LOADPCT": 0.0,
    "LOTRANS": 92.0,
    "MAXTIME": 0,
    "MBATTCHG": 5,
    "MINTIMEL": 3.0,
    "MODEL": "Back-UPS ES 550G",
    "NOMBATTV": 12.0,
    "NOMINV": 120.0,
    "NUMXFERS": "188",
    "SENSE": "Medium",
    "SERIALNO": "REDACTED",
    "STARTTIME": "2017-05-23T13:27:07",
    "STATFLAG": 83886088,
    "STATUS": "ONLINE",
    "TIMELEFT": 335.8,
    "TONBATT": 0,
    "UPSMODE": "Stand Alone",
    "UPSNAME": "raspberrypi-master",
    "VERSION": "3.14.12 (29 March 2014) debian",
    "XOFFBATT": "2017-07-04T03:55:55",
    "XONBATT": "2017-07-04T03:55:53"
}

我尝试将同一对象发布2次不同的时间，并最终收到以下错误消息（每次不同） 

[{'index': 0, 'errors': [{u'debugInfo': u'generic::invalid_argument: Cannot convert value to integer (bad value):CUMONBATT', u'reason': u'invalid', u'message': u'Cannot convert value to integer (bad value):CUMONBATT', u'location': u'cumonbatt'}]}]
[{'index': 0, 'errors': [{u'debugInfo': u'generic::out_of_range: Invalid datetime string "LOADPCT"', u'reason': u'invalid', u'message': u'Invalid datetime string "LOADPCT"', u'location': u'endapc'}]}]

编辑2：

通过使用表的返回模式来构造数组而不是传入对象来修复 

e = []
for schema_item in table._schema:
    e.append(d[schema_item.name])

rows = [e]

1 个解决方案

解决方案1
 4 已采纳  2017-07-04 20:47:58

您用作示例的行确实与您的架构不一致。 应该是这样的： 

{
    "ALARMDEL": "No alarm",
    "APC": "001,035,0928",
    "BATTDATE": "2016-12-02",
    "BATTV": 13.7,
    "BCHARGE": 100.0,
    "CABLE": "USB Cable",
    "CUMONBATT": 2758,
    "DATE": "2017-07-04T20:20:26",
    "DRIVER": "USB UPS Driver",
    "ENDAPC": "2017-07-04T20:20:35",
    "FIRMWARE": "904.W1 .D USB FW:W1",
    "HITRANS": 139.0,
    "HOSTNAME": "raspberrypi-master",
    "LASTXFER": "Unacceptable line voltage changes",
    "LINEV": 120.0,
    "LOADPCT": 0.0,
    "LOTRANS": 92.0,
    "MAXTIME": 0,
    "MBATTCHG": 5,
    "MINTIMEL": 3,
    "MODEL": "Back-UPS ES 550G",
    "NOMBATTV": 12.0,
    "NOMINV": 120,
    "NUMXFERS": 188,
    "SENSE": "Medium",
    "SERIALNO": "REDACTED",
    "STARTTIME": "2017-05-23T13:27:07",
    "STATFLAG": "0x05000008",
    "STATUS": "ONLINE",
    "TIMELEFT": 335.8,
    "TONBATT": 0,
    "UPSMODE": "Stand Alone",
    "UPSNAME": "raspberrypi-master",
    "VERSION": "3.14.12 (29 March 2014) debian",
    "XOFFBATT": "2017-07-04T03:55:55",
    "XONBATT": "2017-07-04T03:55:53"
}

请注意， FLOATS应该表示为FLOAT号，而不是STRING （例如，以19.2代替“ 19.2”）。

剩下的一个问题是字段STATFLAG ，您将其定义为INTEGER，但它具有字符“ x”，也许您必须将其类型更改为STRING 。

[编辑]：

至于python代码，只需记住您应该使用Table类中的insert_data方法，对于输入的rows ，应该有一个包含要插入的每一行的元组的列表，例如： 

rows=[(92.0, '904.W1 .D USB FW:W1', 5, 13.7, 3, '2017-05-23T13:27:07', 12.0, '2017-07-04T20:20:26', 'Back-UPS ES 550G', 139.0, 335.8, 100.0, '2017-07-4T03:55:53', 0, 120.0, '2017-07-04T03:55:55', 'No alarm', 'ONLINE', 758, 'USB Cable', 'Stand Alone', 'Unacceptable line voltage changes', 'USB UPS river', '3.14.12 (29 March 2014) debian', 0.0, 120, '2017-07-4T20:20:35', 'REDACTED', 'raspberrypi-master', 'raspberrypi-master', 88, 'Medium', '001,035,0928', 0, '2016-12-02', 5000008)]
table.insert_data(rows)

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 自动检测 Python 中的非确定性行为 测试python函数的非确定性行为 为什么Python2中的某些代码确定性和Python 3中的非确定性？ 确定性python脚本以非确定性方式运行 使用集合显示非确定性行为的种子 Python RNG Python 中 {ab, abc}* 的非确定性有限自动机 PyTorch 非确定性辍学 scipy linalg确定性/不确定性代码 为什么字典排序是不确定的？ SimpleParse非确定性语法，直到运行
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM