[英]Autodesk Forge - 504 Gateway Timeout when uploading file as chunks to BIM 360 storage in Node JS
I'm getting a 504 Gateway Timeout when I try to upload a large file in "chunks" using the PUT
object resumable API endpoint in Autodesk forge.当我尝试使用 Autodesk forge 中的
PUT
object 可恢复 API 端点以“块”上传大文件时,出现 504 网关超时。
https://forge.autodesk.com/en/docs/data/v2/reference/http/buckets-:bucketKey-objects-:objectName-resumable-PUT/ https://forge.autodesk.com/en/docs/data/v2/reference/http/buckets-:bucketKey-objects-:objectName-resumable-PUT/
My console log output is below, showing the 504 response, along with the PUT request I send.我的控制台日志 output 如下,显示 504 响应以及我发送的 PUT 请求。
I've tried various methods, both using the functions in the forge-apis
SDK in Node, (eg uploadChunk
method from ObjectsAPI
) and also written various functions using request
, request-promise-native
, to check if using await/async, promises vs callbacks can help solve the problem.我尝试了各种方法,既使用 Node 中的 forge
forge-apis
SDK 中的函数(例如uploadChunk
中的ObjectsAPI
方法),还使用request
、 request-promise-native
编写了各种函数,以检查是否使用 await/async、promise vs 回调可以帮助解决问题。
In all cases, I get the same response - a long pause in the log and then a 504 Gateway Timeout - which appears to be coming from the Forge end.在所有情况下,我都会得到相同的响应——日志中长时间停顿,然后是 504 网关超时——这似乎来自 Forge 端。
For context, my Node JS application sits on Heroku, and is triggered from a python script running locally on the client - so Heroku is the 'middleman' - authentication is passed from client-side via python to the node app and then passed on to Forge to authenticate. For context, my Node JS application sits on Heroku, and is triggered from a python script running locally on the client - so Heroku is the 'middleman' - authentication is passed from client-side via python to the node app and then passed on to伪造认证。 Other operations in the flow are successful so I'm confident authentication is working correctly.
流程中的其他操作均成功,因此我确信身份验证正常工作。
Can anyone see anything from my logs below that could be causing the problem?任何人都可以从下面的日志中看到可能导致问题的任何内容吗? Anybody else had similar 504 issues with chunked uploads of large files?
其他人在分块上传大文件时遇到过类似的 504 问题吗? (~130MB in this case)
(在这种情况下约为 130MB)
The first log shows my console log output - I'm sending 5mb chunks at 1-second intervals, and sending the chunk as a Buffer object (bytes) sliced to match the Content-Range第一个日志显示我的控制台日志 output - 我以 1 秒的间隔发送 5mb 块,并将块作为缓冲区 object(字节)切片以匹配 Content-Range
2020-06-22T10:42:33.389835+00:00 app[web.1]: Ready to upload chunk...
2020-06-22T10:42:33.389896+00:00 app[web.1]: simulating waiting for 1000 milliseconds
2020-06-22T10:42:34.391034+00:00 app[web.1]: done waiting
2020-06-22T10:42:34.392893+00:00 app[web.1]: contentRange bytes 129999974-130056191/130056192
2020-06-22T10:42:34.393393+00:00 app[web.1]: requestParams {
2020-06-22T10:42:34.393394+00:00 app[web.1]: headers: {
2020-06-22T10:42:34.393395+00:00 app[web.1]: Authorization: 'Bearer eyJhbGciOiJIxxxxxxxxx',
2020-06-22T10:42:34.393396+00:00 app[web.1]: 'Content-Type': 'application/octet-stream',
2020-06-22T10:42:34.393396+00:00 app[web.1]: 'Content-Range': 'bytes 129999974-130056191/130056192',
2020-06-22T10:42:34.393397+00:00 app[web.1]: 'Content-Length': '130056192',
2020-06-22T10:42:34.393397+00:00 app[web.1]: 'Session-Id': '-75601742'
2020-06-22T10:42:34.393397+00:00 app[web.1]: },
2020-06-22T10:42:34.393398+00:00 app[web.1]: uri: 'https://developer.api.autodesk.com/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable',
2020-06-22T10:42:34.393399+00:00 app[web.1]: url: 'https://developer.api.autodesk.com/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable',
2020-06-22T10:42:34.393399+00:00 app[web.1]: method: 'PUT',
2020-06-22T10:42:34.393400+00:00 app[web.1]: body: <Buffer 6b b1 b4 e4 d7 f5 1e 59 a0 07 0c db 68 7d 9e 98 75 8e 5f fc e8 0c e3 78 a0 9c b9 59 3f e9 0f 28 e5 5a f0 eb 75 7b 68 16 3d 4c c1 ca 20 7a ba 67 f0 69 ... 56167 more bytes>,
2020-06-22T10:42:34.393400+00:00 app[web.1]: resolveWithFullResponse: true
2020-06-22T10:42:34.393400+00:00 app[web.1]: }
2020-06-22T10:42:34.393466+00:00 app[web.1]: Ready to upload chunk...
The response comes back like this for each chunk:每个块的响应都是这样返回的:
2020-06-22T10:43:34.443453+00:00 app[web.1]: Uploading 504 > GATEWAY_TIMEOUT
2020-06-22T10:43:34.443834+00:00 app[web.1]: err: null
2020-06-22T10:43:34.443969+00:00 app[web.1]: res: {
2020-06-22T10:43:34.443969+00:00 app[web.1]: ----"statusCode": 504,
2020-06-22T10:43:34.443970+00:00 app[web.1]: ----"body": "",
2020-06-22T10:43:34.443970+00:00 app[web.1]: ----"headers": {
2020-06-22T10:43:34.443971+00:00 app[web.1]: --------"content-length": "0",
2020-06-22T10:43:34.443971+00:00 app[web.1]: --------"connection": "Close"
2020-06-22T10:43:34.443971+00:00 app[web.1]: ----},
2020-06-22T10:43:34.443972+00:00 app[web.1]: ----"request": {
2020-06-22T10:43:34.443972+00:00 app[web.1]: --------"uri": {
2020-06-22T10:43:34.443973+00:00 app[web.1]: ------------"protocol": "https:",
2020-06-22T10:43:34.443973+00:00 app[web.1]: ------------"slashes": true,
2020-06-22T10:43:34.443973+00:00 app[web.1]: ------------"auth": null,
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"host": "developer.api.autodesk.com",
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"port": 443,
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"hostname": "developer.api.autodesk.com",
2020-06-22T10:43:34.443974+00:00 app[web.1]: ------------"hash": null,
2020-06-22T10:43:34.443975+00:00 app[web.1]: ------------"search": null,
2020-06-22T10:43:34.443975+00:00 app[web.1]: ------------"query": null,
2020-06-22T10:43:34.443975+00:00 app[web.1]: ------------"pathname": "/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable",
2020-06-22T10:43:34.443976+00:00 app[web.1]: ------------"path": "/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable",
2020-06-22T10:43:34.443976+00:00 app[web.1]: ------------"href": "https://developer.api.autodesk.com/oss/v2/buckets/wip.dm.prod/objects/1fad0fad-601b-41b8-bce4-c88edbb353ec.rvt/resumable"
2020-06-22T10:43:34.443977+00:00 app[web.1]: --------},
2020-06-22T10:43:34.443977+00:00 app[web.1]: --------"method": "PUT",
2020-06-22T10:43:34.443977+00:00 app[web.1]: --------"headers": {
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Authorization": "Bearer eyJhbGciOiJxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Content-Type": "application/octet-stream",
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Content-Range": "bytes 129999974-130056191/130056192",
2020-06-22T10:43:34.443978+00:00 app[web.1]: ------------"Content-Length": "130056192",
2020-06-22T10:43:34.443979+00:00 app[web.1]: ------------"Session-Id": "-75601742"
2020-06-22T10:43:34.443979+00:00 app[web.1]: --------}
2020-06-22T10:43:34.443979+00:00 app[web.1]: ----}
2020-06-22T10:43:34.443979+00:00 app[web.1]: }
Thanks to my colleague Luis Felipe Paris - we managed to resolve the problem, after looking at forge-server-utils
as per Petr's advice.感谢我的同事Luis Felipe Paris - 在按照 Petr 的建议查看了
forge-server-utils
之后,我们设法解决了这个问题。
By changing our uploadChunk
function to use the module axios
instead of request
or request-promise
- we found the 504 timeout error disappeared.通过更改我们的
uploadChunk
function 以使用模块axios
而不是request
或request-promise
- 我们发现 504 超时错误消失了。
We were using the npm module request
and request-promise
(even though legacy / deprecated these days) as it supports piping streams from one place to another, and this was working with all other requests we were doing, except uploading a 'resumable' file in separate chunks.我们使用的是 npm 模块
request
和request-promise
(即使这些日子遗留/不推荐使用),因为它支持从一个地方到另一个地方的管道流,这适用于我们正在执行的所有其他请求,除了上传“可恢复”文件在单独的块中。
We had to convert the file payload we were sending from a Buffer
object to an ArrayBuffer
, and slice it correctly to match the Content-Range
of each chunk.我们必须将我们从
Buffer
object 发送的文件有效负载转换为ArrayBuffer
,并将其正确切片以匹配每个块的Content-Range
。
Some of the older forge examples use request
- so I guess this might be useful for other users - if request
gives you problems, try axios
一些较旧的伪造示例使用
request
- 所以我想这可能对其他用户有用 - 如果request
给您带来问题,请尝试axios
EDIT: when using the resumable upload feature, make sure that the Content-Length
request header specifies the length of just the chunk you're sending, not the entire size of the uploaded file.编辑:使用可恢复上传功能时,请确保
Content-Length
请求 header 指定您要发送的块的长度,而不是上传文件的整个大小。
The Forge APIs do have some rate throttling but in that case you should probably get a 4xx status code, not 504. Forge API 确实有一些速率限制,但在这种情况下,您可能应该得到 4xx 状态代码,而不是 504。
Can you try a couple of things for us?你能为我们尝试几件事吗?
And if you're still unable to upload the file, could you please share it with us (confidentially) via forge (dot) help (at) autodesk (dot) com
?如果您仍然无法上传文件,您能否通过
forge (dot) help (at) autodesk (dot) com
与我们(保密)分享? We would be happy to investigate it on our end.我们很乐意最终调查它。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.