Basic Card is not displayed in the Google action simulator console, and also not on my iPhone using google assistant

Question

Basic Card is not displayed in the Google action simulator console, and also not on my iPhone using Google Assistant

Here is the JSON code I send :

{
    "expectUserResponse": true,
    "expectedInputs": [
        {
            "possibleIntents": [
                {
                    "intent": "actions.intent.TEXT"
                }
            ],
            "inputPrompt": {
                "richInitialPrompt": {
                    "items": [
                        {
                            "simpleResponse": {
                                "textToSpeech": "This is a basic card example."
                            }
                        },
                        {
                            "basicCard": {
                                "title": "Title: this is a title",
                                "subtitle": "This is a subtitle",
                                "formattedText": "This is a basic card.  Text in a basic card can include \"quotes\" and\n        most other unicode characters including emoji 📱.  Basic cards also support\n        some markdown formatting like *emphasis* or _italics_, **strong** or\n        __bold__, and ***bold itallic*** or ___strong emphasis___ as well as other\n        things like line  \nbreaks",
                                "image": {
                                    "url": "https://example.com/image.png",
                                    "accessibilityText": "Image alternate text"
                                },
                                "buttons": [
                                    {
                                        "title": "This is a button",
                                        "openUrlAction": {
                                            "url": "https://assistant.google.com/"
                                        }
                                    }
                                ],
                                "imageDisplayOptions": "CROPPED"
                            }
                        }
                    ]
                }
            }
        }
    ]
}

and here is what I get in the simulator and on the iPhone using Google assistant : [object Object]

The debug in the simulator returns :

{
  "response": "[object Object]",
  "expectUserResponse": true,
  "conversationToken": "EroCS2w1Tm...",
  "audioResponse": "//NExAAAAA...",
  "ssmlMarkList": [],
  "debugInfo": {
    "assistantToAgentDebug": {
      "curlCommand": "curl -v https://88.176.64.72:8081/ -H 'Content-Type: application/json;charset=UTF-8' -H 'Google-Actions-API-Version: 2' -H 'Authorization: eyJhbGciOiJSUzI1NiIsImtpZCI6IjM3ODJkM2YwYmM4OTAwOGQ5ZDJjMDE3MzBmNzY1Y2ZiMTlkM2I3MGUiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJhdWQiOiJteS10ZXN0LWFwcC1kMTNkZSIsIm5iZiI6MTU1NTI3NTE2OCwiaWF0IjoxNTU1Mjc1NDY4LCJleHAiOjE1NTUyNzU1ODgsImp0aSI6IjVhZWM0ZjAwNzJiNmNjMTcyMDlmZTdiMmJlZDhjZDRlZTI4ZjExYTIifQ.P-SnzkjiWcr-GubTRdT_juTUVAPBn9J6spSRHPqykwsBq3AppsHg2GNBzlDfwhAXbxZAogW-Mdr4k_U2E1cQMGu-fdGnrkkg4UmjGvYz8za5SGaN-OVx3TNYsoayGIgTFn01gmZOIZfI_33-OucZtFSQCHo82YEmK6ypz3kBq5_vGAjdu01cgYEHAXdT6c53LGSEoewhK4F2M-KphlSx3eFYj2yLWJFn7w9w-Yf3W1n5Rm9q9ZGFJ4vNKIZlX0_J-T-6HhB84OX6k9qJYZ8_1FXp6CS9bPOAo_Nid9k1OeONDIJcCwp1GnQTQB4dek77xybmUn5Qo4-ad1IOzHOkNA'  -A 'Mozilla/5.0 (compatible; Google-Cloud-Functions/2.1; +http://www.google.com/bot.html)' -X POST -d '{\"user\":{\"userId\":\"ABwppHGZMM6CHA-JcPkrCzVpkgGv953hFvVdAGAEGOWQSETGxFO18zXyIbXrhHAlw63M9Gz7dKcFxn3fIGKd2sw\",\"locale\":\"fr-CA\",\"lastSeen\":\"2019-04-14T20:53:31Z\",\"userStorage\":\"{\\\"data\\\":{}}\"},\"conversation\":{\"conversationId\":\"ABwppHEJaVmbTFuCLu0rK3SBxm_bviFYhyoY6oIa8o3MNqI2gHalTVPumj9cetdIsmtbVlfU-vNhIxBGsYRmWvs\",\"type\":\"ACTIVE\",\"conversationToken\":\"{}\"},\"inputs\":[{\"intent\":\"actions.intent.TEXT\",\"rawInputs\":[{\"inputType\":\"KEYBOARD\",\"query\":\"image\"}],\"arguments\":[{\"name\":\"text\",\"rawText\":\"image\",\"textValue\":\"image\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.ACCOUNT_LINKING\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"}]}],\"requestType\":\"SIMULATOR\"}'",
      "assistantToAgentJson": "{\"user\":{\"userId\":\"ABwppHGZMM6CHA-JcPkrCzVpkgGv953hFvVdAGAEGOWQSETGxFO18zXyIbXrhHAlw63M9Gz7dKcFxn3fIGKd2sw\",\"locale\":\"fr-CA\",\"lastSeen\":\"2019-04-14T20:53:31Z\",\"userStorage\":\"{\\\"data\\\":{}}\"},\"conversation\":{\"conversationId\":\"ABwppHEJaVmbTFuCLu0rK3SBxm_bviFYhyoY6oIa8o3MNqI2gHalTVPumj9cetdIsmtbVlfU-vNhIxBGsYRmWvs\",\"type\":\"ACTIVE\",\"conversationToken\":\"{}\"},\"inputs\":[{\"intent\":\"actions.intent.TEXT\",\"rawInputs\":[{\"inputType\":\"KEYBOARD\",\"query\":\"image\"}],\"arguments\":[{\"name\":\"text\",\"rawText\":\"image\",\"textValue\":\"image\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.ACCOUNT_LINKING\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"}]}],\"requestType\":\"SIMULATOR\"}",
      "delegatedRequest": {
        "delegatedRequest": ""
      }
    },
    "agentToAssistantDebug": {
      "agentToAssistantJson": "{\"conversationToken\":\"{}\",\"expectUserResponse\":true,\"expectedInputs\":[{\"inputPrompt\":{\"initialPrompts\":[{\"textToSpeech\":\"[object Object]\"}],\"noInputPrompts\":[]},\"possibleIntents\":[{\"intent\":\"actions.intent.TEXT\"}]}]}",
      "delegatedResponse": {
        "delegatedResponse": ""
      }
    },
    "sharedDebugInfoList": []
  },
  "visualResponse": {
    "visualElementsList": [
      {
        "displayText": {
          "content": "[object Object]"
        }
      }
    ],
    "suggestionsList": [],
    "agentLogoUrl": "https://www.gstatic.com/voice/opa/partner_icons/generic_3p_avatar.png",
    "agentStyle": {
      "primaryColor": "",
      "fontFamily": "",
      "borderRadius": 0,
      "backgroundColor": "",
      "backgroundImageUrl": ""
    }
  },
  "clientError": 0,
  "is3pResponse": true,
  "clientOperationList": [
    {
      "operationType": 1,
      "micUpdatePayLoad": {
        "micMode": 1
      }
    }
  ],
  "projectName": ""
}

Why am I not getting the correct results?

I use Node-Red and the google-action-contrib to make the link beetween ation-on-google and my machine. Everything works fine, I can create a dialog, receive and send sentences... Now, I would like to send a BasicCard (because it seems hat is the correct way to send an image, and many other things, like a button, etc...)

I have put as many datas as possble in this forum, with attached files (wireshark capture, debug information, code): https://discourse.nodered.org/t/google-action-response-with-an-image-basic-card/10145/7

Thanks for your help

Answer 1

It looks like you're not actually sending the JSON as JSON, but rather you've built an object, and are sending the toString version of it. This is suggested by the part of the response logged that says

"response": "[object Object]"

Without seeing the code you are using to send the response, it is pretty difficult to help further.

Answer 2

I assume you've used the sample code in your action. However, unless you changed the url fields, your action can not find the imageUrl and openUrlAction.

If you change url fields with actual(not " http://example.com ") links, your app will respond properly.

Also make sure you've added necessary classes.

eg

const { dialogflow, BasicCard, Image, Button } = require('actions-on-google');