AWS IoT MQTT only works on the example topic?

Question

I've just gotten started in AWS and IoT. Using the documentation and the tutorial I managed to get a working publish app ripped from the sample classes:

    public static void main(String[] args) throws AWSIotException, InterruptedException {
        String clientEndpoint = "<prefix>-ats.iot.us-west-2.amazonaws.com";       // replace <prefix> and <region> with your own
        String clientId = "sdk-java-23";                              // replace with your own client ID. Use unique client IDs for concurrent connections.
        String certificateFile = "athing.cert.pem";                       // X.509 based certificate file
        String privateKeyFile = "athing.private.key";                        // PKCS#1 or PKCS#8 PEM encoded private key file

// SampleUtil.java and its dependency PrivateKeyReader.java can be copied from the sample source code.
// Alternatively, you could load key store directly from a file - see the example included in this README.

        SampleUtil.KeyStorePasswordPair pair = SampleUtil.getKeyStorePasswordPair(certificateFile, privateKeyFile);
        AWSIotMqttClient client = new AWSIotMqttClient(clientEndpoint, clientId, pair.keyStore, pair.keyPassword);

// optional parameters can be set before connect()
        client.connect();

        String topic = "sdk/test/java";
        String payload = "[\n" +
                "{\n" +
                " \"id\": \"1231231234123\",\n" +
                " \"value\": \"25\",\n" +
                " \"unit\": \"°C\",\n" +
                " \"timestamp\": \"1585954728\"\n" +
                "},\n" +
                "{\n" +
                "  \"id\": \"121231231233\",\n" +
                "  \"value\": \"26\",\n" +
                "  \"unit\": \"°B\",\n" +
                "  \"timestamp\": \"1585254728\"\n" +
                "}"+
                "]";

        System.out.println(payload);
        while (true) {
            client.publish(topic, AWSIotQos.QOS0, payload);
            System.out.println("message sent");
            Thread.sleep(2000);
        }

    }

And I can see the messages coming through successfully on the aws console:

But if I change JUST the publish topic from:

String topic = "sdk/test/java";

to:

String topic = "sensors/temperature";

Now it no longer works. I don't see anything appear in the AWS console and the java program is showing some kind of connection error. My first instinct is some kind of security issue where it's not allowed to publish to any topic other than the one that is used in the sample program. I have no experience with IAM, cognito etc. so I would require some guidance (if that is the cause)

Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionSuccess
INFO: Connection successfully established
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionSuccess
INFO: Client connection active: sdk-java
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionFailure
INFO: Connection temporarily lost
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionFailure
INFO: Client connection lost: sdk-java
Apr 04, 2020 4:29:08 PM com.amazonaws.services.iot.client.core.AwsIotConnection$1 run
INFO: Connection is being retried
Apr 04, 2020 4:29:11 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionSuccess
INFO: Connection successfully established
Apr 04, 2020 4:29:11 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionSuccess
INFO: Client connection active: sdk-java

Answer 1

So it turns out it was just a policy issue, I had no idea you had to define which ClientIDs and which topics are allowed to published/subscribed etc.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": [
        "arn:aws:iot:us-west-2:<>:topic/sensors/realtime",
        "arn:aws:iot:us-west-2:<>:topic/sdk/test/java",
        "arn:aws:iot:us-west-2:<>:topic/sdk/test/Python",
        "arn:aws:iot:us-west-2:<>:topic/topic_1",
        "arn:aws:iot:us-west-2:<>:topic/topic_2"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Subscribe"
      ],
      "Resource": [
        "arn:aws:iot:us-west-2:<>:topicfilter/sensors/realtime",
        "arn:aws:iot:us-west-2:<>:topicfilter/sdk/test/java",
        "arn:aws:iot:us-west-2:<>:topicfilter/sdk/test/Python",
        "arn:aws:iot:us-west-2:<>:topicfilter/topic_1",
        "arn:aws:iot:us-west-2:<>:topicfilter/topic_2"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Connect"
      ],
      "Resource": [
        "arn:aws:iot:us-west-2:<>:client/JavaClient2",
        "arn:aws:iot:us-west-2:<>:client/sdk-java",
        "arn:aws:iot:us-west-2:<>:client/basicPubSub",
        "arn:aws:iot:us-west-2:<>:client/sdk-nodejs-*"
      ]
    }
  ]
}