How do I change owner permissions of a directory inside my elasticsearch docker container?

Question

I have an elasticsearch docker container in my VM which works fine with elasticsearch version 2.3 installed using docker image. However on upgrading the elasticsearch to version 7.1.1 (using docker image), I received errors. While investing the errors I found that it's the permissions of the directory which I have mentioned in the volumes for my docker container is the root cause for the error. When I try changing the permission manually ie by running the command chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data in my docker container for elasticsearch, it works. But when I am trying to do the same with my ansible task file it doesn't work and the elasticsearch docker container keeps restarting. Below I have pasted my ansible tasks main.yml file (ansible\\roles\\elasticsearch1\\tasks\\main.yml).

I am new to docker and ansible so any help in this regard would be great.

I have already tried giving command: chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data as one of the params in the docker_container entry in my main.yml.

- name: Data dir exists
  file:
    path: "{{ elasticsearch_data }}"
    state: directory
    mode: 0755

- name: elasticsearch-1 container is running
  docker_container:
    name: elasticsearch-1
    image: "{{elasticsearch_image_name}}:{{elasticsearch_image_version}}"
    state: started
    restart: yes
    restart_policy: "{{ docker_container_restart }}"
    volumes:
      - "{{ elasticsearch_data }}:/usr/share/elasticsearch/data"
    env:
      discovery.type: "single-node"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    published_ports:
      - "{{elasticsearch_rest_port}}:9200"
      - "{{elasticsearch_mgnt_port}}:9300" 

And the below is the error from the docker logs:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.                                                               
{"type": "server", "timestamp": "2019-08-01T12:19:21,708+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "70b2e205184
 thread [main]" ,                                                                                                                                                                                       
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locati
 were started without increasing [node.max_local_storage_nodes] (was [1])?",                                                                                                                            
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                           
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                        
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                           
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.1.jar:7.1.1]",                                                                                     
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.1.jar:7.1.1]",                                                                                                          
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                           
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                            
"Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were starte
torage_nodes] (was [1])?",                                                                                                                                                                              
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                           
"at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                                                
"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                                                
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                               
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                                  
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                                   
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]",                                                                                           
"... 6 more",  

And the output of the playbook is as follows :

PLAY [db1] *********************************************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************

+ app_exit_code=0
+ wait 5898

TASK [Gathering Facts] *********************************************************
task path: /home/system/ansible/00020-elasticsearch1.yml:2
ok: [10.100.192.342]
META: ran handlers

TASK [elasticsearch1 : Data dir exists] ****************************************
task path: /home/system/ansible/roles/elasticsearch1/tasks/main.yml:1
ok: [10.100.192.342] => {"changed": false, "gid": 1002, "group": "hurgrp", "mode": "0755", "owner": "huruser", "path": "/data/elasticsearch1/data", "secontext": "unconfined_u:object_r:default_t:s0", "size": 19, "state": "directory", "uid": 1001}

TASK [elasticsearch1 : elasticsearch-1 container is running] *******************
task path: /home/system/ansible/roles/elasticsearch1/tasks/main.yml:7
changed: [10.100.192.342] => {"ansible_facts": {"docker_container": {"AppArmorProfile": "", "Args": ["eswrapper"], "Config": {"ArgsEscaped": true, "AttachStderr": false, "AttachStdin": false, "AttachStdout": false, "Cmd": ["eswrapper"], "Domainname": "", "Entrypoint": ["/usr/local/bin/docker-entrypoint.sh"], "Env": ["discovery.type=single-node", "ES_JAVA_OPTS=-Xms512m -Xmx512m", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true"], "ExposedPorts": {"9200/tcp": {}, "9300/tcp": {}}, "Hostname": "1ec23e124b08", "Image": "elasticsearch:7.1.1", "Labels": {"license": "Elastic License", "org.label-schema.build-date": "20190305", "org.label-schema.license": "GPLv2", "org.label-schema.name": "elasticsearch", "org.label-schema.schema-version": "1.0", "org.label-schema.url": "https://www.elastic.co/products/elasticsearch", "org.label-schema.vcs-url": "https://github.com/elastic/elasticsearch", "org.label-schema.vendor": "Elastic", "org.label-schema.version": "7.1.1"}, "OnBuild": null, "OpenStdin": false, "StdinOnce": false, "Tty": false, "User": "", "Volumes": {"/usr/share/elasticsearch/data": {}}, "WorkingDir": "/usr/share/elasticsearch"}, "Created": "2019-08-26T13:33:25.098000492Z", "Driver": "overlay2", "ExecIDs": null, "GraphDriver": {"Data": {"LowerDir": "/var/lib/docker/overlay2/c2609676f4fa042fe666d1885ca1ddb3a6f1f2be8d4272a64a901a0ffa5d27f2-init/diff:/var/lib/docker/overlay2/8080db911ac1123a227a623d79054f7b37480d493d254da67073aa197adf48e4/diff:/var/lib/docker/overlay2/ab79afd0a77cd3f3210663033480a99a90581e38414a0b5f084abf98aab3470c/diff:/var/lib/docker/overlay2/181a2facaf7eab27e38ed5d6a403aa5bf1968b2a2da47c5fcf480bcdf855e863/diff:/var/lib/docker/overlay2/7bcd8bdef9bab37695e226fcd0c0984da878516951d3e6af1ef78ae8a02ede60/diff:/var/lib/docker/overlay2/993738850cca9ca3b73bd65cefb07862369705aca8b5d0db5e646d63263e3771/diff:/var/lib/docker/overlay2/b11080b6c1e61ec621e1af3575df720a0b535eda80dc2dc9abee45883badb541/diff:/var/lib/docker/overlay2/3c2669b57199903d1b02811a73d6ec387fbaed6085280979ce29b7b3c09f9331/diff", "MergedDir": "/var/lib/docker/overlay2/c2609676f4fa042fe666d1885ca1ddb3a6f1f2be8d4272a64a901a0ffa5d27f2/merged", "UpperDir": "/var/lib/docker/overlay2/c2609676f4fa042fe666d1885ca1ddb3a6f1f2be8d4272a64a901a0ffa5d27f2/diff", "WorkDir": "/var/lib/docker/overlay2/c2609676f4fa042fe666d1885ca1ddb3a6f1f2be8d4272a64a901a0ffa5d27f2/work"}, "Name": "overlay2"}, "HostConfig": {"AutoRemove": false, "Binds": ["/data/elasticsearch1/data:/usr/share/elasticsearch/data:rw"], "BlkioDeviceReadBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceWriteIOps": null, "BlkioWeight": 0, "BlkioWeightDevice": null, "CapAdd": null, "CapDrop": null, "Cgroup": "", "CgroupParent": "", "ConsoleSize": [0, 0], "ContainerIDFile": "", "CpuCount": 0, "CpuPercent": 0, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpuShares": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DiskQuota": 0, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IOMaximumBandwidth": 0, "IOMaximumIOps": 0, "IpcMode": "", "Isolation": "", "KernelMemory": 0, "Links": null, "LogConfig": {"Config": {}, "Type": "journald"}, "Memory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": -1, "NanoCpus": 0, "NetworkMode": "default", "OomKillDisable": false, "OomScoreAdj": 0, "PidMode": "", "PidsLimit": 0, "PortBindings": {"9200/tcp": [{"HostIp": "0.0.0.0", "HostPort": "9201"}], "9300/tcp": [{"HostIp": "0.0.0.0", "HostPort": "9301"}]}, "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "RestartPolicy": {"MaximumRetryCount": 0, "Name": "unless-stopped"}, "Runtime": "docker-runc", "SecurityOpt": null, "ShmSize": 67108864, "UTSMode": "", "Ulimits": null, "UsernsMode": "", "VolumeDriver": "", "VolumesFrom": null}, "HostnamePath": "/var/lib/docker/containers/1ec23e124b084249946a3e8569c7090b0088eaefd7c8b55aa05c90cca56ca65e/hostname", "HostsPath": "/var/lib/docker/containers/1ec23e124b084249946a3e8569c7090b0088eaefd7c8b55aa05c90cca56ca65e/hosts", "Id": "1ec23e124b084249946a3e8569c7090b0088eaefd7c8b55aa05c90cca56ca65e", "Image": "sha256:b0e9f9f047e6b49bdf540f84a9cd9004886bd17bb5bedd27692f1b4d1ec41355", "LogPath": "", "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c107,c1010", "Mounts": [{"Destination": "/usr/share/elasticsearch/data", "Mode": "rw", "Propagation": "rprivate", "RW": true, "Source": "/data/elasticsearch1/data", "Type": "bind"}], "Name": "/elasticsearch-1", "NetworkSettings": {"Bridge": "", "EndpointID": "14a0263746886f75eb7776af9aa5b2919aef696db76d53f0fde72164107938db", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "HairpinMode": false, "IPAddress": "172.17.0.5", "IPPrefixLen": 16, "IPv6Gateway": "", "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:05", "Networks": {"bridge": {"Aliases": null, "EndpointID": "14a0263746886f75eb7776af9aa5b2919aef696db76d53f0fde72164107938db", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "172.17.0.5", "IPPrefixLen": 16, "IPv6Gateway": "", "Links": null, "MacAddress": "02:42:ac:11:00:05", "NetworkID": "652a5457affbd71402c4c480be83bd0580e25024f9cd5985d7202f2c1170f08a"}}, "Ports": {"9200/tcp": [{"HostIp": "0.0.0.0", "HostPort": "9201"}], "9300/tcp": [{"HostIp": "0.0.0.0", "HostPort": "9301"}]}, "SandboxID": "7a6d886760f0b6ba6abda5ee0d0e86e60ef929a8b8bf6203e142ba997b1ef7a5", "SandboxKey": "/var/run/docker/netns/7a6d886760f0", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null}, "Path": "/usr/local/bin/docker-entrypoint.sh", "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c107,c1010", "ResolvConfPath": "/var/lib/docker/containers/1ec23e124b084249946a3e8569c7090b0088eaefd7c8b55aa05c90cca56ca65e/resolv.conf", "RestartCount": 0, "State": {"Dead": false, "Error": "", "ExitCode": 0, "FinishedAt": "0001-01-01T00:00:00Z", "OOMKilled": false, "Paused": false, "Pid": 11802, "Restarting": false, "Running": true, "StartedAt": "2019-08-26T13:33:25.519298411Z", "Status": "running"}}}, "changed": true}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
10.100.192.342             : ok=3    changed=1    unreachable=0    failed=0

+ db_exit_code=0
+ exit_code=0
+ [[ 0 != 0 ]]
+ [[ 0 != 0 ]]
+ [[ 0 != 0 ]]

Answer 1

I agree with Paul Becotte and you need to give access to elasticsearch user and group inside the container. It might be good idea to find container UID and GID for elasticsearch and give access in ansible script however it will be easy if we might simply give access using below command

- name: Give data dir access to elasticsearch user inside elasticsearch-1 container
  command: docker exec elasticsearch-1 chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data

after docker_container module. so entire ansible script looks like below and hope this helps.

- name: Data dir exists
  file:
    path: "{{ elasticsearch_data }}"
    state: directory
    mode: 0755

- name: elasticsearch-1 container is running
  docker_container:
    name: elasticsearch-1
    image: "{{elasticsearch_image_name}}:{{elasticsearch_image_version}}"
    state: started
    restart: yes
    restart_policy: "{{ docker_container_restart }}"
    volumes:
      - "{{ elasticsearch_data }}:/usr/share/elasticsearch/data"
    env:
      discovery.type: "single-node"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    published_ports:
      - "{{elasticsearch_rest_port}}:9200"
      - "{{elasticsearch_mgnt_port}}:9300" 
- name: Give data dir access to elasticsearch user inside elasticsearch-1 container
  command: docker exec elasticsearch-1 chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data

Answer 2

Your issue is that Docker wraps your process- so user elasticsearch inside the container is NOT the same as user elasticsearch outside the container (they will have different UID and GID).

Assuming that the elasticsearch container uses a fixed UID, you should specify THAT UID in your ansible script to make this work properly.