[英]kubernetes: can't deploy jenkins images with persistent volume with RW access
With kubernetes, I'm trying to deploy jenkins image & a persistent volume mapped to a NFS share (which is mounted on all my workers)使用 kubernetes,我正在尝试部署 jenkins 映像和映射到 NFS 共享的持久卷(安装在我所有的工作人员身上)
[root@pp-tmp-test24 /opt]# df -Th /opt/jenkins.persistent
Filesystem Type Size Used Avail Use% Mounted on
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP nfs4 10G 9.5M 10G 1% /opt/jenkins.persistent
[root@pp-tmp-test24 /opt/jenkins.persistent]# ls -l
total 0
-rwxr-xr-x. 1 root root 0 Oct 2 11:53 newfile
[root@pp-tmp-test24 /opt/jenkins.persistent]# cat newfile
hello
My PersistentVolume yaml我的持久卷 yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-pv-nfs
labels:
type: type-nfs
spec:
storageClassName: class-nfs
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
hostPath:
path: /opt/jenkins.persistent
My PersistentVolumeClaim yaml我的 PersistentVolumeClaim yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc-nfs
namespace: ns-jenkins
spec:
storageClassName: class-nfs
volumeMode: Filesystem
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
selector:
matchLabels:
type: type-nfs
And my deployment还有我的部署
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: ns-jenkins
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- image: jenkins
#- image: httpd:latest
name: jenkins
ports:
- containerPort: 8080
protocol: TCP
name: jenkins-web
volumeMounts:
- name: jenkins-persistent-storage
mountPath: /var/foo
volumes:
- name: jenkins-persistent-storage
persistentVolumeClaim:
claimName: jenkins-pvc-nfs
kubectl create -f
command, all is looking good:在kubectl create -f
命令之后,一切看起来都很好:# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
jenkins-pv-nfs 10Gi RWX Recycle Bound ns-jenkins/jenkins-pvc-nfs class-nfs 37s
# kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ns-jenkins jenkins-pvc-nfs Bound jenkins-pv-nfs 10Gi RWX class-nfs 35s
# kubectl get pods -A |grep jenkins
ns-jenkins jenkins-5bdb8678c-x6vht 1/1 Running 0 14s
# kubectl describe pod jenkins-5bdb8678c-x6vht -n ns-jenkins
Name: jenkins-5bdb8678c-x6vht
Namespace: ns-jenkins
Priority: 0
Node: pp-tmp-test25.mydomain/172.31.68.225
Start Time: Wed, 02 Oct 2019 11:48:23 +0200
Labels: app=jenkins
pod-template-hash=5bdb8678c
Annotations: <none>
Status: Running
IP: 10.244.5.47
Controlled By: ReplicaSet/jenkins-5bdb8678c
Containers:
jenkins:
Container ID: docker://8a3e4871ed64b371818bac59e24d6912e5d2b13c8962c1639d36797fbce8082e
Image: jenkins
Image ID: docker-pullable://docker.io/jenkins@sha256:eeb4850eb65f2d92500e421b430ed1ec58a7ac909e91f518926e02473904f668
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 02 Oct 2019 11:48:26 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/foo from jenkins-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dz6cd (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
jenkins-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: jenkins-pvc-nfs
ReadOnly: false
default-token-dz6cd:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dz6cd
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 39s default-scheduler Successfully assigned ns-jenkins/jenkins-5bdb8678c-x6vht to pp-tmp-test25.mydomain
Normal Pulling 38s kubelet, pp-tmp-test25.mydomain Pulling image "jenkins"
Normal Pulled 36s kubelet, pp-tmp-test25.mydomain Successfully pulled image "jenkins"
Normal Created 36s kubelet, pp-tmp-test25.mydomain Created container jenkins
Normal Started 36s kubelet, pp-tmp-test25.mydomain Started container jenkins
# docker ps |grep jenkins
8a3e4871ed64 docker.io/jenkins@sha256:eeb4850eb65f2d92500e421b430ed1ec58a7ac909e91f518926e02473904f668 "/bin/tini -- /usr..." 2 minutes ago Up 2 minutes k8s_jenkins_jenkins-5bdb8678c-x6vht_ns-jenkins_64b66dae-a1da-4d90-83fd-ff433638dc9c_0
So I launch a shell on my container, and I can see my data on /var/foo
:所以我在我的容器上启动了一个 shell,我可以在/var/foo
上看到我的数据:
# docker exec -t -i 8a3e4871ed64 /bin/bash
jenkins@jenkins-5bdb8678c-x6vht:/$ df -h /var/foo
Filesystem Size Used Avail Use% Mounted on
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP 10G 9.5M 10G 1% /var/foo
jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ ls -lZ /var/foo -d
drwxr-xr-x. 2 root root system_u:object_r:nfs_t:s0 4096 Oct 2 10:06 /var/foo
jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ ls -lZ /var/foo
-rwxr-xr-x. 1 root root system_u:object_r:nfs_t:s0 12 Oct 2 10:05 newfile
jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ cat newfile
hello
I'm trying to write data in my /var/foo/newfile
but the Permission is denied我正在尝试在我的/var/foo/newfile
中写入数据,但权限被拒绝
jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ echo "world" >> newfile
bash: newfile: Permission denied
Same thing in my /var/foo/ directory
, I can't write data我的/var/foo/ directory
中的相同内容,我无法写入数据
jenkins@jenkins-5bdb8678c-x6vht:/var/foo$ touch newfile2
touch: cannot touch 'newfile2': Permission denied
So, I tried an another image like httpd:latest
in my deployment yaml (keeping the same name in my yaml definition)所以,我在我的部署 yaml 中尝试了另一个图像,如httpd:latest
(在我的 yaml 定义中保持相同的名称)
[...]
containers:
#- image: jenkins
- image: httpd:latest
[...]
# docker ps |grep jenkins
fa562400405d docker.io/httpd@sha256:39d7d9a3ab93c0ad68ee7ea237722ed1b0016ff6974d80581022a53ec1e58797 "httpd-foreground" 50 seconds ago Up 48 seconds k8s_jenkins_jenkins-7894877f96-6dj85_ns-jenkins_540b12bd-69df-44d8-b3df-20a0a96cc851_0
In my new container, this time I can Read-Write data:在我的新容器中,这次我可以读写数据:
root@jenkins-7894877f96-6dj85:/usr/local/apache2# df -h /var/foo
Filesystem Size Used Avail Use% Mounted on
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP 10G 9.6M 10G 1% /var/foo
root@jenkins-7894877f96-6dj85:/var/foo# ls -lZ
total 0
-rwxr-xr-x. 1 root root system_u:object_r:nfs_t:s0 12 Oct 2 10:05 newfile
-rw-r--r--. 1 root root system_u:object_r:nfs_t:s0 0 Oct 2 10:06 newfile2
root@jenkins-7894877f96-6dj85:/var/foo# ls -lZ /var/foo -d
drwxr-xr-x. 2 root root system_u:object_r:nfs_t:s0 4096 Oct 2 10:06 /var/foo
root@jenkins-7894877f96-6dj85:/var/foo# ls -l
total 0
-rwxr-xr-x. 1 root root 6 Oct 2 09:55 newfile
root@jenkins-7894877f96-6dj85:/var/foo# echo "world" >> newfile
root@jenkins-7894877f96-6dj85:/var/foo# touch newfile2
root@jenkins-7894877f96-6dj85:/var/foo# ls -l
total 0
-rwxr-xr-x. 1 root root 12 Oct 2 10:05 newfile
-rw-r--r--. 1 root root 0 Oct 2 10:06 newfile2
What I'm doing wrong?我做错了什么? Does the pb is due to jenkins
images who do not allow RW access? pb 是否是由于jenkins
图像不允许 RW 访问? Same pb with a local storage (on my worker) with persistent volume.与具有持久卷的本地存储(在我的工作人员上)相同的 pb。
Other thing, perhaps it is stupid: with my jenkins image, I would like to mount the /var/jenkins_home
dir to a persistent volume in order to keep jenkins's configuration files.另一件事,也许是愚蠢的:使用我的 jenkins 映像,我想将/var/jenkins_home
目录挂载到持久卷以保留 jenkins 的配置文件。 But if I try to mount /var/jenkins_home
instead of /var/foo
, pod is crashinglookbackoff (because there is already data stored in /var/jenkins_home
).但是,如果我尝试挂载/var/jenkins_home
而不是/var/foo
,则 pod 正在崩溃lookbackoff(因为/var/jenkins_home
中已经存储了数据)。
thank you all for your help !谢谢大家的帮助 !
I noticed You are trying to write as jenkins
user on jenkins-5bdb8678c-x6vht
that might not have write permissions in that root:root directory.我注意到您正在尝试在jenkins-5bdb8678c-x6vht
上以jenkins
用户身份写入,该用户可能在该 root:root 目录中没有写入权限。
You might want to change that directory permissions to match jenkins
user privileges.您可能希望更改该目录权限以匹配jenkins
用户权限。
Try to verify that this is causing this issue by using sudo
before writing to file.尝试在写入文件之前使用sudo
验证这是否导致了此问题。
If you sudo
is not installed then exec in with --user
flag as root
user.如果未安装sudo
,则使用--user
标志作为root
用户执行。 So its just like in other cases where writing worked.所以它就像在其他写作工作的情况下一样。
docker exec -t -i -u root 8a3e4871ed64 /bin/bash
@Piotr Malec Thank you. @Piotr Malec 谢谢。 Yes I realized that: jenkins is the default user when I connect to my container:是的,我意识到: jenkins 是我连接到容器时的默认用户:
docker exec -t -i 46d2497d440d /bin/bash
jenkins@jenkins-7bcdd5db57-8qgth:/$
So I have changed permissions on this /opt/jenkins.persistent
to 777 on my worker, in order to try and now I have RW perm on this mount:所以我已经将这个/opt/jenkins.persistent
的权限更改为我的工作人员的 777,以便尝试,现在我在这个挂载上有 RW perm:
xxx.xxx.xxx.xxx:/VR_C_CS003_NFS_KUBERNETESPV_TMP_PP 10G 9.5M 10G 1% /var/foo
jenkins@jenkins-7bcdd5db57-8qgth:/$ cd /var
jenkins@jenkins-7bcdd5db57-8qgth:/$ ls -l
[...]
drwxrwxrwx. 2 root root 4096 Oct 4 13:41 foo
[...]
jenkins@jenkins-7bcdd5db57-8qgth:/$ cd /var/foo
jenkins@jenkins-7bcdd5db57-8qgth:/var/foo $ touch newfile
jenkins@jenkins-7bcdd5db57-8qgth:/var/foo $ ls
newfile
So I added jenkins
user account on my worker and set chown jenkins:jenkins on my /opt/jenkins.persistent
directory.所以我在我的工人上添加了jenkins
用户帐户,并在我的/opt/jenkins.persistent
目录上设置了 chown jenkins:jenkins 。 Now, inside my container I have RW perm:现在,在我的容器内,我有 RW 烫发:
jenkins@jenkins-7bcdd5db57-8qgth:/var$ ls -l
[...]
drwxr-xr-x. 2 jenkins jenkins 4096 Oct 4 13:53 foo
[...]
jenkins@jenkins-7bcdd5db57-8qgth:/var$ cd foo
jenkins@jenkins-7bcdd5db57-8qgth:/var/toto$ touch newfile2
jenkins@jenkins-7bcdd5db57-8qgth:/var/toto$ ls -l
-rw-r--r--. 1 jenkins jenkins 0 Oct 4 13:53 newfile2
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.