Skip to main content

Debugging the issue of using NFS shares for PSMDB on OpenShift


I have recently been trying to use PSMDB (Percona Server for MongoDB) as an open-source and free alternative for MongoDB Enterprise Server. I encountered an issue that the pod could not be initialized successfully with Persistent Volumes using NFS shares. I got the logs from the failed pod as follow:

------

++ id -u

++ id -g

+ install -o 1000730000 -g 0 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

----

I would like to share the steps how I used for debugging. The PSMD StatefulSet was deployed onto my OpenShift 3 OKD.

Check the container mount info

Go to a pod I could see the mount info as below

mongod-data → /data/db read-write

- mongod-data: Persistent volume claim name

- /data/db: container mounted directory

Check Persistent volume binding

Go to the storage, I could know which persistent volume was bound to the corresponding persistent volume claim.

Bound to volume psmdb-mongodb-data-0

Check Persistent volume

Go cluster console > storage > persistent volumes. I had

----

nfs:

 server: files.some.domain.local

 path: /srv/data/psmdb/mongodb/data-0

---

Check NFS shares (host’s mounted directory)

Remote to server and check attributes of the mounted directory

---

ssh someuser@files.some.domain.local

cd /srv/data/psmdb/mongodb

ls -la

---

The info was:

drwxrwxr-x. 2 nfsnobody nfsnobody 6 Jun 22 08:25 data-0

Check the user and group of "nfsnobody".

id nfsnobody

The info was:

uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)

Also, I could also check by "/etc/passwd"

nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

Check container's mounted directory

Create a debug pod

---

# -c: container

# -t: attach a terminal

# --: use the entrypoint

oc debug my-cluster-name-rs0-0 -c mongo-init -t -- /bin/bash

---

Check the mounted directory attributes:

drwxrwxr-x. 2 65534 65534 25 Jun 22 08:43 db

It means the attributes are kept (the same as the host’s one). Check the files inside:

-rw-------. 1 1000920000 nfsnobody 15680 Jun 22 08:48 ps-entry.sh

Also, it was the same group as the host’s one. Only the user id was changed. It made sense. However, it seemed like the file was created but not completely set the attributes (ownership and file mode).

I checked the logged-in user of the container by executing the command "id"

uid=1000990000 gid=0(root) groups=0(root),1000990000

I ran the command from the entry point of the point ("/init-entrypoint.sh") manually within the container.

install -o "$(id -u)" -g "$(id -g)" -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

Then, I got an error:

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

The currently logged-in user within the container doesn’t have permission to change the owner. Yep! I could reproduce the issue.

Hmm...What is the root cause?

The user of the container doesn't belong to group "65534(nfsnobody)", so it could not change the owner of the file. However, I could not do the following actions:

- Change group of container's user to "65543". OpenShift grants a fixed group "0 (root)" for the user.

- Modify the entry point contains the script "install -o ..." since the templates are generated and managed by the PSMDB operator.

Hence, the only way was that I need to change the file attributes of NFS shares. I changed it to group "0".

sudo chown nfsnobody:0 /srv/data/psmdb/mongodb/data-0

Remove the pod and deploy a new one. I got another error:

install: cannot create regular file '/data/db/ps-entry.sh': Permission denied

Strange! The current user has the same group "0" of the folder already. The NFS service doesn’t allow a user without group "nsfnobody" to create a file within a mounted directory.

I granted permission to write for everyone. :frowning::frowning:

sudo chmod o+w /srv/data/psmdb/mongodb/data-0

Another error...

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

Check the file attributes of "/data/db/ps-enty.sh", I had

-rw-------. 1 1000990000 65534 15680 Jun 23 13:09 ps-entry.sh

As observed, it seemed like the NFS shares always use the group "65534" for the created files even the current user has group "0".

I set the attribute `s` to let Linux assign the group of the current folder to the nested files.

sudo chmod g+s /srv/data/psmdb/mongodb/data-0

It worked! Phew...

-rwxr-xr-x. 1 1000990000 root 15680 Jun 23 13:16 /data/db/ps-entry.sh

Labels:

Comments

Post a Comment

Popular posts from this blog

[Snippet] CSS - Child element overlap parent

I searched from somewhere and found that a lot of people says a basic concept for implementing this feature looks like below: HTML code: <div id="parent">  <div id="child">  </div> </div> And, CSS: #parent{   position: relative;   overflow:hidden; } #child{   position: absolute;   top: -1;   right: -1px; } However, I had a lot of grand-parents in my case and the above code didn't work. Therefore, I needed an alternative. I presumed that my app uses Boostrap and AngularJs, maybe some CSS from them affects mine. I didn't know exactly the problem, but I believed when all CSS is loaded into my browser, I could completely handle it. www.tom-collinson.com I tried to create an example to investigated this problem by Fiddle . Accidentally, I just changed: position: parent; to position: static; for one of parents -> the problem is solved. Look at my code: <div class="modal-body dn-placeholder-parent-positi...
Post a Comment
Read more

[Snippet] Generate a new unique "name" string from an existing list

Suppose that we have a list of employees. Everytime, we want to add new employee into this list, the name of the employee will be generated with the following rules: - the name of the new one is set to " [originalname] 1 " - in case the name already exist, " [originalname] 2 " is used, and so on. Here is my code snippet by Javascript: var employees =[ {id: 1, name: 'name'}, {id: 2, name: 'name 1'}, {id: 3, name: 'name 2'}, {id: 5, name: 'name 4'} ]; var commonUtils = { isExistName: function(_name, _collection, _prop) { for(var i = 0; i< _collection.length; i++){ if(_collection[i][_prop].localeCompare(_name)==0){ return true; } } return false; }, generateNewName: function(_name, _collection, _prop){ var i = 1; var searching = true; while (searching) { var newName = _name+ " " + i; if (!this.isExistName(newName, _collection, _pro...
Post a Comment
Read more

DevOps for Dummies

Everyone talks about it, but not everyone knows what it is. Why DevOps? In general, whenever an organization adopts any new technology, methodology, or approach, that adoption has to be driven by a business need. Any kind of system that need rapid delivery of innovation requires DevOps (development and operations). Why? DevOps requires mechanisms to get fast feedback from all the stakeholders in the software application that's being delivered. DevOps approaches to reduce waste and rework and to shift resources to higher-value activities. DevOps aims to deliver value (of organization or project) faster and more efficiently. DevOps Capabilities The capabilities that make up DevOps are a broad set that span the software delivery life cycle. The following picture is a reference architecture which provides a template of a proven solution by using a set of preferred methods and capabilities. My Remarks Okay, that sounds cool. What does it simply mean, again? The f...
Post a Comment
Read more

Coders are NERDS | Learning English with Podcast

Let's learn three English vocabulary words based on real-life context through a humorous video about the life of software coders, especially at big tech companies when they work from home. Credit to Joma Tech. 🤓
Post a Comment
Read more

Coding Exercise, Episode 1

I have received the following exercise from an interviewer, he didn't give the name of the problem. Honestly, I have no idea how to solve this problem even I have tried to read it three times before. Since I used to be a person who always tells myself "I am not the one good at algorithms", but giving up something too soon which I feel that I didn't spend enough effort to overcome is not my way. Then, I have sticked on it for 24 hours. According to the given image on the problem, I tried to get more clues by searching. Thanks to Google, I found a similar problem on Hackerrank (attached link below). My target here was trying my best to just understand the problem and was trying to solve it accordingly by the Editorial on Hackerrank. Due to this circumstance, it turns me to love solving algorithms from now on (laugh). Check it out! Problem You are given a very organized square of size N (1-based index) and a list of S commands The i th command will follow t...
2 comments
Read more