Skip to main content

Debugging the issue of using NFS shares for PSMDB on OpenShift


I have recently been trying to use PSMDB (Percona Server for MongoDB) as an open-source and free alternative for MongoDB Enterprise Server. I encountered an issue that the pod could not be initialized successfully with Persistent Volumes using NFS shares. I got the logs from the failed pod as follow:

------

++ id -u

++ id -g

+ install -o 1000730000 -g 0 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

----

I would like to share the steps how I used for debugging. The PSMD StatefulSet was deployed onto my OpenShift 3 OKD.

Check the container mount info

Go to a pod I could see the mount info as below

mongod-data → /data/db read-write

- mongod-data: Persistent volume claim name

- /data/db: container mounted directory

Check Persistent volume binding

Go to the storage, I could know which persistent volume was bound to the corresponding persistent volume claim.

Bound to volume psmdb-mongodb-data-0

Check Persistent volume

Go cluster console > storage > persistent volumes. I had

----

nfs:

 server: files.some.domain.local

 path: /srv/data/psmdb/mongodb/data-0

---

Check NFS shares (host’s mounted directory)

Remote to server and check attributes of the mounted directory

---

ssh someuser@files.some.domain.local

cd /srv/data/psmdb/mongodb

ls -la

---

The info was:

drwxrwxr-x. 2 nfsnobody nfsnobody 6 Jun 22 08:25 data-0

Check the user and group of "nfsnobody".

id nfsnobody

The info was:

uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)

Also, I could also check by "/etc/passwd"

nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

Check container's mounted directory

Create a debug pod

---

# -c: container

# -t: attach a terminal

# --: use the entrypoint

oc debug my-cluster-name-rs0-0 -c mongo-init -t -- /bin/bash

---

Check the mounted directory attributes:

drwxrwxr-x. 2 65534 65534 25 Jun 22 08:43 db

It means the attributes are kept (the same as the host’s one). Check the files inside:

-rw-------. 1 1000920000 nfsnobody 15680 Jun 22 08:48 ps-entry.sh

Also, it was the same group as the host’s one. Only the user id was changed. It made sense. However, it seemed like the file was created but not completely set the attributes (ownership and file mode).

I checked the logged-in user of the container by executing the command "id"

uid=1000990000 gid=0(root) groups=0(root),1000990000

I ran the command from the entry point of the point ("/init-entrypoint.sh") manually within the container.

install -o "$(id -u)" -g "$(id -g)" -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

Then, I got an error:

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

The currently logged-in user within the container doesn’t have permission to change the owner. Yep! I could reproduce the issue.

Hmm...What is the root cause?

The user of the container doesn't belong to group "65534(nfsnobody)", so it could not change the owner of the file. However, I could not do the following actions:

- Change group of container's user to "65543". OpenShift grants a fixed group "0 (root)" for the user.

- Modify the entry point contains the script "install -o ..." since the templates are generated and managed by the PSMDB operator.

Hence, the only way was that I need to change the file attributes of NFS shares. I changed it to group "0".

sudo chown nfsnobody:0 /srv/data/psmdb/mongodb/data-0

Remove the pod and deploy a new one. I got another error:

install: cannot create regular file '/data/db/ps-entry.sh': Permission denied

Strange! The current user has the same group "0" of the folder already. The NFS service doesn’t allow a user without group "nsfnobody" to create a file within a mounted directory.

I granted permission to write for everyone. :frowning::frowning:

sudo chmod o+w /srv/data/psmdb/mongodb/data-0

Another error...

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

Check the file attributes of "/data/db/ps-enty.sh", I had

-rw-------. 1 1000990000 65534 15680 Jun 23 13:09 ps-entry.sh

As observed, it seemed like the NFS shares always use the group "65534" for the created files even the current user has group "0".

I set the attribute `s` to let Linux assign the group of the current folder to the nested files.

sudo chmod g+s /srv/data/psmdb/mongodb/data-0

It worked! Phew...

-rwxr-xr-x. 1 1000990000 root 15680 Jun 23 13:16 /data/db/ps-entry.sh

Labels:

Comments

Post a Comment

Popular posts from this blog

Regex - Check a text without special characters but German, French

Special characters such as square brackets ([ ]) can cause an exception " java.util.regex.PatternSyntaxException " or something like this if we don't handle them correctly. I had met this issue. In my case, my customers want our application should allow some characters in German and French even not allow some special characters. The solution is that we limit the allowed characters by showing the validation message on GUI. For an instance, the message looks like the following: "This field can't contain any special characters; only letters, numbers, underscores (_), spaces and single quotes (') are allowed." I used Regular Expression to check it. For entering Germany and French, I actually don't have this type of keyboard, so I referred these sites: * German characters: http://german.typeit.org/ * French characters: http://french.typeit.org/ Here is my code: package vn.nvanhuong.practice; import java.util.regex.Matcher; import java.util
Post a Comment
Read more

JSF 2 - Dynamically manipulating the component tree with system events

Let's suppose we want to modify the metadata (attributes)  of elements such as render , requried , maxlength but we do not define in JSF tags. The manipulating components can be conducted in Drools  files, for example. How could we do? I think that is what we need to change something of component tree during JSF life-cycle. JSF supports event handling throughout the JSF life-cycle. In this post, I use two events: postAddToView for scanning components tree and preRenderView for manipulating the meta of components before rendering to GUI. I modified my own project from previous post for this example. This is my first further JSF trying out with the project as I said before. :) We define the tags f:event below the form - a container component of the components which we want to work on. The valid values for the attribute type for f:event can be found from tag library document  of JSF 2. <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml" xmlns:
Post a Comment
Read more

The HelloWorld example of JSF 2.2 with Myfaces

I just did by myself create a very simple app "HelloWorld" of JSF 2.2 with a concrete implementation Myfaces that we can use it later on for our further JSF trying out. I attached the source code link at the end part. Just follow these steps below: 1. Create a Maven project in Eclipse (Kepler) with a simple Java web application archetype "maven-archetype-webapp". Maven should be the best choice for managing the dependencies , so far. JSF is a web framework that is the reason why I chose the mentioned archetype for my example. 2. Import dependencies for JSF implementation - Myfaces (v2.2.10) into file pom.xml . The following code that is easy to find from  http://mvnrepository.com/  with key words "myfaces". <dependency> <groupId>org.apache.myfaces.core</groupId> <artifactId>myfaces-api</artifactId> <version>2.2.10</version> </dependency> <dependency> <groupId>org.apache.myfaces.core<
Post a Comment
Read more

[Snippet] CSS - Child element overlap parent

I searched from somewhere and found that a lot of people says a basic concept for implementing this feature looks like below: HTML code: <div id="parent">  <div id="child">  </div> </div> And, CSS: #parent{   position: relative;   overflow:hidden; } #child{   position: absolute;   top: -1;   right: -1px; } However, I had a lot of grand-parents in my case and the above code didn't work. Therefore, I needed an alternative. I presumed that my app uses Boostrap and AngularJs, maybe some CSS from them affects mine. I didn't know exactly the problem, but I believed when all CSS is loaded into my browser, I could completely handle it. www.tom-collinson.com I tried to create an example to investigated this problem by Fiddle . Accidentally, I just changed: position: parent; to position: static; for one of parents -> the problem is solved. Look at my code: <div class="modal-body dn-placeholder-parent-position&quo
Post a Comment
Read more

PSMDB - A MongoDB alternative for having Encryption At Rest

Encryption is the most popular tool for securing data both in transit and at rest. - For protecting data in transit, we can configure to use the TLS connection - For protecting data at rest, we can use Percona Server for MongoDB (PSMDB), an open-source alternative for MongoDB Enterprise. License PSMDB Docker images follow the SSPL license. Therefore, it is not a problem when I only have my containers deployed in on-premises environments. Running MongoDB Replication on OpenShift I have successfully installed the replication by following the guide Install Percona Server for MongoDB on OpenShift . In order to make it work properly with my needs, I disabled some features from the default deployment. See the detail in this change Basically, I needed to create a CRD (Custom Resource Definition) to let OpenShift/Kubernetes what PSMDB is. Then, I deployed the Operator pod. Finally, I deployed the PSMDB StatefulSet. I used NFS shares for Persistent Volumes. Create CRD for PSMDB 2 git clone http
Post a Comment
Read more