Skip to main content

Debugging the issue of using NFS shares for PSMDB on OpenShift


I have recently been trying to use PSMDB (Percona Server for MongoDB) as an open-source and free alternative for MongoDB Enterprise Server. I encountered an issue that the pod could not be initialized successfully with Persistent Volumes using NFS shares. I got the logs from the failed pod as follow:

------

++ id -u

++ id -g

+ install -o 1000730000 -g 0 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

----

I would like to share the steps how I used for debugging. The PSMD StatefulSet was deployed onto my OpenShift 3 OKD.

Check the container mount info

Go to a pod I could see the mount info as below

mongod-data → /data/db read-write

- mongod-data: Persistent volume claim name

- /data/db: container mounted directory

Check Persistent volume binding

Go to the storage, I could know which persistent volume was bound to the corresponding persistent volume claim.

Bound to volume psmdb-mongodb-data-0

Check Persistent volume

Go cluster console > storage > persistent volumes. I had

----

nfs:

 server: files.some.domain.local

 path: /srv/data/psmdb/mongodb/data-0

---

Check NFS shares (host’s mounted directory)

Remote to server and check attributes of the mounted directory

---

ssh someuser@files.some.domain.local

cd /srv/data/psmdb/mongodb

ls -la

---

The info was:

drwxrwxr-x. 2 nfsnobody nfsnobody 6 Jun 22 08:25 data-0

Check the user and group of "nfsnobody".

id nfsnobody

The info was:

uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)

Also, I could also check by "/etc/passwd"

nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

Check container's mounted directory

Create a debug pod

---

# -c: container

# -t: attach a terminal

# --: use the entrypoint

oc debug my-cluster-name-rs0-0 -c mongo-init -t -- /bin/bash

---

Check the mounted directory attributes:

drwxrwxr-x. 2 65534 65534 25 Jun 22 08:43 db

It means the attributes are kept (the same as the host’s one). Check the files inside:

-rw-------. 1 1000920000 nfsnobody 15680 Jun 22 08:48 ps-entry.sh

Also, it was the same group as the host’s one. Only the user id was changed. It made sense. However, it seemed like the file was created but not completely set the attributes (ownership and file mode).

I checked the logged-in user of the container by executing the command "id"

uid=1000990000 gid=0(root) groups=0(root),1000990000

I ran the command from the entry point of the point ("/init-entrypoint.sh") manually within the container.

install -o "$(id -u)" -g "$(id -g)" -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

Then, I got an error:

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

The currently logged-in user within the container doesn’t have permission to change the owner. Yep! I could reproduce the issue.

Hmm...What is the root cause?

The user of the container doesn't belong to group "65534(nfsnobody)", so it could not change the owner of the file. However, I could not do the following actions:

- Change group of container's user to "65543". OpenShift grants a fixed group "0 (root)" for the user.

- Modify the entry point contains the script "install -o ..." since the templates are generated and managed by the PSMDB operator.

Hence, the only way was that I need to change the file attributes of NFS shares. I changed it to group "0".

sudo chown nfsnobody:0 /srv/data/psmdb/mongodb/data-0

Remove the pod and deploy a new one. I got another error:

install: cannot create regular file '/data/db/ps-entry.sh': Permission denied

Strange! The current user has the same group "0" of the folder already. The NFS service doesn’t allow a user without group "nsfnobody" to create a file within a mounted directory.

I granted permission to write for everyone. :frowning::frowning:

sudo chmod o+w /srv/data/psmdb/mongodb/data-0

Another error...

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

Check the file attributes of "/data/db/ps-enty.sh", I had

-rw-------. 1 1000990000 65534 15680 Jun 23 13:09 ps-entry.sh

As observed, it seemed like the NFS shares always use the group "65534" for the created files even the current user has group "0".

I set the attribute `s` to let Linux assign the group of the current folder to the nested files.

sudo chmod g+s /srv/data/psmdb/mongodb/data-0

It worked! Phew...

-rwxr-xr-x. 1 1000990000 root 15680 Jun 23 13:16 /data/db/ps-entry.sh

Labels:

Comments

Post a Comment

Popular posts from this blog

Styling Sort Icons Using Font Awesome for Primefaces' Data Table

So far, Primefaces has used image sprites for displaying the sort icons. This leads to a problem if we want to make a different style for these icons; for example, I would make the icon "arrow up" more blurry at the first time the table loading because I want to highlight the icon "arrow down". I found a way that I can replace these icons with Font Awesome icons. We will use "CSS Pseudo-classes" to achieve it. The hardest thing here is that we should handle displaying icons in different cases. There is a case both "arrow up" and "arrow down" showing and other case is only one of these icons is shown. .ui-sortable-column-icon.ui-icon.ui-icon-carat-2-n-s { background-image: none; margin-left: 5px; font-size: 1.1666em; position: relative; } .ui-sortable-column-icon.ui-icon.ui-icon-carat-2-n-s:not(.ui-icon-triangle-1-s)::before { content: "\f106"; font-family: "FontAwesome"; position: ...
Post a Comment
Read more

Building a Wizard with Chain of Responsibility Pattern

What is the Idea? We want to create a page that there are some steps and each step has its own business. Users are able to click on a step and its status could be changed. Primefaces owns a component " Wizard " but it it quite hard for us in order to apply our very specific and complicated business domain logic on each step; even we cannot click on a step of this component. We somehow are able to use the component " TabView " works with a strong back-end mechanism. A backend mechanism! what do I mean? Yes, we need it because we want to abstract the behaviors of each step otherwise we will get trouble with many events handling. Obviously, each step has some behaviors  such as "next", "back" and "switch' are the same and they are related to each other; but the business of these behaviors can be different totally. That is where the pattern "Chain of Responsibility" can be applied. Step by Step Building It! In this simple pr...
Post a Comment
Read more

Updates to the Scrum Guide - The 5 Scrum Values

This article is available at blog.scrum.org , here I just quote my favorite points and give my comment at the end of this post. Ken Schwaber and Jeff Sutherland, the creators of Scrum delivered a webinar on their latest update to the Scrum Guide .  The update was a simple one, adding the 5 values of Scrum to the Guide: These values sound easy? Well, there are many misunderstandings and common problems when applying these values. Here are some example: Value Misunderstanding Getting the value right Commitment Committing to something that you don’t understand because you are told to by your boss. Committing yourself to the team and Sprint Goal. Focus Focusing on keeping the customer happy Being focused on the sprint and its goal. Openness Telling everyone everything about all your work Highlighting when you have challenges and problems that are stopping you from success Respect Thinking you are helping the team by being a hero Helping peop...
Post a Comment
Read more

Junit - Test fails on French or German string assertion

In my previous post about building a regex to check a text without special characters but allow German and French . I met a problem that the unit test works fine on my machine using Eclipse, but it was fail when running on Jenkins' build job. Here is my test: @Test public void shouldAllowFrenchAndGermanCharacters(){ String source = "ÄäÖöÜüß áÁàÀâÂéÉèÈêÊîÎçÇ"; assertFalse(SpecialCharactersUtils.isExistSpecialCharater(source)); } Production code: public static boolean isExistNotAllowedCharacters(String source){ Pattern regex = Pattern.compile("^[a-zA-Z_0-9_ÄäÖöÜüß áÁàÀâÂéÉèÈêÊîÎçÇ]*$"); Matcher matcher = regex.matcher(source); return !matcher.matches(); } The result likes the following: Failed tests: SpecialCharactersUtilsTest.shouldAllowFrenchAndGermanCharacters:32 null A guy from stackoverflow.com says: "This is probably due to the default encoding used for your Java source files. The ö in the string literal in the J...
5 comments
Read more

Multiple Inheritance of State and Implementation

Today, I was just curious about why an enum can not extend anything else. I took a look on the Oracle document here , and I found the answer is below: "All enums implicitly extend java.lang.Enum. Because a class can only extend one parent (see Declaring Classes), the Java language does not support multiple inheritance of state (see Multiple Inheritance of State, Implementation, and Type), and therefore an enum cannot extend anything else." I have been learned of it before. But, wait a sec...! Why Java does not support multiple inheritance of state? Since I have worked with other programming languages like C++, I was able to make a class extend some other classes. The short answer is to avoid the issues of multiple inheritance of state .  I wonder if other programming languages have these below terms but Java does. Multiple inheritance of state It is the ability to inherit fields from multiple classes. There is a problem and Java avoids it. "For exa...
Post a Comment
Read more