Skip to main content

Debugging the issue of using NFS shares for PSMDB on OpenShift


I have recently been trying to use PSMDB (Percona Server for MongoDB) as an open-source and free alternative for MongoDB Enterprise Server. I encountered an issue that the pod could not be initialized successfully with Persistent Volumes using NFS shares. I got the logs from the failed pod as follow:

------

++ id -u

++ id -g

+ install -o 1000730000 -g 0 -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

----

I would like to share the steps how I used for debugging. The PSMD StatefulSet was deployed onto my OpenShift 3 OKD.

Check the container mount info

Go to a pod I could see the mount info as below

mongod-data → /data/db read-write

- mongod-data: Persistent volume claim name

- /data/db: container mounted directory

Check Persistent volume binding

Go to the storage, I could know which persistent volume was bound to the corresponding persistent volume claim.

Bound to volume psmdb-mongodb-data-0

Check Persistent volume

Go cluster console > storage > persistent volumes. I had

----

nfs:

 server: files.some.domain.local

 path: /srv/data/psmdb/mongodb/data-0

---

Check NFS shares (host’s mounted directory)

Remote to server and check attributes of the mounted directory

---

ssh someuser@files.some.domain.local

cd /srv/data/psmdb/mongodb

ls -la

---

The info was:

drwxrwxr-x. 2 nfsnobody nfsnobody 6 Jun 22 08:25 data-0

Check the user and group of "nfsnobody".

id nfsnobody

The info was:

uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody)

Also, I could also check by "/etc/passwd"

nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

Check container's mounted directory

Create a debug pod

---

# -c: container

# -t: attach a terminal

# --: use the entrypoint

oc debug my-cluster-name-rs0-0 -c mongo-init -t -- /bin/bash

---

Check the mounted directory attributes:

drwxrwxr-x. 2 65534 65534 25 Jun 22 08:43 db

It means the attributes are kept (the same as the host’s one). Check the files inside:

-rw-------. 1 1000920000 nfsnobody 15680 Jun 22 08:48 ps-entry.sh

Also, it was the same group as the host’s one. Only the user id was changed. It made sense. However, it seemed like the file was created but not completely set the attributes (ownership and file mode).

I checked the logged-in user of the container by executing the command "id"

uid=1000990000 gid=0(root) groups=0(root),1000990000

I ran the command from the entry point of the point ("/init-entrypoint.sh") manually within the container.

install -o "$(id -u)" -g "$(id -g)" -m 0755 -D /ps-entry.sh /data/db/ps-entry.sh

Then, I got an error:

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

The currently logged-in user within the container doesn’t have permission to change the owner. Yep! I could reproduce the issue.

Hmm...What is the root cause?

The user of the container doesn't belong to group "65534(nfsnobody)", so it could not change the owner of the file. However, I could not do the following actions:

- Change group of container's user to "65543". OpenShift grants a fixed group "0 (root)" for the user.

- Modify the entry point contains the script "install -o ..." since the templates are generated and managed by the PSMDB operator.

Hence, the only way was that I need to change the file attributes of NFS shares. I changed it to group "0".

sudo chown nfsnobody:0 /srv/data/psmdb/mongodb/data-0

Remove the pod and deploy a new one. I got another error:

install: cannot create regular file '/data/db/ps-entry.sh': Permission denied

Strange! The current user has the same group "0" of the folder already. The NFS service doesn’t allow a user without group "nsfnobody" to create a file within a mounted directory.

I granted permission to write for everyone. :frowning::frowning:

sudo chmod o+w /srv/data/psmdb/mongodb/data-0

Another error...

install: cannot change ownership of '/data/db/ps-entry.sh': Operation not permitted

Check the file attributes of "/data/db/ps-enty.sh", I had

-rw-------. 1 1000990000 65534 15680 Jun 23 13:09 ps-entry.sh

As observed, it seemed like the NFS shares always use the group "65534" for the created files even the current user has group "0".

I set the attribute `s` to let Linux assign the group of the current folder to the nested files.

sudo chmod g+s /srv/data/psmdb/mongodb/data-0

It worked! Phew...

-rwxr-xr-x. 1 1000990000 root 15680 Jun 23 13:16 /data/db/ps-entry.sh

Labels:

Comments

Post a Comment

Popular posts from this blog

[Snippet] CSS - Child element overlap parent

I searched from somewhere and found that a lot of people says a basic concept for implementing this feature looks like below: HTML code: <div id="parent">  <div id="child">  </div> </div> And, CSS: #parent{   position: relative;   overflow:hidden; } #child{   position: absolute;   top: -1;   right: -1px; } However, I had a lot of grand-parents in my case and the above code didn't work. Therefore, I needed an alternative. I presumed that my app uses Boostrap and AngularJs, maybe some CSS from them affects mine. I didn't know exactly the problem, but I believed when all CSS is loaded into my browser, I could completely handle it. www.tom-collinson.com I tried to create an example to investigated this problem by Fiddle . Accidentally, I just changed: position: parent; to position: static; for one of parents -> the problem is solved. Look at my code: <div class="modal-body dn-placeholder-parent-positi...
Post a Comment
Read more

What the heck is Meteor DDP?

I was using Meteor for my messenger project. I was so curious about the real time connection. I wanted to know how exactly this mechanism works. In this post, I will go through the DDP Specification, an overview of WebSocket, and a simple demo about how to subscribe a publication of Rocket.Chat (containing a DDP server) from an external webpage. At a glance, I knew that Meteor invented a protocol called DDP which uses for handling real time connection. So then, what is DDP? "DDP (Distributed Data Protocol) is the stateful WebSocket protocol that Meteor uses to communicate between the client and the server." [1] All right! Why does DDP matter? "DDP is a standard way to solve the biggest problem facing client-side JavaScript developers: querying a server-side database, sending the results down to the client, and then pushing changes to the client whenever anything changes in the database" . [2] In order to understand deeply the protocol, I decided ...
Post a Comment
Read more

MS SQL Server Views

"Creates a virtual table whose contents (columns and rows) are defined by a query. Use this statement to create a view of the data in one or more tables in the database. For example, a view can be used for the following purposes: - To focus, simplify, and customize the perception each user has of the database. - As a security mechanism by allowing users to access data through the view, without granting the users permissions to directly access the underlying base tables. - To provide a backward compatible interface to emulate a table whose schema has changed." [1] Beside that, our team used view in order to improve the performance of our web apps when a database has a very complicated relationship between its tables by using ORM Frameworks such as Hibernate. Example code: --create CREATE VIEW placeholders AS SELECT EMPKEY AS empkey, CONNUMB AS connumb, EMPNBR AS empNbr, ACEEMPN AS empFirstName, ACEEMPFN AS empLastName, EMPNAM AS empFullName, ...
Post a Comment
Read more

Why Business Rules Engine Matter

"A good DSL minimizes the 'communication gap' between a domain concept and the code that implements it" - Robert C. Martin A Use Case It looks like adopting a new technology is always driven by a business need. The organizations, such as banks, have their own business processes (such as data gathering and document management). These processes are different from others but usually the differences are not big. How do we build a product which can be reused the similar features but still be adaptable with the specific requirements of each bank? Then, the answer is we should have a library/framework. There is an idea that we can implement this library/framework by using a business rules engine. Which has the following benefits: Be able to modify implementation of business domain at runtime (such as XML, CSV) S ource code is more readable for both developers and domain experts. Therefore, source code can be consider it as a document Since Business Rules Engine...
Post a Comment
Read more

Why Functional Programming Matter

What issues do we concern when implementing and maintaining systems? One of the most concern is debugging during maintenance: "this code crashed because it observed some unexpected value." Then, it turns out that the ideas of  no side effects  and  immutability , which functional programming promotes, can help. Shared mutable data is the root cause Shared mutable data are read and updated by more than one of the methods. Share mutable data structures make it harder to track changes in different parts of your program. An immutable object is an object that can't change its state after it's instantiated so it can't be affected by the actions of a function. It would be a dream to maintain because we wouldn't have any bad surprises about some object somewhere that unexpectedly modifies a data structure. A new thinking: Declarative programming There are two ways thinking about implementing a system by writing a program. - Imperative programming: has...
Post a Comment
Read more