[BlueOnyx:14298] Re: updatedb strange behavior

[Michael Stauber]

Hi Dirk,

> I am wondering why since 21.01., 22.00 (german time) 
> this process is running at 100% at several (!)
> servers and why I cannot see the process with ps.
> 
> Is nobody experiencing the same issue?

I just checked my servers and a couple of client servers that I have
under support contract. None of them is exhibiting that behavior.

Just to clarify:

The process "updatedb" shows in "top", but not in "ps"?

THAT is not good. Please do this:

1.) Note down the PID of the process in question.

2.) Verify the integrity of your PS command via "rpm -V procps". It
should come back blank without flagging any of the files that this RPM
contains.

3.) Find out which path "updatedb" usually has:

[root at 5108r /]# which updatedb
/usr/bin/updatedb

4.) Use "lsof" to see which files, pipes, devices and/or sockets are
held open by the process with the PID you noted:

lsof -nl|grep <PID>

Does it call /usr/bin/updatedb, too? Or is it another "updatedb" located
somewhere else?

"lsof" is very useful, as it can shed some light on the "legality" of
the process in question. If it hides from "ps", it is suspicious.

5.) If lsof says it's the same /usr/bin/updatedb, we verify its
integrity as well:

[root at 5108r /]# rpm -q --whatprovides /usr/bin/updatedb
mlocate-0.22.2-4.el6.x86_64
[root at 5108r /]# rpm -V mlocate-0.22.2-4.el6.x86_64
[root at 5108r /]#

That looks good.

6.) You also might want to check the directory /proc/<PID>/

Processes can hide from commands such as "ps", but the presence in
/proc/<PID>/ can't be masked that easily. The contends of these process
specific directories shed a lot of light on the nature of processes. For
examle the "exe" symbolic link on there will point to the exact binary
that was called, including the full path. One can also check the
processes environment and stack there, among some other things.

Lastly, it could be that it's hanging (and not showing in "ps") due to a
kernel error. I've seen weird segfaults which can cause something like
that, but it wouldn't happen at the same time on different boxes.

If you need any help with this or want me to take a look, just say so.
You have my number. :-)

-- 
With best regards

Michael Stauber