Topic: Cannot access console and server manager

[antystein]

I've been running SME server 8.1 for sometime now without any problems, supporting about 30 windows XP workstations. Trying today to access the console on the server I realized the following:

• Cannot login as root from the console on the server as I get “login incorrect”. I can login as Admin from the same prompt on the server console, but if I try to reboot, shutdown or reconfigure nothing happens. Also nothing is displayed on the current RAID status although the 2 hard disks are working fine. The password of admin and root are the same so typing wrong password cannot be the cause.
• Cannot connect through telnet (using putty from any machine on the network), getting the error “connection refused”. I’ve been using putty a long time now, I am using saved settings to connect and nothing has been changed lately (neither on putty, firewalls, sme settings).
• Cannot access server-manager as I get “500 internal server error” I get the prompt to enter login credentials and then the error appears. The same error comes up using lynx through the server console.

The only way to shutdown the server is to press the power button of the machine which initiates the shutdown sequence normally.

During the shutdown and boot sequences I get no errors at all.

Despite all the above the server is working fine, stations can logon to the domain normally, and the file sharing and ftp also work fine.

I usually do maintenance checks on the server 1 -2 times per week, so I know for sure that the above problems were not present a week ago. Actually I used all 3 of the above 5 days ago.

I have been searching for solutions for the above as individual problems, but unfortunately all the solutions I have found suggest typing commands on the command prompt which at the moment it’s impossible.

Is there any chance that the above conditions are related somehow and break down to a single cause? And if so is it possible to tackle this problem through other means (for example appending commands through the grub boot settings, or using some other way to execute commands on the server)?

thanks

[CharlieBrady]

The password of admin and root are the same so typing wrong password cannot be the cause.

Should be the same, but won't be the same, if hackers have got into your system and changed root but not admin. That could happen if you had enabled remote ssh access and don't have a very strong password.

Cannot connect through telnet (using putty from any machine on the network), getting the error “connection refused”.

You mean SSH, not telnet, I hope. "connection refused" means that sshd is not running, either because it has been disabled, or because it is broken - e.g. by filesystem corruption.

Cannot access server-manager as I get “500 internal server error” I get the prompt to enter login credentials and then the error appears.

That likely means a hardware problem or filesystem corruption. There should be some details of the error in /var/log/httpd/admin_error_log.

And if so is it possible to tackle this problem through other means (for example appending commands through the grub boot settings, or using some other way to execute commands on the server)?

You can boot in single user mode, at the grub prompt, or you can boot in rescue mode from the SME server CD, and look around. Nothing simple and straightforward, I'm afraid.

[janet]

antystein

Possibly your server has been hacked.
You mention using ftp which is an insecure protocol.
Is the server a web server for web apps eg Wordpress, Joomla etc ?
Have you kept your web apps up to date (particularly anything written using php) ?
Have you kept your server fully up to date with latest packages ?

You do not mention what recent changes you have made, please provide details.

You can insert the install CD for sme 8.0 & boot up your system in Rescue mode.
You should be able to then fix the root password.
Refer to FAQ & Howto documentation for details, links at top of Forum.

If you have been hacked, just fixing the password may not be enough, you need to ascertain how it happened & whether any backdoors have been installed etc.
Usually it's best to Restore from a known good backup from before the hack.

Of course your problem may be something lesser than having been hacked, but we really need to hear a lot more about the history of the server, & what it is you have done in these twice weekly maintenance sessions, in order to help determine how your problem came to be.
eg exactly what changes did you make in the last few days ?

Have you looked at log files ?
You should be able to do this in Rescue mode.

[antystein]

Hello,

Thank you for your replies. My first thought was that the server could have been hacked but this server is used only locally there is no access to it outside the LAN. There are not any web services running and the FTP is also used locally in rare cases. There is no remote access through SSH to the console only local connections accepted. Actually when i installed this server i took extra care to ensure that it could not be accessed remotely, first because it wasn't necessary (web services run on other servers) and second because at the time of installation I had no experience on the operation of this particular system. Of course I could have missed something!

I haven't made any changes recently, during the routine maintenance only the good operation of the server is checked including some SMART checks.

This server is running in a school environment so there was very small - if any - activity during the Christmas holidays. There were not any changes in the latest two weeks, not even updates. Actually i was about to install updates today...

It is used mostly for local file sharing and windows domain control. Users do not have roaming profiles and the operation is kept as basic as possible. I also haven't installed any extensions on the server. The only modification is a couple of locally shared folders through samba. (file /etc/e-smith/templetes-custom/61Profilesshare was modified when the server was installed, i.e. two years ago! and not in the latest days for sure)

I am using the server for two years, so my experience on this particular system has a limited extend.

I will try rescue mode in order to change the root password and look at the logs at first, and i will post here my findings.

Also considering file corruption as the case, would I be able to run a disk error check through rescue mode?

thanks

[janet]

antystein

Connected workstations may have introduced a virus/hackers to the server.

I think you can run smart checks from rescue mode, try.
http://wiki.contribs.org/Monitor_Disk_Health

Alternatively/as well, download the UBCD (Ultimate Boot CD) & run manufacturers diagnostic tests against your server hard disks, check them all.

Unless you tell us the exact details of your server configuration (from server manager & console) & run some diagnostic tests, then it is difficult to comment about whether you configured the server & network correctly or not.

Run these commands (when you can)
/sbin/e-smith/audittools/newrpms
/sbin/e-smith/audittools/templates

[larieu]

I suggest you to use the classical reset password ( enter into single mode)
and type a new password

http://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Reset_the_root_and_admin_password

after this try again with the new password to connect to console / ssh  / server-manager
if it was hacked this could help gaining back access

[antystein]

I suggest you to use the classical reset password ( enter into single mode)
and type a new password

http://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Reset_the_root_and_admin_password

after this try again with the new password to connect to console / ssh  / server-manager
if it was hacked this could help gaining back access

I tried the above procedure and changed the root password to something very simple to avoid typos. Unfortunately I still get the same error: "Login incorrect" trying to login through the server console. Also the rest problems reported are still present. I haven't exported any logs or system information yet because it is rush hour and I cannot keep the server in single user mode for a long time without warning the users beforehand.

I will post more technical information when possible.

[CharlieBrady]

I haven't exported any logs or system information yet because it is rush hour and I cannot keep the server in single user mode for a long time without warning the users beforehand.

The longer you keep this system running, the more likely it will be that you lose all data from it. Keep that in mind when deciding the relative priority of keeping it running or taking it out of service to fix it.

[antystein]

I got some error logs for server-manager any ideas?

var/log/httpd/admin_error_log

[Thu Jan 09 18:29:32 2014] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 09 18:29:32 2014] [notice] Digest: done
[Thu Jan 09 18:29:32 2014] [notice] Apache configured -- resuming normal operations
[Thu Jan 09 18:30:10 2014] [error] [client 127.0.0.1] fileparse(): need a valid pathname at /usr/lib/perl5/vendor_perl/5.8.8/CGI/Persistent.pm line 30
[Thu Jan 09 18:30:10 2014] [error] [client 127.0.0.1] Premature end of script headers: index.cgi
[Thu Jan 09 18:30:22 2014] [notice] caught SIGTERM, shutting down

var/log/httpd/error_log server name replaced by asterisks

[Thu Jan 09 18:29:33 2014] [notice] SSL FIPS mode disabled
[Thu Jan 09 18:29:33 2014] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jan 09 18:29:33 2014] [warn] RSA server certificate CommonName (CN) `*****************' does NOT match server name!?
[Thu Jan 09 18:29:33 2014] [notice] Digest: generating secret for digest authentication ...
[Thu Jan 09 18:29:33 2014] [notice] Digest: done
[Thu Jan 09 18:29:39 2014] [notice] SSL FIPS mode disabled
[Thu Jan 09 18:29:39 2014] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jan 09 18:29:39 2014] [warn] RSA server certificate CommonName (CN) `*****************' does NOT match server name!?
[Thu Jan 09 18:29:39 2014] [notice] Apache configured -- resuming normal operations
[Thu Jan 09 18:30:07 2014] [error] [client 192.168.1.50] File does not exist: /home/e-smith/files/ibays/Primary/html/favicon.ico
[Thu Jan 09 18:30:07 2014] [error] [client 192.168.1.50] File does not exist: /home/e-smith/files/ibays/Primary/html/favicon.ico
[Thu Jan 09 18:30:24 2014] [notice] caught SIGTERM, shutting down

The above were produced by a single attempt to connect to server-manager, getting a 500 internal server error.

[CharlieBrady]

Doing a google search for:

site:contribs.org "fileparse(): need a valid pathname"

will find you various reports similar to yours. e.g.

http://bugs.contribs.org/show_bug.cgi?id=6851

[CharlieBrady]

Next time you log in, collect 'rpm -qa --last | head -12'. That'll identify any recently updated packages.

'rpm -Va' will capture any damaged/incomplete rpms - but will also show lots of configuration files, which you of course expect to be modified.

[antystein]

Success!

I found the same exact bug report (http://bugs.contribs.org/show_bug.cgi?id=6851) yesterday searching for "Internal Server Error server-manager sme" but I was not sure if its the same problem and if it is safe to execute the commands not having any error logs in hand. I avoid using remedies not knowing enough information.

thanks to CharlieBrady for reminding me the bug report I did some searching and I understood the operation of the following commands found on the bug report,

for f in $(rpm -qa); do echo $f; rpm --setugids $f; done
for f in $(rpm -qa); do echo $f; rpm --setperms $f; done

and decided to execute them. I started the server in single user mode (following http://wiki.contribs.org/SME_Server:Documentation:FAQ:Section01#Reset_the_root_and_admin_password procedure - of course without changing passwords again) and executed the commands. It took about 20 minutes for the two operations to complete and now two out of tree problems i had are fixed!

The server-manager works again, and I can now login as root normally on the server shell. The only problem remaining is that I can't connect to the secure shell with putty. I am still getting the connection refused error. I tried to disable and re-enable the secure shell access through server-manager, but the problem persists.

I did focused on the other problems till now so I haven't done any searching for the secure shell problem. If you have any idea would be very helpful, in the meantime I will try to find a solution and report back. I will also attempt to locate what caused the problem in the first place - still if you have any clue would help!

I thank you all for your replies and your valuable time. Every little clue each one of you gave me helped me to solve the problem to this point.

[CharlieBrady]

and executed the commands. It took about 20 minutes for the two operations to complete and now two out of tree problems i had are fixed!

That's good, but unfortunately you don't know what permissions or ownerships were wrong, so you can't work out how and why they were wrong, or know what you need to do to prevent it happening again. Unless you kept the output of 'rpm -Va'.

For your ssh problem, look in /var/log/sshd/* and /var/log/secure* for clues. Then do:

cd /service/sshd
sv status .
sv d .
./run

then ^C. If sshd does not start running, you will see an error message stating what the problem is.

[antystein]

Fixed!

doing what CharlieBrady suggested returned the following

@4000000052ce9a6d1ad0c8c4 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@4000000052ce9a6d1ad0ccac @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@4000000052ce9a6d1ad1e9d4 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@4000000052ce9a6d1ad1e9d4 Permissions 0777 for '/etc/ssh/ssh_host_rsa_key' are too open.
@4000000052ce9a6d1ad1edbc It is recommended that your private key files are NOT accessible by others.
@4000000052ce9a6d1ad1f1a4 This private key will be ignored.
@4000000052ce9a6d1ad1f1a4 bad permissions: ignore key: /etc/ssh/ssh_host_rsa_key
@4000000052ce9a6d1ad1f58c Could not load host key: /etc/ssh/ssh_host_rsa_key
@4000000052ce9a6d1ad23024 Disabling protocol version 2. Could not load host key
@4000000052ce9a6d1ad2340c sshd: no hostkeys available -- exiting.

I found the original permissions for the files in /etc/ssh/ and restored them by doing:

#chmod 600 moduli
#chmod 644 ssh_config
#chmod 644 ssh_host_dsa_key.pub
#chmod 644 ssh_host_key.pub
#chmod 644 ssh_host_rsa_key.pub
#chmod 600 ssh_host_dsa_key
#chmod 600 ssh_host_key
#chmod 600 ssh_host_rsa_key
#chmod 640 sshd_config

sshd works now!

It seems that somehow all (or many) permissions on the server were changed to 777 resulting on several services not to work. Based on the fact that the server during the period that this happened, which is actually 3 days, was not in use (meaning that no users were connected etc, i.e. it was just running "idle") I don't have any clues of what could have caused it. Looking now at most of the access logs for that period I can't see any strange activity, that is, evidence of hackers. I will keep an eye on that and if I locate any possible cause I will post it here.

Thank you again for your valuable help.

[stephdl]

Fixed!

Thank you again for your valuable help.

Maybe you can ask to your customer to make a  donation http://wiki.contribs.org/Donate
as suggested at the top of forums page,sme is free to use but not free to build

If you want to participate to the adventure, the wiki and bugzilla are good places to play.

see you soon