Topic: Loosing internet connection after post-upgrade of server

[ReetP]

I have made a clean SME10 installation and it works fine. The problem comes after the USB disk restore.

So you have to do the restore and then work out why it doesn't work at that point without starting to hack stuff about.

Watch the logs as you restore. Check them all after reboot. What do the various dns logs say?

Also the old SME9 server is quite standard

'Quite'.

As I said above please post the audittools output so we can see what your v9 has.

I have not any settings in the Linksys router that will disallow dns traffic to the SME10. Only simple NAT forward settings to the SME9 server. It also work if I do the admin server console hack so in my view it can not be the router that is to blame

Something is blocking it. We haven't worked out what yet. So best look at facts, not opinions and guesses.

[TerryF]

My apoligies, went to quote beast and inadvertently deleted the post, FML..here it is

Quote from: TerryF on July 27, 2021, 01:33:54 AM
You can do a restore from the console after the sme10 is fully installed and operational, reconfig/reboot required after restore, beware of custom templates, ip etc will be reset as per sme9 settings end quote

I tried this with the same result that DNS is not working. E.g. did a SME10 install, yum upgrade and testing everything and later I made the restore - same end result.

Now I will try with a different backup - just to make sure this is not the problem

[TerryF]

I tried this with the same result that DNS is not working. E.g. did a SME10 install, yum upgrade and testing everything and later I made the restore - same end result.

This is what I was trying to ask..

After the install and yum update you had connectivity, system is working as expected, after the restore it is lost.

Sorry again for my ham fingered keyboard skills..

[beast]

This is what I was trying to ask..

After the install and yum update you had connectivity, system is working as expected, after the restore it is lost.

Sorry again for my ham fingered keyboard skills..

Yes

[beast]

Like to add that when I have a working system after restore the system is still working right after the "signal-event post-upgrade". It is not working after the "signal-event reboot" have completed

[beast]

I have made some tar archives of the log folder during the process.

I made some notes about when in the process I made the archive but is unable to find it now

As far as I remember log2 is when the system is not working

https://drive.google.com/drive/folders/1Seg8N-df2-MB6fkAbWEQfjA9hTrZfG_h?usp=sharing

I will have to repeat the process when it is working and when it is not and make a archive again

I have tried to remove all custom templates to begin with.

[beast]

Have now made 2 more files at the same location.

https://drive.google.com/drive/folders/1Seg8N-df2-MB6fkAbWEQfjA9hTrZfG_h?usp=sharing

log_working.tar (start condition after the dns hack in admin server console)
log_notworking.tar (after a "signal-event post-upgrade; signal-event reboot")

[TerryF]

Thansk mate, logs essential in trying to find the why..time for some reading

At least its definitive that it is something being restored from the sme9 setup that is creating the issue the sme10.

Guarantee it will be something small,.straws, we all clutch at straws when situations like this arise.

Above you detailed -

I have not any settings in the Linksys router that will disallow dns traffic to the SME10. Only simple NAT forward settings to the SME9 server. It also work if I do the admin server console hack so in my view it can not be the router that is to blame.

Have you tried setup of NAT forwarding for the IP to the new sme10? Like I said just straws..

We'll get there, just a matter of finding the right rabbit hole

[beast]

Have now tried with a new backup on a new HDD. It has to be < 2 Tb or it does not work. But the result is the same

I was thinking about doing this upgrade in a different way with PHP and rsync:

Code: [Select]

$returnstr = shell_exec("signal-event pre-backup");

$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /home/e-smith/* root@x.x.x.x::beastserver/home/e-smith/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/e-smith/templates-custom/* root@x.x.x.x::beastserver/etc/e-smith/templates-custom/ --ignore-missing-args");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/e-smith/templates-user-custom/* root@x.x.x.x::beastserver/etc/e-smith/templates-user-custom/ --ignore-missing-args");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/ssh/* root@x.x.x.x::beastserver/etc/ssh/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /root/* root@x.x.x.x::beastserver/root/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/sudoers root@x.x.x.x::beastserver/etc/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/passwd root@x.x.x.x::beastserver/etc/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/shadow root@x.x.x.x::beastserver/etc/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/group root@x.x.x.x::beastserver/etc/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/gshadow root@x.x.x.x::beastserver/etc/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/dehydrated root@x.x.x.x::beastserver/etc/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/samba/secrets.tdb root@x.x.x.x::beastserver/etc/samba/");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/samba/smbpasswd root@x.x.x.x::beastserver/etc/samba/");

$returnstr .= shell_exec("signal-event post-backup");
Destination is a USB disk and this will make it more easy for me to find out where the problem is located as I may be able to upgrade different parts of the system. But I do not know the parts I have to replace at the same time to have a working system?

[ReetP]

I think I said this to you earlier.

Make sure there are no custom templates anywhere.

So why have you got this?

Code: [Select]

$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/e-smith/templates-custom/* root@x.x.x.x::beastserver/etc/e-smith/templates-custom/ --ignore-missing-args");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/e-smith/templates-user-custom/* root@x.x.x.x::beastserver/etc/e-smith/templates-user-custom/ --ignore-missing-args");

Next, I can see an issue in your logs but can't remember the solution - I think it is to do with missing information in ldap.

e.g.

Jul 31 16:24:55 beastserver2 esmith::event[4127]: You are about to be asked to enter information that will be incorporated
Jul 31 16:24:55 beastserver2 esmith::event[4127]: into your certificate request.
Jul 31 16:24:55 beastserver2 esmith::event[4127]: What you are about to enter is what is called a Distinguished Name or a DN.
Jul 31 16:24:55 beastserver2 esmith::event[4127]: There are quite a few fields but you can leave some blank
Jul 31 16:24:55 beastserver2 esmith::event[4127]: For some fields there will be a default value,
Jul 31 16:24:55 beastserver2 esmith::event[4127]: If you enter '.', the field will be left blank.
Jul 31 16:24:55 beastserver2 esmith::event[4127]: -----
Jul 31 16:24:55 beastserver2 esmith::event[4127]: Country Name (2 letter code) [XX]:State or Province Name (full name) []:Locality Name (eg, city) [Default City]:Organization Name (eg, company) [Default Company Ltd]:Organizational Unit Name (eg, section) []:Common Name (eg, your name or your server's hostname) []:Email Address []:expanding /var/service/sqpsmtpd/runenv

Please show:

Code: [Select]

config show ldap
Also I can see you are running owncloud..... and you have some issues in there eg

Jul 30 08:05:53 test console_restore: tar: home/e-smith/files/ibays/owncloud/html/data/......lnk: implausibly old time stamp 1970-01-01 01:00:00

You ought to go and fix some of the files - see your logs for which ones.

We also did ask you to try and get the output from newrpms and templates. This is why - so we know what else you have squirrelled away there that might be causing issues.

Ahhh, and then this....

Jul 30 14:29:34 test esmith::event[2651]: 
Jul 30 14:29:34 test esmith::event[2651]: Your ENVIRONMENT IS INSANE! Users in files "/etc/passwd" and "/etc/shadow are diffrent!!!
Jul 30 14:29:34 test esmith::event[2651]: I'll continue, but it is YOUR RISK! You'll probably go into BIG troubles!
Jul 30 14:29:34 test esmith::event[2651]: 
Jul 30 14:29:34 test esmith::event[2651]:  at /usr/share/perl5/vendor_perl/esmith/Backup.pm line 374.
Jul 30 14:29:34 test esmith::event[2651]: 
Jul 30 14:29:39 test esmith::event[2651]: 
Jul 30 14:29:39 test esmith::event[2651]: Your ENVIRONMENT IS INSANE! Users in files "/etc/passwd" and "/etc/shadow are diffrent!!!
Jul 30 14:29:39 test esmith::event[2651]: I'll continue, but it is YOUR RISK! You'll probably go into BIG troubles!
Jul 30 14:29:39 test esmith::event[2651]: 
Jul 30 14:29:39 test esmith::event[2651]:  at /usr/share/perl5/vendor_perl/esmith/Backup.pm line 382.
Jul 30 14:29:39 test esmith::event[2651]: 

And then this (a small section as there is a lot more)

Jul 30 14:29:54 test esmith::event[2651]: S10update-grub=action|Event|post-upgrade|Action|S10update-grub|Start|1627648190 503844|End|1627648194 380767|Elapsed|3.876923
Jul 30 14:29:54 test esmith::event[2651]: Running event handler: /etc/e-smith/events/post-upgrade/S15group-modify-unix
Jul 30 14:29:54 test esmith::event[2651]: usermod: no changes
Jul 30 14:29:54 test esmith::event[2651]: CPU: ldapUserMod: ldap_modify_s: No such object
Jul 30 14:29:54 test esmith::event[2651]:      The user 'all_group' specified could not be found in the directory.
Jul 30 14:29:54 test esmith::event[2651]:      Make sure the user is valid and the correct base for the user has been
Jul 30 14:29:54 test esmith::event[2651]:      specified.
Jul 30 14:29:54 test esmith::event[2651]: Failed to modify (ldap) group description for all_group.
Jul 30 14:29:54 test esmith::event[2651]: CPU: ldapGroupMod: ldap_modify_s: No such object
Jul 30 14:29:54 test esmith::event[2651]:      The group 'all_group' specified could not be found in the directory.
Jul 30 14:29:54 test esmith::event[2651]:      Make sure the group is valid and the correct base for the group has been
Jul 30 14:29:54 test esmith::event[2651]:      specified.
Jul 30 14:29:54 test esmith::event[2651]: Failed to modify (ldap) group description/email for all_group.

And then after you reboot you seem to have major borkage:

Jul 31 13:35:27 beastserver2 esmith::event[7142]: Running event handler: /etc/e-smith/events/post-upgrade/S15group-modify-unix
Jul 31 13:35:27 beastserver2 esmith::event[7142]: usermod: Ingen Test
Jul 31 13:37:34 beastserver2 esmith::event[7142]: CPU: ldapOperation: ldap_bind_s: Can't contact LDAP server
Jul 31 13:37:34 beastserver2 esmith::event[7142]:      The LDAP server specified at localhost could not be contacted.
Jul 31 13:37:34 beastserver2 esmith::event[7142]:      Your LDAP server may be down or incorrectly specified.
Jul 31 13:37:34 beastserver2 esmith::event[7142]: Failed to modify (ldap) group description for all_group.
Jul 31 13:39:41 beastserver2 esmith::event[7142]: CPU: ldapOperation: ldap_bind_s: Can't contact LDAP server
Jul 31 13:39:41 beastserver2 esmith::event[7142]:      The LDAP server specified at localhost could not be contacted.
Jul 31 13:39:41 beastserver2 esmith::event[7142]:      Your LDAP server may be down or incorrectly specified.
Jul 31 13:39:41 beastserver2 esmith::event[7142]: Failed to modify (ldap) group description/email for all_group.

Looks like your v9 server/backup is not well. What have you been up to?

On your v9 server please show:

Code: [Select]

rpm -qa |grep samba

[beast]

I think I said this to you earlier.

So why have you got this?

Code: [Select]

$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/e-smith/templates-custom/* root@x.x.x.x::beastserver/etc/e-smith/templates-custom/ --ignore-missing-args");
$returnstr .= shell_exec("rsync -az -e \"ssh -p 22\" /etc/e-smith/templates-user-custom/* root@x.x.x.x::beastserver/etc/e-smith/templates-user-custom/ --ignore-missing-args");

I have deleted all files in those folders.
I am only planning to transfer those files to the new server to find a possible error.

Next, I can see an issue in your logs but can't remember the solution - I think it is to do with missing information in ldap.

e.g.

Please show:

Code: [Select]

config show ldap

[root@beastserver2 ~]# config show ldap
ldap=service
    Authentication=disabled
    Bootstrap=run
    TCPPort=389
    TCPPorts=389,636
    access=private
    defaultCity=[deleted]
    defaultCompany=Privat
    defaultDepartment=Main
    defaultPhoneNumber=+45 [deleted]
    defaultStreet=[deleted]
    status=enabled

Also I can see you are running owncloud..... and you have some issues in there eg

You ought to go and fix some of the files - see your logs for which ones.

Owncloud is messed up and not working anymore after an upgrade - not important for me

We also did ask you to try and get the output from newrpms and templates. This is why - so we know what else you have squirrelled away there that might be causing issues.

I do not understand what to do here?

Ahhh, and then this....

And then this (a small section as there is a lot more)

And then after you reboot you seem to have major borkage:

Looks like your v9 server/backup is not well. What have you been up to?

On your v9 server please show:

Code: [Select]

rpm -qa |grep samba

[root@beastserver ~]# rpm -qa |grep samba
samba-winbind-clients-3.6.23-53.el6_10.x86_64
e-smith-samba-2.4.0-27.el6.sme.noarch
samba-client-3.6.23-53.el6_10.x86_64
samba-common-3.6.23-53.el6_10.x86_64
samba-3.6.23-53.el6_10.x86_64
samba-winbind-3.6.23-53.el6_10.x86_64

[TerryF]

apoligies beast, downloaded all the logs and got sidetracked..just back to it..

I think you may be better off doing a straight sme10 install, copy over users and their data, mysql, wiki has how to, and then install any contribs..

Might be quickest way to get a working system

[beast]

apoligies beast, downloaded all the logs and got sidetracked..just back to it..

I think you may be better off doing a straight sme10 install, copy over users and their data, mysql, wiki has how to, and then install any contribs..

Might be quickest way to get a working system

Yes - agree. Just trying to find out what is wrong to help other.

[TerryF]

Yes - agree. Just trying to find out what is wrong to help other.

Oh yeah, definetly, if nothing else makes for a better nights sleep

[ReetP]

I think you may be better off doing a straight sme10 install, copy over users and their data, mysql, wiki has how to, and then install any contribs.

With the corruption in there I'm not sure how easy that is going to be with users.

Not sure how it has happened. I suspect either the famous samba from smedev, or possibly an issue with owncloud at somestage.

I would imagine there are some errors evident in the v9 logs too.