Koozali.org: home of the SME Server

How to update completely and properly a SME?

Offline Arnaud

  • *
  • 143
  • +0/-0
    • GuedeL
How to update completely and properly a SME?
« on: December 31, 2016, 12:44:08 PM »
Hello,

with this question, I don't mean a  simple "yum update"......"
yum update --enablerepo=*" is wrong too!

Due to dependencies coming sometimes from different repos, I'm looking for a way to update all the rpms from the repo of which they were installed from:
eg. if XYZ.rpm has been installed from "epel", yum should only look for in the "epel" repo for this rpm and make the update if present into epel. If XYZ.rpm hast got a more recent release in another repo, this more recent release should be ignore.

Of course, I don't want to list the rpm's manually...

Thanks.
Bye
Arnaud

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: How to update completely and properly a SME?
« Reply #1 on: December 31, 2016, 01:35:22 PM »
Code: [Select]
yum update

will update only OS

Code: [Select]
yum --enablerepo=smecontribs update

wil update both contribs and OS

I understand you need, but there are more cons that pro to update massively a server..
the external repos (epel, dag, whatever) must be enabled only when you need to update a single contrib

something like
Code: [Select]
yum --enablerepo=smecontribs,epel update smeserver-mycontrib

you'd try to play with the check_updates (or something similar) script to make it do some work for you.. be aware: it can break your server, play on a test machine

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: How to update completely and properly a SME?
« Reply #2 on: December 31, 2016, 04:25:47 PM »
Arnaud

Interesting idea.
You should lodge a New Feature Request (NFR) bug report at Bugzilla, & see what the devs think of your idea.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: How to update completely and properly a SME?
« Reply #3 on: December 31, 2016, 04:27:34 PM »
This forum post from 2011 describes a way to configure yum repositories to make subsequent updates easy:
https://forums.contribs.org/index.php?topic=48005.0

Since that process is so painful to maintain on my test system, I tend to do this:

* yum update --enablerepo=* update
--> answer "no"
--> note the non-sme packages with pending updates
-->  ** update the non-sme packages manually, one-by-one, with the minimum number of repos enabled to accomplish the udpate
--> finish with yum update

** This step depends on memory & research.  It has always frustrated me that /var/log/yum.log does not indicate the repository from which a particular package was installed...

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: How to update completely and properly a SME?
« Reply #4 on: December 31, 2016, 04:54:26 PM »
this could be a start:
Code: [Select]
yum list installed | grep -v '@anaconda' | grep -v '@base'

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: How to update completely and properly a SME?
« Reply #5 on: December 31, 2016, 05:24:22 PM »
One way with epel could be to create a myepel repo in yum repo db, identical to the epel one, except for the name and the inclusion of a Include statement listing all the rpm you decided to add once, and keep it uptodate. Also you can set this one enabled so it is used for auto update.

Offline Arnaud

  • *
  • 143
  • +0/-0
    • GuedeL
Re: How to update completely and properly a SME?
« Reply #6 on: December 31, 2016, 05:27:33 PM »
Hello,

thanks for your replies.
Quote
something like
Code: [Select]
yum --enablerepo=smecontribs,epel update smeserver-mycontrib
unfortunately, this would only update "smeserver-mycontrib" but not its dependencies.

I've got a concrete example with my test VM:
smeserver-php-scl installed by a "yum install smeserver-php-scl --enablerepo=smecontribs,remi,epel" a few weeks ago.
The repo "remi" gives a lot of dependencies "php55...", "php54..."
Today
Code: [Select]
yum --enablerepo=smecontribs,remi,epel update smeserver-php-scl returns that "smeserver-php-scl" is up to date.
Today
Code: [Select]
yum update php5* --enablerepo=remi returns that 16 rpm's are for update available. I know that these are all dependencies of "smeserver-php-scl"

If I would install a new machine today, these updates would be installed by the "yum install smeserver-php-scl --enablerepo=smecontribs,remi,epel"

As a help for manual update:
Code: [Select]
yum list installed | grep remi or as a basic for a script that would  return a clean list of rpm that could be drag&drop in a yum command.

Bye

PS: ups, Stefano and JP are quicker.....
edit: wrong command!


« Last Edit: December 31, 2016, 05:31:28 PM by Arnaud »

Offline Arnaud

  • *
  • 143
  • +0/-0
    • GuedeL
Re: How to update completely and properly a SME?
« Reply #7 on: December 31, 2016, 05:56:51 PM »
on my test VM:
Code: [Select]
yum install yum-utils-> 2x rpm / 400kB
Code: [Select]
repoquery -a --installed --qf "%{name} %{ui_from_repo}" | grep @smecontribs
BackupPC @smecontribs
awstats @smecontribs
fuse-encfs @smecontribs
libmcrypt @smecontribs
.....
very clean output => "just have to" remove the strings beginning by "@" and the "return" in order to have an "horizontal" list......

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: How to update completely and properly a SME?
« Reply #8 on: January 01, 2017, 04:07:49 PM »
Here's a start...  on my system I have a bunch of rpms installed using 'localinstall' which cause problems...

Code: [Select]
for f in $(/sbin/e-smith/audittools/newrpms |awk '/i386|i586|noarch/ {print $1}';); do \
repo=$(repoquery $f |awk '/Repo/ {print $3}';);  \
echo yum --enablerepo=$repo $f; \
done

Offline Arnaud

  • *
  • 143
  • +0/-0
    • GuedeL
Re: How to update completely and properly a SME?
« Reply #9 on: January 02, 2017, 03:18:26 PM »
Hi mmccarn,
I can't get your script running properly and I can't understand it completely... :oops: No "update" into the yum command?
I guess that it generates a command "yum --enablerepo=THE-REPO ????? the_rpm" for each rpm.

Bye
Arnaud

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: How to update completely and properly a SME?
« Reply #10 on: January 02, 2017, 03:55:14 PM »
Hi mmccarn,
I can't get your script running properly and I can't understand it completely... :oops: No "update" into the yum command?
I guess that it generates a command "yum --enablerepo=THE-REPO ????? the_rpm" for each rpm.

Bye
Arnaud

Yes; I somehow lost the "-i" argument for repoquery, and there are other problems.

Here is a command that "works", but is still only a partial solution. 

This command will show the various "yum update" commands that need to be run, but it would still potentially pull in unwanted dependencies from the non-SME repositories.

Code: [Select]
for f in $(/sbin/e-smith/audittools/newrpms |awk '/i386|i586|noarch/ {print $1}';); do \
repo=$(repoquery -i $f |awk '/Repo/ {print $3}';); \
if [[ -z "$repo" ]]; then \
echo "# $f not installed from yum repository"; \
else \
echo "yum --enablerepo=$repo update $f"; \
fi; \
done

A full solution would require the creation of new repos  using IncludePkgs.

A modified version of this script could be used to:
* identify the non-SME packages that have been installed (/sbin/e-smith/audittools/newrpms)
* display the repos they were installed from (if any)



Offline Arnaud

  • *
  • 143
  • +0/-0
    • GuedeL
Re: How to update completely and properly a SME?
« Reply #11 on: January 02, 2017, 11:00:08 PM »
I'm preparing "something"  :-? based on your first command. It should execute a yum command for each repo and not for each rpm.
The central point is:
Code: [Select]
#!/bin/bash

repo="sogo";

for rpm in $(/sbin/e-smith/audittools/newrpms |awk -v repo_awk=@"$repo" '/@sogo/ {print $1}'); do     ####### -v repo unnecessary in this command, but just for tests to make sure that it doesn't make troubel
#for rpm in $(/sbin/e-smith/audittools/newrpms |awk -v repo_awk=@"$repo" '/$repo_awk/ {print $1}}'); do  ########## WHY DOESN'T IT RUN????? "$repo_awk" doesn't get recognized

     rpms=$rpm' '$rpms

done

yum --enablerepo=$repo update $rpms;

exit 0

In this exemple, for repo "sogo" the script "loads" in my case:
Code: [Select]
yum --enablerepo=sogo update sope49-xml.x86_64 sope49-sbjson.x86_64 sope49-mime.x86_64 sope49-ldap.x86_64 sope49-gdl1-mysql.x86_64 sope49-gdl1-contentstore.x86_64 sope49-gdl1.x86_64 sope49-core.x86_64 sope49-cards.x86_64 sope49-appserver.x86_64 sogo-tool.x86_64 sogo-ealarms-notify.x86_64 sogo-activesync.x86_64 sogo.x86_64 libwbxml.x86_64 libmemcached.x86_64 gnustep-base.x86_64 and ask Y/n for processing.
=> this is in the correct direction.

Now, I would like not to entering "@sogo" in the script but use a variable "repo" (that could be set by hand or from the command "yum --repolist --all") and I'm definitively not able to put the variable "repo_awk" in the condition of the regex of awk. (see above)  :sad:
=> the help of an expert with regex is need!!

Thanks
Arnaud

PS: I hate bash and regex hate me! :evil:
« Last Edit: January 02, 2017, 11:03:19 PM by Arnaud »

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: How to update completely and properly a SME?
« Reply #12 on: January 03, 2017, 04:39:23 AM »
Arnaud,
rpm from epel, should be present in smecontribs repo. If not you could open a bug against the contrib that does not has it to help us correct that.

rpm from remi, needed by contribs will soon be imported into smecontribs repo.

Of course you present work might still be needed for other rpm you might need not related to a contrib in smecontribs.

Offline Arnaud

  • *
  • 143
  • +0/-0
    • GuedeL
Re: How to update completely and properly a SME?
« Reply #13 on: January 03, 2017, 10:43:42 PM »
Hi
@JP: thanks for making me aware that there won't be a real need for this in the future but my "special" friends "bash" and "regex" won't let me sleep in peace until I show them who is stronger.........
So just for fun and personal satisfaction:
Code: [Select]
#!/bin/bash

echo "List of available repos beside SME:";
#list the disabled repos
for repo in $(yum repolist disabled |awk  '$1 !~ /id|Modules|repolist:/ {print $1}'); do

      echo $repo;

done
echo "";

# repo selection
index=0;
repos="";
proceed="proceed";

until [ "$repos" == "$proceed" ]; do

        repos[$index]=$repos;
        ((index=index+1));
        read -p 'Enter 1x repo name for selection or "proceed" to start the updating: ' repos;

done

# remove the repo "proceed"
repos=("${repos[@]:1}");


for repo in "${repos[@]}"; do
        echo "";
        echo "======================================";
        echo -e "\tUPDATE from repo: "$repo;
        echo "======================================";

        # generate the list of rpm installed from the repo
        for rpm in $(/sbin/e-smith/audittools/newrpms |awk -v repo_awk=@$repo 'repo_awk==$3 {print $1}'); do

                rpms=$rpm' '$rpms

        done

# updating
        yum --enablerepo=$repo update $rpms;

done

exit 0

I tested it on the VM and it seems to work.
The User manual is: enter each updating repo (only 1 repo per line) and enter "proceed" to close the selection and start the update. Don't worry: yum will ask for "real" updating as usual after checking the updates.

For me it is enough.
Maybe we could add a listing of the rpm installed beside the repos by a "localinstall".......

Have fun.
Arnaud

PS: I would be interested to get improvements of the syntax or for the structure of this script, not only for my "special" friends..... :cool: but in order to learn something. Thanks.

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: How to update completely and properly a SME?
« Reply #14 on: February 25, 2017, 08:09:35 PM »
Here is a modified version of your script.

This version:

* runs 'newrpms' to get all repos that have been used to install non-standard packages
* creates a list of rpms for each such repo
* use "--setopts" to specify "includepkgs" for each repo during update
* runs the update

My objectives are two-fold:
* A single command to update all non-SME packages
* No unintended packages from the wrong repo

Bonus: the updates seem to run faster using 'includepkgs'.

Code: [Select]
#!/bin/bash

for repo in $(/sbin/e-smith/audittools/newrpms |grep \@ |awk ' {print $3}' |sort -u |sed s/@//); do

        # generate the list of rpm installed from the repo
        for rpm in $(/sbin/e-smith/audittools/newrpms |awk -v repo_awk=@$repo 'repo_awk==$3 {print $1}'); do
                rpms=$rpm' '$rpms
        done

        echo -e "\n\n===";
        echo -e "Repo: "$repo;
        echo -e "\nIncludePkgs: "$rpms;
        echo "===";
        # updating
        yum --enablerepo=$repo --setopt="$repo.includepkgs='$rpms'" update

done

exit 0