Koozali.org: home of the SME Server

In the Geoip contrib GeoIP.dat does not get copied to /usr/share/GeoIP

Offline purvis

  • ****
  • 567
  • +0/-0
http://wiki.contribs.org/GeoIP
after following directions in the GeoIP  contrib wiki
The new GeoIp.dat file created by way of the crontab into the directory /var/lib/GeoIp does not update(copy) the GeoIP.dat file into /usr/share/GeoIP.
I understand the geoiplookup program defaults to looking for the file GeoIP.dat in the directory of /usr/share/GeoIP.
I tested the geoiplookup after deleting the file in the directory of /usr/share/GeoIP (rm -f /usr/share/GeoIp/GeoIP.dat) and the geoiplookup program does not return anything if the file is not there at that location.  geoiplookup does have an override option to specify the database ( geoiplookup -f /var/lib/GeoIP/GeoIP.dat  google.com).

to show your both the current GeoIP.dat files here is the commands
Code: [Select]
ls -l /var/lib/GeoIP
ls -l /usr/share/GeoIP


This bash file does download the new GeoIP.dat file and updates the /usr/share/GeoIP directory as well.
geoipdatupdate
Code: [Select]
#!/bin/bash
cd /
mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP

cd /var/lib/GeoIP
rm -f /var/lib/GeoIP/GeoIP.dat.gz
rm -f /var/lib/GeoIP/GeoIP.dat.gz.*

wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz > /dev/null

if [ -f "GeoIP.dat.gz" ];then
   rm -f /var/lib/GeoIP/GeoIP.dat
   gunzip GeoIP.dat.gz > /dev/null
fi

if [ -f "GeoIP.dat" ];then
   rm -f /usr/share/GeoIP/GeoIP.dat
   cp GeoIP.dat /usr/share/GeoIP
fi

#ls -l /var/lib/GeoIP
#ls -l /usr/share/GeoIP

exit 0

The crontab shows a once a month chance to update the GeoIP.dat file.
I think that this can be improved on by having more than one chance.
I will look into creating a hourly or daily cron that updates the GeoIP.dat file.
The GeoIP.dat file is changed monthly on the GeoIP.dat supplier side and that occurs more towards the beginning of the month, from what I understand.

Offline purvis

  • ****
  • 567
  • +0/-0
I created a bash routine to update the GeoIP.dat files on our servers to be placed in the /etc/cron.hourly or /etc/cron.daily directory.
This should try to update the GeoIP.dat file starting immediately if the GeoIP.dat file does not exist, GeoIP.dat is 2 months old or after the first Tuesday of the month. From the Q&A on the Maxmind web site, the GeoIP.dat country database file is suppose to be updated on the first tuesday of each month.

If the GeoIP.dat file is considered up to date. No updating will be done.

for what is worth to others here is my code
Code: [Select]
#!/bin/bash


#geoipdatupdate
#date edited 2013-04-25
#can be placed in the directory /etc/cron.hourly or /etc/cron.daily directory
#this bash routine updates the GeoIP.dat country database file in /usr/share/Geoip from the internet

#website location of where to get geoip.dat.gz
weblocationofgeoipdatgz="geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"


function getgeoipdatstatus {
   updatethefiles=0
   mkdir -p /var/lib/GeoIP
   mkdir -p /usr/share/GeoIP
   if [ ! -f /usr/share/GeoIP/GeoIP.dat ]
      then
      let updatethefiles=1
      return
   fi
   # get the date of the file GeoIP.dat in /usr/share/GeoIP directory
   geoipdatdate=$(stat -c %y /usr/share/GeoIP/GeoIP.dat)
   geoipdatdate=${geoipdatdate%% *}
   geoipdatdateyear=${geoipdatdate:0:4}
   geoipdatdatemth=${geoipdatdate:5:2}
   geoipdatdateday=${geoipdatdate:8:2}
   # get the current date and place in months
   currentyear=$(date +%Y)
   currentmth=$(date +%m)
   currentday=$(date +%d)
   #if the date of the file matches the current month abort update
   if [ $geoipdatdateyear == $currentyear ] && [ $geoipdatdatemth == $currentmth ]
      then
      return
   fi
   #if the date of the GeoIP.dat files is two months old, update it
   if [ $updatethefiles -eq 0 ]
      then
      date1=$((geoipdatdateyear * 12 + geoipdatdatemth ))
      date2=$((currentyear * 12 + currentmth ))
      date3=$((date2-date1))
      if [ $date3 -gt 1 ];then let updatethefiles=1;fi
   fi
   # if the date of the GeoIP.dat file is one month old and past the first tuesday of the month
   # iwait till the 4rd day of the month before trying to update
        if [ $geoipdatdatemth != $currentmth ] || [ $geoipdatdateyear != $currentyear ]
           then
           if [ $currentday -gt 7 ];then let updatethefiles=1;return;fi
           dayofweek=$(date +%a)
           if [ $dayofweek == "Wed" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Thu" ] && [ $currentday -ge 2 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Fri" ] && [ $currentday -ge 3 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Sat" ] && [ $currentday -ge 4 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Sun" ] && [ $currentday -ge 5 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Mon" ] && [ $currentday -ge 6 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Tue" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
   fi
return
}

getgeoipdatstatus
if [ $updatethefiles -eq 0 ];then exit 0;fi

mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP

cd /var/lib/GeoIP
rm -f /var/lib/GeoIP/GeoIP.dat.gz
rm -f /var/lib/GeoIP/GeoIP.dat.gz.*

wget -q http://$weblocationofgeoipdatgz > /dev/null

if [ -f "GeoIP.dat.gz" ];then
   rm -f /var/lib/GeoIP/GeoIP.dat
   gunzip GeoIP.dat.gz > /dev/null
fi

if [ -f "GeoIP.dat" ];then
   rm -f /usr/share/GeoIP/GeoIP.dat
   cp -p GeoIP.dat /usr/share/GeoIP
fi

#ls -l /var/lib/GeoIP
#ls -l /usr/share/GeoIP

exit 0
« Last Edit: April 25, 2013, 11:35:23 AM by purvis »

Offline purvis

  • ****
  • 567
  • +0/-0
After reading that Maxmind updates the GeoIP.dat country database file on the first Tuesday of each month.
I altered the script to update on any day after the first Tuesday of the month has passed.
I removed the part of the code where the updates where to be made starting xx(default was 4) number of days after the beginning of the month.
If the GeoIP country database is older than 1 month old, the update will try tried immediately.
An update is tried only once per script run, this is why my favorite place to hold this script will be in cron.hourly directory so that failures in downloading from the internet can be overcome in a near future if a failure occurred.

Offline purvis

  • ****
  • 567
  • +0/-0
In the GeoIp contrib located here
http://wiki.contribs.org/GeoIP
There is written today are instructions to create a cron-job and here are the basic instructions
Code: [Select]
mkdir -p /etc/e-smith/templates-custom/etc/crontab

vim /etc/e-smith/templates-custom/etc/crontab/91_Update_GeoIP_db
#insert the below 2 lines in the file 91_Update_GeoIP_db and save the file
# Updating the GeoIP database monthly on the 5th at 0:00h.
0 0 5 * * root /usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O /var/lib/GeoIP/GeoIP.dat.gz; /bin/gunzip -f /var/lib/GeoIP/GeoIP.dat.gz

expand-template /etc/crontab

The code above does not copy the GeoIP.dat file into the /usr/share/GeoIP directory.
This code also it tries to update the GeoIP.dat file only once per month.
I am using the bash file as written above to update GeoIP.dat and place the file where it is needed. A good directory to place the bash is in the dialy cron directory "/etc/cron.daily".
I have cable internet and the bash file above only takes seconds to update the GeoIP.dat file. I have placed the bash routine in my hour cron directory "/etc/cron.hourly".
I deleted the cron-job created from the instructions at http://wiki.contribs.org/GeoIP
Code: [Select]
rm -f /etc/e-smith/templates-custom/etc/crontab/91_Update_GeoIP_db
expand-template /etc/crontab


The command to check the current date of the GeoIP.dat file in the /usr/share/GeoIp directory
Code: [Select]
geoiplookup -v /usr/share/GeoIP/GeoIP.dat
The returned text from the command will likely have a date of the file in the format of YYYYYMMDD.
As of today, the default location geolookup uses for the data file for GeoIP /usr/share/GeoIP/
« Last Edit: April 29, 2013, 07:11:27 PM by purvis »

Offline purvis

  • ****
  • 567
  • +0/-0
I oddly found some bash code that did not work today.

It occurs when there is a zero before a number like in a day  08 for 8th day of the month.
I am new to bash programming and this is the kind of stuff that get's into your gut.
I think the new code to assign dates values number will work now.
I wanted to get this fixed and updated here on the contribs.org forum.

Here is the new bash code that updates the GeoIP.dat file.
Once again, I put the bash routine in the directory of /etc/cron.hourly so it will get run as soon as possible when there is a internet connection.
Code: [Select]
#!/bin/bash

#geoipdatupdate
#date edited 2013-05-15
#can be placed in the directory /etc/cron.hourly or /etc/cron.daily directory
#this bash routine updates the GeoIP.dat file in /usr/share/Geoip from the internet

#website location of where to get geoip.dat.gz
weblocationofgeoipdatgz="geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"


function getgeoipdatstatus {
   local stemp="0"
   updatethefiles=0
   mkdir -p /var/lib/GeoIP
   mkdir -p /usr/share/GeoIP
   if [ ! -f /usr/share/GeoIP/GeoIP.dat ]
      then
      let updatethefiles=1
      return
   fi
   if [ ! -f /var/lib/GeoIP/GeoIP.dat ]
      then
      let updatethefiles=1
      return
   fi

   # get the date of the file GeoIP.dat in /usr/share/GeoIP directory
   geoipdatdate=$(stat -c %y /usr/share/GeoIP/GeoIP.dat)
   geoipdatdate=${geoipdatdate%% *}
   geoipdatdateyear=${geoipdatdate:0:4}
   geoipdatdatemth=${geoipdatdate:5:2}
   geoipdatdateday=${geoipdatdate:8:2}
 
   geoipdatdateyear=`expr $geoipdatdateyear + 0`
   geoipdatdatemth=`expr $geoipdatdatemth + 0`
   geoipdatdateday=`expr $geoipdatdateday + 0`

# get the current date and place in months
   currentyear=$(date +%Y)
   currentmth=$(date +%m)
   currentday=$(date +%d)
   
   currentyear=`expr $currentyear + 0`
   currentmth=`expr $currentmth + 0`
   currentday=`expr $currentday + 0`

   #if the date of the file matches the current month abort update
   if [ $geoipdatdateyear == $currentyear ] && [ $geoipdatdatemth == $currentmth ]
      then
      return
   fi
   #if the date of the GeoIP.dat files is two months old, update it
   if [ "$updatethefiles" -eq 0 ]
      then
      date1=$((geoipdatdateyear * 12 + geoipdatdatemth ))
      date2=$((currentyear * 12 + currentmth ))
      date3=$((date2-date1))
      if [ $date3 -gt 1 ];then let updatethefiles=1;fi
   fi
   # if the date of the GeoIP.dat file is one month old and past the first tuesday of the month
   # iwait till the 4rd day of the month before trying to update
        if [ $geoipdatdatemth != $currentmth ] || [ $geoipdatdateyear != $currentyear ]
           then
           if [ $currentday -gt 7 ];then let updatethefiles=1;return;fi
           dayofweek=$(date +%a)
           if [ $dayofweek == "Wed" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Thu" ] && [ $currentday -ge 2 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Fri" ] && [ $currentday -ge 3 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Sat" ] && [ $currentday -ge 4 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Sun" ] && [ $currentday -ge 5 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Mon" ] && [ $currentday -ge 6 ];then let updatethefiles=1;fi
           if [ $dayofweek == "Tue" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
   fi
return
}

getgeoipdatstatus
if [ $updatethefiles -eq 0 ];then exit 0;fi

mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP

cd /var/lib/GeoIP
rm -f /var/lib/GeoIP/GeoIP.dat.gz
rm -f /var/lib/GeoIP/GeoIP.dat.gz.*

wget -q http://$weblocationofgeoipdatgz > /dev/null

if [ -f "GeoIP.dat.gz" ];then
   rm -f /var/lib/GeoIP/GeoIP.dat
   gunzip GeoIP.dat.gz > /dev/null
fi

if [ -f "GeoIP.dat" ];then
   rm -f /usr/share/GeoIP/GeoIP.dat
   cp -p GeoIP.dat /usr/share/GeoIP
fi

#ls -l /var/lib/GeoIP
#ls -l /usr/share/GeoIP

exit 0

« Last Edit: May 15, 2013, 11:22:39 AM by purvis »

Offline Jean-Philippe Pialasse

  • *
  • 2,746
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
alternatively

wiki.contribs.org/Talk:GeoIP


be aware that some versions of geoip will expect databases at another place, here is a modified version of cron part to avoid this :

# Updating the GeoIP database monthly on the 5th at 0:00h.
 0 0 5 * * root /usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O /var/lib/GeoIP/GeoIP.dat.gz; /bin/gunzip -f /var/lib/GeoIP/GeoIP.dat.gz; cp /var/lib/GeoIP/*.dat /usr/share/GeoIP/
 0 0 5 * * root /usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz -O /var/lib/GeoIP/GeoIPCity.dat.gz; /bin/gunzip -f /var/lib/GeoIP/GeoIPCity.dat.gz; cp /var/lib/GeoIP/*.dat /usr/share/GeoIP/


Offline purvis

  • ****
  • 567
  • +0/-0
Because the newer GeoIp.dat files are posted some time on the first Tuesday of every month. It would make more since to have the cron do it's job on the 8th or the 9th of each month.
Personally, I think having a cron trying only once a month to update the GeoIP.dat file is too chancy.

Offline kruhm

  • *
  • 680
  • +0/-0
Hi,

I've developed and maintain the smeserver-geoip rpm. Thanks for your interest in improving it. I can see that you've put much thought and energy into it recently.

Here are my thoughts after briefly reading through the info above:
-if something doesn't work as expected, file a bug. I happen to run into this thread but that won't always happen.

Quote
The new GeoIp.dat file created by way of the crontab into the directory /var/lib/GeoIp does not update(copy) the GeoIP.dat file into /usr/share/GeoIP.
-I'm not sure why I thought it's required to be the /var/lib/GeoIP/ directory. I must have had good reason to though. Maybe this changed somewhere along the way and the wiki never got updated.

Quote
The crontab shows a once a month chance to update the GeoIP.dat file.
I think that this can be improved on by having more than one chance.
What's the point? It updates once a month via cron. That's the end of it. Updating more than that will do absolutely nothing.

Thanks,