Topic: [ANNOUNCE] Cyrus-IMAP for SME 7 Release for testing

[jonroberts]

The cyrus-imap contrib has now been updated for SME 7 and is available for download from http://www.westcountrybusiness.com (via 'Tech Forum' link).

This update contains all new RPMS, an enhanced Server-Manager panel and an uninstall feature (Yes, finally).  There is also a script for uninstalling on SME6 and how to upgrade from SME6 to SME7.

Testing of this contrib has so far been limited to two test servers.  The SME6 version experienced problems in that it failed when certain updates were applied.  I hope this has been resolved, but there are no guarrantees.  

Make backups first - test at your own risk.  Please post your comments / feedback.

[soprom]

Thanks! Great work Jon!

Works fine on a fresh install!
Also, on a modified SME7, installed and uninstalled correctly. Mails were left intact after uninstallation. This SME has fetchmail, openxchange, asterisk.

[soprom]

After a yum update, the imap reverted to the standard one.
It's because I mistakenly set properties in the server-manager email settings concerning imap and pop. They must be left set to off.
Access to imap accounts came back.

[jonroberts]

Sophie,

Thanks for this.  The original cyrus install should have already made all the settings in your script & I didn't think that a Yum update made any config changes, although maybe if one of the SME core RPMs was updated, it might have reset some settings.  (Not quite sure how YUM works in detail yet ...)

I'd think that re-running the cyrus install script would also fix the problem.  I'll give it a try to make sure as soon as I get chance.  

Keep up the testing  

Jon

[jonroberts]

Sophie

I've successfully performed a YUM update without a problem, so I guess it depends on what's being updated.

At least if there is a problem, its fairly easily reset using either your script or re-running the install script, but I'll keep testing just in case ...

Jon

[soprom]

Sorry, it was my error since I activated pop and imap in the manager after installing the pakage. They must be left disabled. If they are re-enabled from the manager, the system revert to the standard services.

Sorry about the false alarm!

[jonroberts]

Sorry, it was my error since I activated pop and imap in the manager after installing the pakage. They must be left disabled. If they are re-enabled from the manager, the system revert to the standard services.

Thanks for the update, Sophie.  You're right, of course.  For reference (for anyone else watching this thread)  the cyrus-imap install replaces the standard SME packages for both POP and IMAP, so these services are not lost. - just provided by cyrus-imap instead of dovecot.

I did consider uninstalling the redundant SME packages all together, but felt that would make it harder to uninstall cyrus & revert back to 'standard' SME if required.  That's why I opted for just disabling them.  Of course you've found the flaw in this plan, which is that if you happen to re-enable them manually, they will conflict.  Still, you can't win 'em all ...

[soprom]

I did the migration from the standard mail to the cyrus mail using a mail client (thunderbird).

WIth the standard services, I downloaded the mail items with POP. Then after switching to cyrus, I dragged the mail items from the local pop to the imap account.

It's Ok for a small group but there must be a better way for a larger group.

[jonroberts]

It's Ok for a small group but there must be a better way for a larger group.

Sophie - I agree, but its not an area I've ever spent much time looking into.  I use cyrus-imap as a back-end for Outlook users.  This works by syncronising the contents of the local Outlook folders (.pst file) with the IMAP server.  So if I ever need to upgrade / migrate, I can build an emtpy cyrus-imap DB & reconnect the Outlook PST files & re-sync from local to server.

Of course, that's no help to you.  I guess you could do something similar with an IMAP client that will 'download' the content of the user folders for 'off-line' access & then use this offline data to re-sync (ie import) back into cyrus-imap - but that's pretty much what you're doing already with Thunderbird.

I googled a bit and found a couple of items (nothing definitive).  Here's a link that may be of interest (I expect there's more out there if you've got the patience to dig a little deeper).

http://barnson.org/node/81

[mudtoe]

I just installed CYRUS, and I've got a couple of questions.  First, I think I've found a bug in that it looks like CYRUS doesn't pay attention to the Local Networks settings, as I've been unable to establish an IMAP connection to it via Outlook or IMAPNotify from a VPN subnet that I have setup in SME as a local network.  It works fine from a machine on the local network, and it works find from the machine having the problem if I pass the IMAP datastream through a proxy server that's on the same subnet as CYRUS.

Second question, who is Sherwood?  That's what I have on my SME admin screen for CYRUS as the topic under the heading "Unknown".  

Lastly, and this is probably more a toltec question than an CYRUS one, but when I install toltec (I installed CYRUS so that I could use toltec to share calendars and such), does all the email data get preserved when I switch the email account from IMAP to toltec?  

Thanks for the assistance,

mudtoe

[jonroberts]

Mudtoe - I've never needed to connect to Cyrus via a VPN connection & so haven't come across this problem.  However, I have just created a VPN connection to a client site running Cyrus (on SME6) and was able to connect via Toltec.  I was also able to open IMAP folders on that server using Outlook Express, so I'm not sure I can replicate what you're seeing.

'Sherwood' is the name of the people that did the original work for the Cyrus-Imap contrib, but I thought I'd removed the reference to them (at least in the English version).  Which version of SME are you running?  I think a reboot may solve the label problem, but to be sure issue the following 3 commands:

 /sbin/e-smith/db /home/e-smith/db/navigation/navigation.en set sherwood panel Description "cyrus-IMAP" DescriptionWeight 6750 Heading Configuration HeadingWeight 6000
signal-event console-save
signal-event reboot

Finally, Toltec is for use with Outlook.  Do I understand that you are using IMAP for your e-mail with Outlook?  Toltec will share the folders in your standard PST file (your Personal Folders) by synchronising them with the Cyrus-IMAP server.  If you're using IMAP for e-mail, then you've probably got 2 sets of folders (your IMAP accounts & your Outlook personal folders) in which case installing Totlec wont affect your Inbox.  

When I use Toltec, I just use one set of folders (the Personal Folders) & POP for retrieving mail into the Inbox & allow Totlec's synchronisation to enable me to share the other folders.

[soprom]

With a VPN connection, it could help to set a reference to the host name in the client's PC in the hosts file.

[mudtoe]

I ran the commands and rebooted, but sherwood is still alive and well.  I'm going to put the machine having the subnet problem on the same subnet as the SME server just to make sure the problem has nothing to do with something screwy in the machine itself.  Rebooting the SME server didn't solve the subnet problem.  

With regard to the VPN and hosts file, I'm using the direct numeric IP addresses in the outlook email configurations, so I don't think I've got any kind of DNS problem.  It's not a routing problem either as I'm able to access the SME server and Horde web pages from the same machine.

Thanks for the info on Toltec.  I do already have several folders for Outlook, one for the calendar, contacts, and Todo list, which is a local PST file for each copy of Outlook, and a number of shared IMAP folders, one for each email address I'm using.  Is there any benefit to converting the email accounts back to POP3, putting them back in the local PST file, and then letting Toltec control the sharing between the copies of Outlook?  I thought I saw a comment that said the new mail indicator wouldn't work with Toltec, and that issue already exists with IMAP under Outlook, although once in a while I get a new mail icon in the tray.  I'm using a program I downloaded called IMAPNotify that sits in the tray and performs the notification function for IMAP email accounts.

P.S.

I'm seeing lots of the following messages in the message log since I installed CYRUS.  Everything seems to be working, but even though I don't know exactly what these mean, I get the idea that something is broken:

Sep 19 00:12:31 srw-smeserver imap[3522]: sql_select option missing
Sep 19 00:12:31 srw-smeserver imap[3522]: auxpropfunc error no mechanism available
Sep 19 00:12:31 srw-smeserver imap[3522]: _sasl_plugin_load failed on sasl_auxprop_plug_init for plugin: sql

[soprom]

What I found about these errors is this:

This is a result of Cyrus trying to use the /usr/lib/sasl2/pwauxprop
SASL plugin. It appears that this plugin handles password policies such
as minimum length and whether the user has to have numbers in the
password or not. Use of this plugin requires reading in the password
server database in /var/db/authserver. When DirectoryServices calls
this plugin it’s running as root, so it can easily read the database
in. However, Cyrus runs as the cyrus user which has no ability to read
that file. As such the plugin fails. Since cyrus has no need to check
password policy, you don’t really have a way to change your pass
through Cyrus, this isn’t an issue.

Other responses on forums are saying that these lines are "Ok" but still there show up constantly...

So there's nothing wrong with your setup. It is just Cyrus SASL being rather
strange in how it treats this "error".

You might take this to the Cyrus SASL list though, so they can get an idea how
annoying this "feature" is.

[mudtoe]

Can I change the permissions on that one particular file so that the cyrus userid can access the file?  I'm not an expert Unix person, so I'm not sure how granular the file permissions are (i.e. can I simply allow access to that one user, or would I have to open up the file to everyone?).  I would like to get rid of these messages, if for no other reason than they are clogging up the message log and making it difficult to see what else is going on.  

On another topic, I've confirmed that the problem is with CYRUS not recognizing the local networks settings from SME Server.  I took the laptop that was having the problem and put it on the same subnet as the SME server, and when I did that I was able to access the IMAP folders directly, without using the proxy server.   If someone knows where and how to report this bug to the CYRUS people I'll do it, along with the bad message thing.

I installed Toltec this afternoon, and it seems to be working OK, except for the fact that I guess I should have emptied my contacts folder before I synchronized, because it duplicated everything from both computers I was testing it on (they had identical contact lists because I had manually synchronized them a week or so before), and I had two of everything.  I deleted everything and started over by just restoring one system's contact list and allowing it to be propagated to the other system.  I kind of thought Toltec had the ability to identify and discard duplicates during a synchronization, but I guess not.  

mudtoe

[soprom]

A PST file used as a folder for Toltec has to be 'detached' before connecting to a new or a different server. That explains problem you're having.

The filename referenced in the error messages don't seem to exist on SME7.

[jonroberts]

As Sophie says, you need to detach a folder in Toltec if disconnecting.  What I tend to do is ensure all my Outlook folders are synchronising via Toltec (set frequency as required to balance between update interval and amount of network traffic), so the IMAP folders are pretty much a backup of my users PST files at any time.

When re-installing Outlook or migrating or whatever, I create a blank PST and then on the first synch all data is recovered.  You can of course also Import from an exisiting PST file.  

I try and avoid connecting a PST that contains data with an IMAP folder that also already contain data because that can lead to duplicates.  Once the initial syncronisation is done, its generally pretty reliable.

[mudtoe]

Yea, I figured that out after I had three of everything.  I had exported a copy of my contacts to a separate PST file before I started messing around with Toltec, and after the duplicaiton happened, I deleted all the contacts, which synched across all three outlooks correctly, and then imported the contacts from the backup PST file into just one of the copies, and then let it synch to the others.  

After that it seems that things are working OK.  The only odd thing that I've noticed is that it seems like if you add or change a contact or calendar item, it doesn't synch right away unless you hit the synch button.  I've got toltec set to synch on startup, when selecting a folder, and every 30 minutes for calendar and tasks, and every 180 minutes for contacts.  I'm not synching email items, I just left those as generic IMAP.  

mudtoe

[jonroberts]

I've noticed is that it seems like if you add or change a contact or calendar item, it doesn't synch right away unless you hit the synch button.

You're right.  It synchs according to the frequency settings on the Toltec section of the folder properties.  Bear in mind this means synching with server, not other PCs.  So if you have two users sharing the same calendar and both set to synch every 30 minutes, in theory (worst case) it could take the best part of an hour to work around (30 mins from PC 1 to server & if other PC just synched, best part of another 30 mins till its next sync).

Having said that, its only really a problem if both users remain in that folder (i.e. both have the same calendar open).  The most likely scenario is this - You make an entry in your calendar & then carry on working.  This entry will take 30 mins to sync.  The other person wants to check your calendar, had some other folder open (e.g. their inbox, own calendar etc) & so clicking on your Calendar to view it forces a sych (with synch when selected on).  This is why I generally set frequencies on own folders higher (e.g. every 1, 5 or 10 mins) than on otherusers folders (e.g. every 15 or 30 and when selected).  Of course this will depend on the user environment & which folders are most used.

[honmanm]

Jon, thanks for doing the SME7 edition of Cyrus. Just getting into it now.

I saw your note about  storing the cyrus password in the SME database - this would be a great idea. Right now the password stuff is very messy, as the password is embedded in a number of world-readable files.

Although this morning I was inspired by your suggestion and set out to hack it into the software, it turned out to be more complicated (and with more far-reaching consequences) than I had thought.

So in the meantime I've just edited the files and changed the password in them. However if you have in mind a particular way of tackling this task, I'd be glad to do the grunt work. Through right now the SME template system is a "maze of twisty little passages, all alike" and without a bit of guidance I'm more likely to do harm than good!

I've been thinking the best way to integrate this would be to treat "cyrus" much like a normal SME user. However in this particular case the password needs to get set in the db as well as with passwd & smbpasswd. The problem with this is that passwords get set in perl-script-land (/usr/lib/perl5/site_perl/esmith/util.pm)... probably not a good idea to be messing around here.

BTW I've found it necessary to change the default quota changed 1000000 in create-accounts, user-create-cyrus and user-create-cyrus-70 (some workgroups have massive systems of shared folders).

I would have liked to change cyrus' shell to /bin/false in /etc/passwd, but unfortunately it's needed by the startup script.

I thought to change the file mode on the /etc/e-smith/actions/ files relating to cyrus, from 554 to 550. But since all of the action scripts are 554 with ownership root:root I'm guessing that this change of file mode would break all sorts of things. Well I'm going to give it a go, easy enough to change back...

One final question. There are a number of files which don't appear to be used... the user-create-unix, user-delete-unix files, as well as the user-create-cyrus-70 and user-delete-cyrus-70 files. If not needed, it would be handy to eliminate these from the file distribution.

Mark

PS I'm using the toltec-ready version of cyrus

[jonroberts]

Mark,

Glad to hear your using the contrib.  I'll help where I can, but I tend to dive into the code occassionally, undertake a crash-course refresher & then forget it all just as fast (well almost   ) - but if I can help, I will.

I thought I'd got rid of all the files that weren't needed, but I think you're right about the files you mention.  They were part of the V6 install & I thought they may have been part of the uninstall, but a quick look would suggest not.

The install script is a pretty good guide.  If its not in there, I'm pretty sure its not needed.

I never did get to grips with the password question - so any suggestions welcome.

I've got to go now, but I'll see if I can grab some time to give your comments some more thought.  In the mean time, if you have any questions - just post.

Cheers
Jon

[soprom]

<can't delete this>

[oblooblo2000]

I,
great job with your contrib !! very cool !!
Just question, when deleting a mailbox from the sme pannel, when I want to recreate it, it says

Error: The folder user.gilc already exists

Is it normal, because I have to manualy create it in cyadm !!!
I think it's because you check with

Code: [Select]

foreach my $Users (@Users)
    {
        if ( $in{'nom_boite_mail'} eq "user.$Users" )
        {
            $compte_existe = 1;
        }
    }

instead of looking in the database (variable @mailboxes)??? AM i right ??? or there is some infos that are not deleted (folders, files...)???


Now I 've got error when creating shared folder:
Please login first at /etc/e-smith/web/panels/manager/cgi-bin/sherwood line 118.
I don't know what I did!
(yet, I reinstall your contrib and it's working again, i think it should be a authentification problem, I changed the cyrus password, maibe different/iamp)

Howaever, is it possible to have the sync with the main pst outlook folders, because alert are not present, our task are not shared (or not sync with mobile device ...)

thanks for all

[soprom]

I've upgraded to 7.1 and no mails are coming in since...

/var/log/qmail/current shows this:

@40000000459dd1b32655be5c delivery 976: deferral: preline:_fatal:_unable_to_run_/usr/lib/cyrus-imapd/deliver:_access_denied/preline:_fatal:_unable_to_copy_input:_broken_pipe/
@40000000459dd1b32655d5cc status: local 0/10 remote 0/20

The file "/usr/lib/cyrus-imapd/deliver" is owned by cyrus:mail (4754)
The file "/var/qmail/bin/preline" is owned by qmail:root (0755)

username/.qmail has:

| preline -f /usr/lib/cyrus-imapd/deliver -m "$USER" "$USER"

qmail is: e-smith-qmail-1.10.0-10.el4.sme

I uninstalled cyrus and rebooted, then mail is received ok.
Then I reinstalled cyrus and rebooted...
The only error I got is:

problem connecting to "localhost", port 143: Connexion refused at ./create-accounts line 21

but the install went through with success and mail service is back, with accounts intact.

Summary:
Upgrading to SME7.1 requires to uninstall/reinstall cyrus because I don't know much about Linux.

Note that the server has a lot of addons.

[steveyb]

Hey Jon,

Nice work on the cyrus contrib, I installed the Cyrus-Toltec contrib. for SME 7.0 I’m currently running SME 7.1.3 have problems with remote pop3 access any ideas on how to fix this. Any help would be greatly appreciated.

Best Reguards,

Steve

[jonroberts]

Hi Steve,

Unfortunately I have a confession to make .. our business has taken a change of direction this year and so deflected focus away from the SME server.

While we still support our existing installs, we haven't done any new development for quite a while, and we've not even moved to 7.1 ourselves.  Personally, I've not worked on the Cyrus contrib since the move to 7.0 (see earlier postings).

So I don't think I'm going to be much help to you, I'm afraid.  If you come up with anything and want to run it past me, please feel free.  I'll do what I can but can't promise anything.

Sorry to be so little help.

Jon

[soprom]

An hypothesis is that POP access being disabled, traffic is also blocked.
Then, you might have to re-enable traffic even if POP/IMAP are disabled from the server-manager.

Example to be adapted:

Code: [Select]

config set POPcustom service status enabled access public TCPPort 110
signal-event remoteaccess-update.

[steveyb]

It worked

thanks, Soprom

[steveyb]

Help Sophie,

I spoke too soon, I still cant access pop3 email remotely from outlook. any idels.

Steve

[soprom]

My suggestion is not really applicable since POP service is disabled and the server is limited "only" to IMAP. Isn't it by design that POP is not working?

Did you get external access to IMAP though?

[steveyb]

Sophie

No external Imap access either I wonder if the sme pop3 and Imap service needs to be removed first before installing cyrus. any help will be greatly appreciated.

Thanks,

Steve

[soprom]

During the installation of the package, these lines were executed:

print "Disabling existing IMAP & POP services: ";
system("/sbin/e-smith/config setprop imap status disabled");
system("/sbin/e-smith/config setprop imaps status disabled");
system("/sbin/e-smith/config setprop pop3 status disabled");
system("/sbin/e-smith/config setprop pop3s status disabled");
print "OK. \n";