Koozali.org: home of the SME Server

E-mail archive, backup solution, huge size of MailDir

Offline bbialy

  • ***
  • 105
  • +0/-0
E-mail archive, backup solution, huge size of MailDir
« on: July 22, 2012, 11:59:12 AM »
Hi all,
I've SME server with approx 25 users.
Few of them has tons of email. (email folder have approx 20 - 30 GB) :-)
id there any contribs which can decrease size of email folder to normal size 1-2 GB but keep searchable previous e-mails.
E-mail client is MS outlook.

Local copy (on stations in *.pst file) of emails is not acceptable because of security reason.

I was reading about mailarchive but I'm not sure if it is exactly this tool i need


Any bright ideas are very very pleased.

Best Maciej
Reading with understanding is the hardest thing IN THE WORLD

Offline bbialy

  • ***
  • 105
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #1 on: July 30, 2012, 07:24:29 PM »
Guys!!!???
I need any solution for users, who has 10-20 or even 40 gigs of email and all of them are: "Extremely Important for our Business and We need access to them 24/7" :-)

Best
Bialy
Reading with understanding is the hardest thing IN THE WORLD

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #2 on: July 30, 2012, 08:20:47 PM »
bbialy

Please learn to search the forums.
Try mailarchiva

Alternatively (cheap & easy), increase the size of hard disk(s) on your server.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #3 on: July 30, 2012, 10:50:46 PM »
I would add that you should replace outlook with thunderbird

Offline idp_qbn

  • ****
  • 346
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #4 on: July 31, 2012, 04:36:49 AM »
Hi, Bbialy
It is not good office practice for your users to have mail files that big. The files are mostly  (99% probability and rising) filled with attachments taking up space. Most will never be looked at again.

This is not really a technical issue, it is a human problem.

You system allows them to not bother about filing important stuff, they can get away with just leaving it in the mailbox. This is a bit like leaving every file you may ever need sitting on your desk, just in case....and not using the filing cabinet in the corner.

If attachments are important, they can be saved to a folder and deleted from the .PST file.
With a .PST file of 30Gb, just think how long that takes to load or save and the amount of network traffic.

I know it is not easy (almost impossible) to change user's habits so there is another possibility:

Archive all emails over 12 months old (or some other period) into a .PST file called by year:
eg
archive 2009.pst
archive 2010.pst
etc
current mail is in outlook.pst
AT the end of 2012, archive all mail over 12 months and save it as archive 2011.pst

This breaks up the 30Gb into smaller chunks which the user can look at if they need to find an old email, then close it afterwards.

When OUTLOOK starts, it only loads outlook.pst (unless you have permanently told it to open an archive as well)
But they stall have those ever-so-important emails handy.

Good luck with this.
Ian
PS - consider using quotas !
___________________
Sydney, NSW, Australia

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #5 on: July 31, 2012, 05:59:27 AM »
With a .PST file of 30Gb, just think how long that takes to load or save and the amount of network traffic.

And think how likely it will become corrupted and you will lose all mail!

Use IMAP. Maildir format is *extremely* reliable.

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #6 on: July 31, 2012, 06:00:32 AM »
Archive all emails over 12 months old (or some other period) into a .PST file called by year:

No, don't do that. Avoid using .PST files. Ever.

Offline bbialy

  • ***
  • 105
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #7 on: July 31, 2012, 09:55:13 AM »
Thank you all,


idp_qbn: I think that *.pst files are not good idea because its large, and because of size vulnerable for corruption.  but thank you for advise. It was my first idea.

Stefano: As far i see you are one of the masters of SME. I understand technical advantages Thunderbird against MS outlook but it is only my point of view. Layers (i mean Lawyers :-)) are used to use Outlook because of their's habits and exchange meeting appointments with calendar, syncing calendar with mobile devices and ..... - they want to have outlook.

Mary: I would like to assure you that i know how to use search on forum and i wrote in my first post that i was thinking about MailArcihiva but this is not exactly solution i was thinking. there is a chance that I didn't understand idea of MailArchiva. in that case please correct me please and if you have any expiriences please share with us (share with me)

I saw how Symantec Vault is working and it is fantastic solution (except price) all email are tagged and searchable (attachments also) directly from MS-Outlook, but e-mails are archived elsewhere. when user click on e-mail (in fact link of email) which was looking for server is unpacking it and push full message directly into Inbox. in that case mail box with 30 gigs of e-mail is using 500-600 MB ost (ost is pst file but for MS-Exchange accounts) file locally . I was just wondering is there somewhere similar solution for opensource platforms. Maybe with kind of email client plugins or so.

BTW i know that bigger hard drive is cheap solution but performance for searching email is kind of nightmare :-)

On the other hand i can understand my Layers-Lawyers that they need access to previous emails from past years because court cases in Poland can take 3 to 5 years if case is simple :D 


Thank you once again and waitng for more

Best bbialy

Reading with understanding is the hardest thing IN THE WORLD

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #8 on: July 31, 2012, 10:08:14 AM »
bbialy

Your users can keep using Outlook, but change the mail system to use IMAP folders, so that messages reside on the server rather than in large (& potentially corruptible) pst files.
Move all the messages from the pst files to the IMAP folders (setup both in the Outlook client to allow this or use some other tool).
Increase the server hard disk size as required to cater for increased storage space requirements.
You can create archive folders based by year etc or as desired, anything will be better than having all the messages lumped in one folder.
Install Thunderbird as well to read the IMAP folders and you have great searching capability as well, for those who want something better than Outlook to search with.

Using IMAP will also allow access from anywhere and any device or client eg Outlook Express, Outlook, Thunderbird, iPhones etc, home & work PC's (all configured for IMAP), and all changes will appear on any device.
« Last Edit: July 31, 2012, 10:31:12 AM by mary »
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #9 on: July 31, 2012, 10:18:58 AM »
bbialy.. SOGo is the answer (with TB, imap, SME, calendars etc..)

if they want to use outlook with so much mail, then tell them to pay for symantec vault

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #10 on: July 31, 2012, 02:11:11 PM »
I second the recommendation to use Thunderbird for large mail stores.

Note that this does not prevent them from continuing to use Outlook - they can use outlook for all current email, but use Thunderbird when searching for old emails.

I find the Thunderbird "search" function on IMAP folders to be HUNDREDS of times faster than search in Outlook on the same message store.  I've seen this with mailboxes ranging from 1GB - 20GB, and would expect the differential to only increase with larger mailboxes.

So, you could do this:
- create a script to "archive" emails on the server to different IMAP folders by year or by year month, or use the Thunderbird "archive" add-in to do the same thing (or create a separate "archive" account for each user, and use imapsync to move old messages from the "active" to the "archive" account, or setup the archive account in Outlook and teach users how to move messages into it).

- Train your la(w)yers to keep Thunderbird open, and use that for searching archived emails.

Gotchas:
If you do setup users with both Outlook and Thunderbird, be sure to set both systems to use the same folders on the server for Drafts, Deleted Items and Sent Items.

Offline bbialy

  • ***
  • 105
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #11 on: August 01, 2012, 11:00:22 PM »
Thanks guys.
I want to assure you that implementing of thunderbird as search tool will be extremely hard.

Mary please clarify, what do you mean move mails from pst to imap folder.
My outlook accounts are configured as imap account. How to tell outlook to keep last 200-500 emails and rest on imap folder

For now for the hardest user SSD SATA 6G was the answer :D
But it is not a point :D
Reading with understanding is the hardest thing IN THE WORLD

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #12 on: August 02, 2012, 02:29:02 AM »
I want to assure you that implementing of thunderbird as search tool will be extremely hard.

I assure you, not nearly as hard as making Outlook with .PST files reliable, or making Outlook search fast.

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: E-mail archive, backup solution, huge size of MailDir
« Reply #13 on: August 02, 2012, 04:22:45 AM »
bbialy

Quote
Mary please clarify, what do you mean move mails from pst to imap folder.
My outlook accounts are configured as imap account. How to tell outlook to keep last 200-500 emails and rest on imap folder

I meant not to read mail messages from pst files, but to read them directly from the IMAP folders on the server, but you now advise you are already using IMAP, you did not tell us that previously.
Did you configure Outlook to download copies of messages, if so change Outlook configuration.
I'm not a frequent user of Outlook so cannot advise more appropriately without further investigation and knowledge of how you have set everything up.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.