All posts by donn

Postfix SMTP configuration: Sending (relay) email to Gmail and other Internet mail servers

 

Postfix Server diagram

This might be helpful for people like me who recently started learning Postfix:
If you want to eliminate the “red padlock” icon in Gmail, you do not need to get a certificate. Mail servers like Gmail don’t require you to have a certificate (aka client certificate) to connect to them over a secure TLS connection, and subsequently send mail to them (however, things like SPF TXT records and DKIM are needed to avoid Gmail marking your mail as spam).

To send mail to Gmail (and others) with TLS and get rid of the “red padlock”, you only need:

smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

…in /etc/postfix/main.cf

TLS-security-level “may” (“You *may* use TLS”) means your mail will be relayed even if the other mail server lacks TLS.  This is represented by the BLUE arrow in the diagram showing mail sent to “example.com”. In other words, such mail will be sent unencrypted, but it will successfully reach example.com.

“smtp_*” are the parameters for the Postfix SMTP Client (the code that talks to public Internet mail servers like Gmail’s mail servers). The “smtpd_*” parameters are for the Postfix SMTP Server (the code that your users connect to when they need to send email to Gmail or some other public Internet mail server).

Make sure ca-certificates.txt exists in postfix’s chroot “jail” (on my ubuntu server it was: /var/spool/postfix/etc/ssl/certs/ca-certificates.crt). This is a database of certs of well-known CAs that your postfix server needs to know when it connects to Gmail (or other mail server). When your postfix server connects to Gmail, Gmail will present to postfix *Gmail’s server cert*, and that server cert will be signed by one of these well-known CAs.

I’m running postfix 2.11.0 on ubuntu 14.04.

You may configure smtp_tls_ciphers and smtp_tls_protocols, but the defaults are OK and recommended. The default for smtp_tls_ciphers is ‘medium’. If you do ‘high’, there’s a (small) chance some of your mail won’t reach destinations that don’t support the strongest ciphers. The default for smtp_tls_protocols is ‘!SSLv2, !SSLv3’ (disable SSL v2 and v3), which is considered safe; it allows TLSv1.

Svenn (https://www.svennd.be) wrote very helpful articles about how to use LetsEncrypt. Such certs are needed when *your* remote users (email clients) need to connect to your postfix server over a secure TLS connection. That’s another article.

OwnCloud SMTP config error: “A problem occurred while sending the email” (Authentication failed)

Problem: With correct login and password, and correct SMTP settings for Gmail SMTP, owncloud “Test email settings” button fails with:

  • A problem occurred while sending the email. Please revise your settings. (Error: Failed to authenticate on SMTP server with username “bob@gmail.com” using 1 possible authenticators)

Other symptom (and hint): Gmail login works fine at other locations, home vs. work, for example.

First, in your gmail account settings, change the “Allow less secure apps” setting to ON. This is found at https://myaccount.google.com in section “Signing in to Google”. NOTE: This makes your gmail account less secure so you might want to create a throwaway gmail account just for SMTP (that’s what I did). I would not use my valuable gmail accounts:

  • Allow less secure apps: ON

Other things to check:

  • Ensure your owncloud user profile (not owncloud admin settings, but your actual user’s account) has an email address set. This address will receive email from owncloud for password reset email messages and email notifications.  Find your user profile in the upper-right part of the web interface: your_name > Personal.

OwnCloud admin config for smtp:

Send mode: smtp
Encryption: TLS
From address: bob
@ (domain): gmail.com
Authentication method: Login
Authentication required: [checked]
Server address: smtp.gmail.com
: (port) 587
Credentials: bob@gmail.com, mypassword

If you don’t use the webui, Owncloud’s {$owncloud_dir}/config/config.php has these text configuration lines for smtp:

 'mail_smtpmode' => 'smtp',
 'mail_smtpsecure' => 'tls',
 'mail_from_address' => 'bob',
 'mail_domain' => 'gmail.com',
 'mail_smtpauthtype' => 'LOGIN',
 'mail_smtpauth' => 1,
 'mail_smtpport' => '587',
 'mail_smtphost' => 'smtp.gmail.com',
 'mail_smtpname' => 'bob@gmail.com',
 'mail_smtppassword' => 'mypassword',

Still doesn’t work?  I had to also do the following:

Basically, google is smart and treats logins from different geographical locations with different security restrictions (blocks).  In my case, my owncloud server was a VPS thousands of miles away from my laptop location.  So I guessed that google didn’t like that some random location (my vps) was trying to access my gmail account (even though I had “Allow less secure apps” enabled.

I found a big hint that you can “unlock” or re-auth your google account with the following url:

https://accounts.google.com/UnlockCaptcha

So basically, to prove to google that my VPS’s IP address is legit, I had to do this UnlockCaptcha from my VPS. BUT, I have no web browser (gui) on my VPS!  Except for ‘lynx’, the shell/cli based web browser!  Lynx does work for passing the UnlockCaptcha url 🙂

#

Juniper SSG 5 Error when upgrading via USB flash device

Problem: SSG5 (SSG 20) doesn’t upgrade via it’s usb port and reports error “USB flash is not existed. Please insert USB first!”

Solution: You need to use a usb flash drive/stick that is 4GB OR SMALLER!  And formatted FAT (aka FAT16).  FAT32 will probably work too (I haven’t tried).

More detail:

If you are on the SSG’s console, you will see the following error if you attempt to use a usb flash device bigger than 4GB:

“Usb disk size is larger than 4G.Mount failed!”

When you use a 4GB or smaller usb flash disk, you will see success:

“usb device (usb) ready.”

Again, this is on the SSG’s console.

Then you can upgrade via usb (put the *unzipped* screenos image in the *root* directory of the FAT usb drive):

ssg5-> save soft from usb ssg5ssg20.6.3.0r21.0 to flash

Then reboot:

ssg5-> reset

SSG5/SSG20 is a legacy Netscreen ScreenOS firewall/router.

 

My reaction to “Warren Buffett: Why stocks beat gold and bonds”

Warren Buffett recently argued for stocks vs. gold (and vs. bonds/currency) in http://finance.fortune.cnn.com/2012/02/09/warren-buffett-berkshire-shareholder-letter

My take is that, yes, I like to diversify, so I have equities in my portfolio. Some equities are like hard resources and thus similar to gold. Eg. XOM and WMT have physical assets, distribution, and business that is not easy to replicate overnight (or over a decade even). Income producing real estate (and your house) is also a real, physical asset. My goal is to reduce exposure to fiat money and its related risk, and so gold and some stocks are in my portfolio.


I should add that my own personal experience is that it is difficult and risky to think I, or any "experts", can predict what products people will want and subsequently exchange for "what they produce" [as Buffett wrote]. Especially when companies fall out of favor. This is the inherent risk in stocks. When the iphone was released, I thought Blackberry was toast, but there are always "experts" and people who argue the iphone will be a flop; so I cannot be sure my crystal ball is better than theirs. Moreover, when data is released that Blackberry’s market share dropped last quarter, it’s too late to react because the market reflects such news in microseconds.

Anyone can claim that stocks are "the best" but that is because they are using a perfect crystal ball: the past performance of stocks they have cherry-picked to make their argument. This is basically what Buffett and others are doing.

It would be better if such "experts" like Buffett make a FUTURE prediction on a basket of stocks, and then we use a time machine to zip 10-20 years into the future and see if they are right. Expanding on this exercise, we ask them to also declare when they will shift in & out of each stock and how much. The fact they cannot and will not do this, shows the risk of stock investing. Even still, I invest in stocks to the best of my ability by trying to predict the things people will want in the future. Good luck to you and the "experts" in doing the same.


Where Buffett is wrong: Gold has been money to humans consistently for some 3000 years.  His attempt to group it with tulips, seashells, and the like is rather sophomoric.  Because he has been alive for only a few decades and not during the entire 3000 year span, we must give him some slack: He doesn’t have a lot of personal experience with things other than "modern" instruments like paper money and shares.
 

Bio for Donn Lee

Donn Lee joined Facebook in 2007. As a Sr. Network Engineer, his duties include designing networks, evaluating products, optimizing performance, and performing escalation troubleshooting. Previous to Facebook, Donn worked in Google’s Network Architecture group for four years and during rapid growth of Google’s backbone, optical, and datacenter networks. While working as a Consulting Systems Engineer at Cisco Systems (CCIE #3262) he worked on large global networks and wrote his book, Enhanced IP Services for Cisco Networks, that is published by Cisco Press. He holds a bachelor’s degree in Electrical Engineering from UCLA.

Image
Continue reading Bio for Donn Lee

Using mutt instead of MS Outlook on an Exchange server

Problem Statement

  • MS Outlook & Exchange sucks but that’s what your company supports.
  • Mutt over IMAP is too slow. What you want is something like mutt + spool like the good email old days.
  • Just about every GUI-based email client sucks (any OS). They are too slow compared to mutt.

Design Goals

  • Use mutt from local disk to keep mutt fast.
  • Keep mail on corporate IMAP/Exchange servers so it is still backed up & maintained by the IT dept.
  • Use IMAP because Exchange servers support it.
  • No manual sync or push. Automate everything so all you have to do is use mutt.
  • Allow straight-forward use of GUI email clients should you ever need them.

Design Concepts

  • OfflineIMAP synchronizes mail folders between the Exchange server and your machine (mutt’s local disk).
  • You read mail in mutt. Mutt reads mail from disk.
  • You write mail filtering rules in fdm.  fdm filters mail into folders to your liking.
  • Use cron to automate syncs and filtering jobs behind the scenes.

Howto

Step 1: Install ubuntu packages: mutt, offlineimap, fdm, msmtp (ubuntu 10.04 LTS package names)

Step 2: Config mutt to send mail using msmtp

~/.msmtprc

account default
host smtp.mymailserver.com
port 587
from donn@bigcorp.com
auth login
user donn
password s00perSekrit 
tls on
tls_starttls on
tls_certcheck off
#Or:
#tls_fingerprint FE:39:F9:B4:64:31:0E:DF:31:51:72:DA:A7:4F:35:4B
logfile ~/.msmtp.log

Paste view: http://pastie.org/9129773

~/.muttrc

set sendmail=/usr/bin/msmtp

Step 3: Setup Maildir directory

Add to ~/.muttrc:

set mbox_type=Maildir
set folder="~/Maildir"
mailboxes `echo -n "+ "; for file in ~/Maildir/*; do box=$(basename "$file"); echo -n "\"+$box\" "; done`
set spoolfile="~/Maildir/work/INBOX"
set postponed="+Drafts"
set record="+work/Sent"

Paste view: http://pastie.org/9129744

Step 4: Setup OfflineIMAP

The key point of offlineimap: Mutt can read your messages from local hard disk, instead of over the network. This greatly reduces the lag / delay when reading messages quickly.

OfflineIMAP syncs your ~/Maildir/work message base with MS Exchange over IMAP. So, if you read a message in mutt, it will be marked read in ~/Maildir, and then offlineimap will sync this change with Exchange. If you later check your inbox with Thunderbird, OWA, Outlook, or other client, that message will have ‘read’ status (awesome). OfflineIMAP downloads new mail from Exchange and stores them in ~/Maildir/work/INBOX, but this is really the syncing process: Exchange has the new messages, ~/Maildir/work/INBOX does not, until after a sync.

offlineimaprc:

[general]
accounts = Work
 
[Account Work]
localrepository = Local
remoterepository = Remote
 
[Repository Local]
type = Maildir
localfolders = ~/Maildir/work
 
[Repository Remote]
type = IMAP
remotehost = mail.mymailserver.com
ssl = yes
remoteuser = donn
remotepass = myPass
# Folders to skip during sync.
folderfilter = lambda foldername: foldername not in ['Deleted Items', 'Contacts', 'Calendar', 'Trash', 'MyFolderOfReallyOldMail']

Paste view: http://pastie.org/9129763

Step 5: Setup FDM

FDM is used to filter mail between your INBOX and your other Maildir “folders”. For example, I filter mailing-list email to a separate folder. OfflineIMAP puts ALL incoming mail into INBOX, then FDM moves messages from INBOX to other folders based on your rules. All of this FDM work is done on local Maildir subdirs (local disk). I also have FDM move (ie. archive) all email over 30 days to a local, “old mail” folder.

WARNING: Be extra cautious as you develop your FDM rules. If you make a typo (eg. regex typo), you could accidentally delete incoming mail. Use the -n switch (test syntax) and -v (verbose output) to check your rules carefully.

#unmatched keep default. A lot of these, just to be safe.
set unmatched-mail keep
 
# Delivery actions.
action "INBOX" maildir "%h/Maildir/work/INBOX"
action "widgets-list" maildir "%h/Maildir/work/widgets-list"
action "ix_email" maildir "%h/Maildir/work/ix_email"
action "inbox_overflow" maildir "%h/Maildir/work/inbox_overflow"
 
#Bread and Butter INBOX account only operates on one folder
account "WORK" maildir "%h/Maildir/work/INBOX"
 
# Match regex's are *not* case-sensitive by default.
match account "WORK" {
 match "^subject:.*\\[ubuntu-widgets\\].*" in headers {
 match all action "widgets-list"
 }
 # Negative match regex.
 match "^(to:|cc:)(.*linx.net|.*tech-l@ams-ix.*)" in headers {
 match "^From:(?!.*linx.net.*)" in headers {
 # If not from linx admins, move it.
 match all action "ix_email"
 }
 }
 # Move older msgs to overflow box so fdm doesn't have to process them over and over.
 # My email is donn @ bigcorp.com
 match age > 1 months {
 match "^To:(?!.*donn@.*)" in headers {
 # If not to-donn, archive the old mail to overflow box.
 match all action "inbox_overflow"
 }
 }
 # Last catchall match term.
 match unmatched action keep
}
# Send all mail to inbox.
match unmatched action keep

Paste view: http://pastie.org/9129776

Step 6: Setup mutt basics

muttrc:

set hostname="bigcorp.com"
set realname="Donn Lee"
set from="donn@bigcorp.com"
set envelope_from=yes
set hidden_host=yes
set use_domain=yes
set pager_stop=yes
my_hdr From: donn@bigcorp.com
set attribution="%n <%a> wrote on %{%a} [%{%Y-%b-%d %H:%M:%S %Z}]:"
set strict_threads = no
set date_format="!%a, %b %d, %Y at %I:%M:%S%p %Z"
set index_format="%4C %Z[%[%a %m/%d %H:%M]] %-16.16L%?X? [%X]? (%?l?%4l&%4c?) %s"
set pager_index_lines=20
# Change default subject format for fwd'd msgs.
set forward_format="Fwd: %s"
# Use emacs to compose mail.
set editor = "/usr/bin/emacs -nw %s"
# Create a nice status bar.
set status_format=" %r %b %f New:%n Del:%d TotMsgs:%m Tag:%t Flag:%F Size:%l %> Lim:%V (%P)"
# Ignore all lines by default
ignore *
# Set what I want to see
unignore from to cc subject date reply-to mail-followup-to x-url organisation organization x-mailer user-agent xmail-sent-to
hdr_order from to cc subject date reply-to mail-followup-to x-url organisation organization x-mailer user-agent xmail-sent-to
# Create a cache for performance.
# http://www.mutt.org/doc/devel/manual.html#header-cache
set header_cache="~/.mutt_cache"
set maildir_header_cache_verify="yes"
set header_cache_pagesize="65536"
# And all your other mutt goodies...

Paste view: http://pastie.org/9129782

Step 7: Read messages that are html formatted

Add to muttrc:

# View html email
# Must also add these two lines to ~/.mailcap
# text/html; links %s; nametemplate=%s.html
# text/html; links -dump %s; nametemplate=%s.html; copiousoutput
auto_view text/html

Paste view: http://pastie.org/9129786

Step 8: Opening email attachments

I use Outlook Web App (OWA) to open an attachment (eg. a Powerpoint file). I always have a browser up and OWA is usually loaded in one of the tabs.

#
Continue reading Using mutt instead of MS Outlook on an Exchange server

Prius accident

Prius bumper damage 11/3/2008. Click to enlarge photo.  Old guy didn’t notice his light was red and almost T-boned the Prius. By luck, a police car was immediately behind the Prius and saw everything.
Image