All posts by donn

Mosh with iTerm2’s Tmux Integration

I have found terminal/shell nirvana on my Mac with mosh + tmux + iTerm2 Tmux Integration, but it wasn’t easy.

My dream setup was these 3 running together:

1) mosh: Runs on client and on server. An ssh replacement that is secured with AES-128 and ssh. Virtually indestructible ssh-like sessions that remain “live” even after you change IP addresses (ie. physical locations), VPNs, or network interfaces. I can login to a server and never need to re-login for *months*. Whenever I open my macbook, my shell sessions are exactly where they were before and ready for the next command.  If your IP address changes while you commute (eg. train) or you are on VPN a lot, you really should use mosh instead of ssh. It’s not just for unreliable connections, I use mosh everywhere because it saves me time.

2) tmux: Runs on the server. Replacement for the old ‘screen’ utility. It allows you to keep active windows (and panes) in a session that remains alive even after you disconnect from the remote server.

3) iTerm2’s Tmux Integration: Runs on Mac. Very cool iTerm2 feature that renders your tmux windows as native iTerm2 tabs. Allows you to scroll back through your tmux window with Macbook touchpad gestures and iTerm hotkeys. Supports iTerm2’s very quick & capable Cmd-F (Find) instead of tmux’s Find.  Supports intuitive text selection and advanced text selection (discontiguous select & copy) built into iTerm. Switching between tabs with keyboard shortcuts. Basically everything you can do in iTerm2 regular tabs, you can probably do with your tmux session rendered by iTerm2’s Tmux Integration. It rocks.


The Problem

The problem is iTerm’s Tmux integration works fine when using ssh, but not when using mosh.


The Solution

With this howto, you can build a patched version of mosh (client and server) that is compatible with iTerm’s Tmux Integration.  Mosh is a small program, so the build is very quick.

Moreover, this howto allows you to try the patched mosh binaries without touching your existing mosh installation. This is done by specifying the ‘–client’ and ‘–server’ options when running mosh.

Once you are happy with how the patched mosh is working, you can move the patched mosh to a location in your path (need to do this on both client and server).

Note, if you are on wifi all the time, you can use Eternal Terminal instead of this howto. I use hard-wired ethernet at my desk and wifi when I leave my desk (eg. walking to a meeting). It so happens, this switching of network interfaces seems to break Eternal Terminal and close my session (in my testing).

In my setup I have a macbook (mosh client) connecting to an ubuntu 16.04 server (mosh server).

First, we’ll build mosh on the Macbook (mosh-client).


Build patched mosh-client on Macbook

Create a directory for the code:

dlee-mbp:~ donn$ mkdir -pv ~/workspace/git/

dlee-mbp:~ donn$ cd ~/workspace/git/

Grab the code:

dlee-mbp:rledisez donn$ git clone

dlee-mbp:rledisez donn$ cd mosh

Checkout the patched mosh branch called “localScrollback-1.3.2”:

dlee-mbp:mosh donn$ git checkout -b localScrollback-1.3.2 origin/localScrollback-1.3.2


Use Homebrew to install dependencies:

dlee-mbp:mosh donn$ brew install protobuf automake pkg-config


Build patched mosh binaries:

dlee-mbp:mosh donn$ ./ installing ‘./ar-lib’ installing ‘./compile’ installing ‘./install-sh’ installing ‘./missing’

src/crypto/ installing ‘./depcomp’

parallel-tests: installing ‘./test-driver’



<See many lines of output>



<See many lines of output>


You don’t have to do ‘make install’ at this point. You can try the binary without installing it (see below).


But, we also need a patched mosh on the server, so next…


Build mosh on ubuntu server

Install debian package dependencies:

Note: Boost (libboost-dev) not needed for mosh 1.2+ so I didn’t install it.

sudo apt-get install automake libtool g++ protobuf-compiler libprotobuf-dev libutempter-dev libncurses5-dev zlib1g-dev libio-pty-perl libssl-dev pkg-config

Build mosh-client and mosh-server:

git clone

cd mosh

git checkout -b localScrollback-1.3.2 origin/localScrollback-1.3.2





Again, you don’t have to ‘make install’ if you just want to try things out.


Running the patched mosh

Locate the path to patched mosh-client on my Macbook:


Locate the path to patched mosh-server on my ubuntu server:



With this info, I can try my first iTerm + tmux + mosh session:

The ‘mosh’ command is found in the ‘scripts’ subdirectory of the source code directory.


dlee-mbp:mosh donn$ scripts/mosh \
--client=/Users/donn/workspace/git/ \


After logging in to, start tmux on remote host:

remote_host$ tmux -CC

[or ‘tmux -CC a’ if resuming an existing tmux session]


… and then see iTerm2 window with Tmux Integration enabled.  Cmd-T to open a new tab.



Switching to patched mosh permanently


Mac: Just put mosh and mosh-client in your path.  To see your installed version of mosh:

$ which mosh

$ which mosh-client


To see your path:

$ echo $PATH


Maybe copy your originals as mosh.orig, mosh-client.orig


Ubuntu server: Same thing but with mosh-server.  Maybe save your original as mosh-server.orig


From this point forward, be aware that normal, standard mosh clients will not be compatible with patched mosh on the server.  If you want to support both, then use the ‘–server’ option when starting a mosh session to specify which version of mosh-server will be run on the server (eg. mosh-server or mosh-server.orig).


Fixing Problems


If your session dies abruptly with an error like the following, it means your mosh-client or your mosh-server is not running the patched version of mosh; it is probably running your normal, installed version of mosh.


Assertion failed: (*i == *my_it), function diff_from, file, line 69.

Abort trap: 6




iTerm2’s Tmux Integration:


Build instructions for mosh:


Patched mosh that supports tmux control-mode (tmux -CC). Original patch by github user 4ast. Rebased on mosh 1.3.2 by rledisez:

At the time of this article, v1.3.2 was the latest stable version for download at


Original patched mosh:

Note: Commit d5bd1d31d86d4003705e69f87466aa7e10f9c5b9 “add support for resize events” is already part of mosh mainline.


“tmux integration hangs when logged in with mosh (ok w/ ssh)”


Homebrew package manager for Mac:

Bounty ($$$) for adding tmux control-mode support to mosh:


Postfix SMTP configuration: Sending (relay) email to Gmail and other Internet mail servers


Postfix Server diagram

This might be helpful for people like me who recently started learning Postfix:
If you want to eliminate the “red padlock” icon in Gmail, you do not need to get a certificate. Mail servers like Gmail don’t require you to have a certificate (aka client certificate) to connect to them over a secure TLS connection, and subsequently send mail to them (however, things like SPF TXT records and DKIM are needed to avoid Gmail marking your mail as spam).

To send mail to Gmail (and others) with TLS and get rid of the “red padlock”, you only need:

smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

…in /etc/postfix/

TLS-security-level “may” (“You *may* use TLS”) means your mail will be relayed even if the other mail server lacks TLS.  This is represented by the BLUE arrow in the diagram showing mail sent to “”. In other words, such mail will be sent unencrypted, but it will successfully reach

“smtp_*” are the parameters for the Postfix SMTP Client (the code that talks to public Internet mail servers like Gmail’s mail servers). The “smtpd_*” parameters are for the Postfix SMTP Server (the code that your users connect to when they need to send email to Gmail or some other public Internet mail server).

Make sure ca-certificates.txt exists in postfix’s chroot “jail” (on my ubuntu server it was: /var/spool/postfix/etc/ssl/certs/ca-certificates.crt). This is a database of certs of well-known CAs that your postfix server needs to know when it connects to Gmail (or other mail server). When your postfix server connects to Gmail, Gmail will present to postfix *Gmail’s server cert*, and that server cert will be signed by one of these well-known CAs.

I’m running postfix 2.11.0 on ubuntu 14.04.

You may configure smtp_tls_ciphers and smtp_tls_protocols, but the defaults are OK and recommended. The default for smtp_tls_ciphers is ‘medium’. If you do ‘high’, there’s a (small) chance some of your mail won’t reach destinations that don’t support the strongest ciphers. The default for smtp_tls_protocols is ‘!SSLv2, !SSLv3’ (disable SSL v2 and v3), which is considered safe; it allows TLSv1.

Svenn ( wrote very helpful articles about how to use LetsEncrypt. Such certs are needed when *your* remote users (email clients) need to connect to your postfix server over a secure TLS connection. That’s another article.

OwnCloud SMTP config error: “A problem occurred while sending the email” (Authentication failed)

Problem: With correct login and password, and correct SMTP settings for Gmail SMTP, owncloud “Test email settings” button fails with:

  • A problem occurred while sending the email. Please revise your settings. (Error: Failed to authenticate on SMTP server with username “” using 1 possible authenticators)

Other symptom (and hint): Gmail login works fine at other locations, home vs. work, for example.

First, in your gmail account settings, change the “Allow less secure apps” setting to ON. This is found at in section “Signing in to Google”. NOTE: This makes your gmail account less secure so you might want to create a throwaway gmail account just for SMTP (that’s what I did). I would not use my valuable gmail accounts:

  • Allow less secure apps: ON

Other things to check:

  • Ensure your owncloud user profile (not owncloud admin settings, but your actual user’s account) has an email address set. This address will receive email from owncloud for password reset email messages and email notifications.  Find your user profile in the upper-right part of the web interface: your_name > Personal.

OwnCloud admin config for smtp:

Send mode: smtp
Encryption: TLS
From address: bob
@ (domain):
Authentication method: Login
Authentication required: [checked]
Server address:
: (port) 587
Credentials:, mypassword

If you don’t use the webui, Owncloud’s {$owncloud_dir}/config/config.php has these text configuration lines for smtp:

 'mail_smtpmode' => 'smtp',
 'mail_smtpsecure' => 'tls',
 'mail_from_address' => 'bob',
 'mail_domain' => '',
 'mail_smtpauthtype' => 'LOGIN',
 'mail_smtpauth' => 1,
 'mail_smtpport' => '587',
 'mail_smtphost' => '',
 'mail_smtpname' => '',
 'mail_smtppassword' => 'mypassword',

Still doesn’t work?  I had to also do the following:

Basically, google is smart and treats logins from different geographical locations with different security restrictions (blocks).  In my case, my owncloud server was a VPS thousands of miles away from my laptop location.  So I guessed that google didn’t like that some random location (my vps) was trying to access my gmail account (even though I had “Allow less secure apps” enabled.

I found a big hint that you can “unlock” or re-auth your google account with the following url:

So basically, to prove to google that my VPS’s IP address is legit, I had to do this UnlockCaptcha from my VPS. BUT, I have no web browser (gui) on my VPS!  Except for ‘lynx’, the shell/cli based web browser!  Lynx does work for passing the UnlockCaptcha url 🙂


Juniper SSG 5 Error when upgrading via USB flash device

Problem: SSG5 (SSG 20) doesn’t upgrade via it’s usb port and reports error “USB flash is not existed. Please insert USB first!”

Solution: You need to use a usb flash drive/stick that is 4GB OR SMALLER!  And formatted FAT (aka FAT16).  FAT32 will probably work too (I haven’t tried).

More detail:

If you are on the SSG’s console, you will see the following error if you attempt to use a usb flash device bigger than 4GB:

“Usb disk size is larger than 4G.Mount failed!”

When you use a 4GB or smaller usb flash disk, you will see success:

“usb device (usb) ready.”

Again, this is on the SSG’s console.

Then you can upgrade via usb (put the *unzipped* screenos image in the *root* directory of the FAT usb drive):

ssg5-> save soft from usb ssg5ssg20.6.3.0r21.0 to flash

Then reboot:

ssg5-> reset

SSG5/SSG20 is a legacy Netscreen ScreenOS firewall/router.


My reaction to “Warren Buffett: Why stocks beat gold and bonds”

Warren Buffett recently argued for stocks vs. gold (and vs. bonds/currency) in

My take is that, yes, I like to diversify, so I have equities in my portfolio. Some equities are like hard resources and thus similar to gold. Eg. XOM and WMT have physical assets, distribution, and business that is not easy to replicate overnight (or over a decade even). Income producing real estate (and your house) is also a real, physical asset. My goal is to reduce exposure to fiat money and its related risk, and so gold and some stocks are in my portfolio.

I should add that my own personal experience is that it is difficult and risky to think I, or any "experts", can predict what products people will want and subsequently exchange for "what they produce" [as Buffett wrote]. Especially when companies fall out of favor. This is the inherent risk in stocks. When the iphone was released, I thought Blackberry was toast, but there are always "experts" and people who argue the iphone will be a flop; so I cannot be sure my crystal ball is better than theirs. Moreover, when data is released that Blackberry’s market share dropped last quarter, it’s too late to react because the market reflects such news in microseconds.

Anyone can claim that stocks are "the best" but that is because they are using a perfect crystal ball: the past performance of stocks they have cherry-picked to make their argument. This is basically what Buffett and others are doing.

It would be better if such "experts" like Buffett make a FUTURE prediction on a basket of stocks, and then we use a time machine to zip 10-20 years into the future and see if they are right. Expanding on this exercise, we ask them to also declare when they will shift in & out of each stock and how much. The fact they cannot and will not do this, shows the risk of stock investing. Even still, I invest in stocks to the best of my ability by trying to predict the things people will want in the future. Good luck to you and the "experts" in doing the same.

Where Buffett is wrong: Gold has been money to humans consistently for some 3000 years.  His attempt to group it with tulips, seashells, and the like is rather sophomoric.  Because he has been alive for only a few decades and not during the entire 3000 year span, we must give him some slack: He doesn’t have a lot of personal experience with things other than "modern" instruments like paper money and shares.

Bio for Donn Lee

Donn Lee joined Facebook in 2007. As a Sr. Network Engineer, his duties include designing networks, evaluating products, optimizing performance, and performing escalation troubleshooting. Previous to Facebook, Donn worked in Google’s Network Architecture group for four years and during rapid growth of Google’s backbone, optical, and datacenter networks. While working as a Consulting Systems Engineer at Cisco Systems (CCIE #3262) he worked on large global networks and wrote his book, Enhanced IP Services for Cisco Networks, that is published by Cisco Press. He holds a bachelor’s degree in Electrical Engineering from UCLA.

Continue reading Bio for Donn Lee

Using mutt instead of MS Outlook on an Exchange server

Problem Statement

  • MS Outlook & Exchange sucks but that’s what your company supports.
  • Mutt over IMAP is too slow. What you want is something like mutt + spool like the good email old days.
  • Just about every GUI-based email client sucks (any OS). They are too slow compared to mutt.

Design Goals

  • Use mutt from local disk to keep mutt fast.
  • Keep mail on corporate IMAP/Exchange servers so it is still backed up & maintained by the IT dept.
  • Use IMAP because Exchange servers support it.
  • No manual sync or push. Automate everything so all you have to do is use mutt.
  • Allow straight-forward use of GUI email clients should you ever need them.

Design Concepts

  • OfflineIMAP synchronizes mail folders between the Exchange server and your machine (mutt’s local disk).
  • You read mail in mutt. Mutt reads mail from disk.
  • You write mail filtering rules in fdm.  fdm filters mail into folders to your liking.
  • Use cron to automate syncs and filtering jobs behind the scenes.


Step 1: Install ubuntu packages: mutt, offlineimap, fdm, msmtp (ubuntu 10.04 LTS package names)

Step 2: Config mutt to send mail using msmtp


account default
port 587
auth login
user donn
password s00perSekrit 
tls on
tls_starttls on
tls_certcheck off
#tls_fingerprint FE:39:F9:B4:64:31:0E:DF:31:51:72:DA:A7:4F:35:4B
logfile ~/.msmtp.log

Paste view:


set sendmail=/usr/bin/msmtp

Step 3: Setup Maildir directory

Add to ~/.muttrc:

set mbox_type=Maildir
set folder="~/Maildir"
mailboxes `echo -n "+ "; for file in ~/Maildir/*; do box=$(basename "$file"); echo -n "\"+$box\" "; done`
set spoolfile="~/Maildir/work/INBOX"
set postponed="+Drafts"
set record="+work/Sent"

Paste view:

Step 4: Setup OfflineIMAP

The key point of offlineimap: Mutt can read your messages from local hard disk, instead of over the network. This greatly reduces the lag / delay when reading messages quickly.

OfflineIMAP syncs your ~/Maildir/work message base with MS Exchange over IMAP. So, if you read a message in mutt, it will be marked read in ~/Maildir, and then offlineimap will sync this change with Exchange. If you later check your inbox with Thunderbird, OWA, Outlook, or other client, that message will have ‘read’ status (awesome). OfflineIMAP downloads new mail from Exchange and stores them in ~/Maildir/work/INBOX, but this is really the syncing process: Exchange has the new messages, ~/Maildir/work/INBOX does not, until after a sync.


accounts = Work
[Account Work]
localrepository = Local
remoterepository = Remote
[Repository Local]
type = Maildir
localfolders = ~/Maildir/work
[Repository Remote]
type = IMAP
remotehost =
ssl = yes
remoteuser = donn
remotepass = myPass
# Folders to skip during sync.
folderfilter = lambda foldername: foldername not in ['Deleted Items', 'Contacts', 'Calendar', 'Trash', 'MyFolderOfReallyOldMail']

Paste view:

Step 5: Setup FDM

FDM is used to filter mail between your INBOX and your other Maildir “folders”. For example, I filter mailing-list email to a separate folder. OfflineIMAP puts ALL incoming mail into INBOX, then FDM moves messages from INBOX to other folders based on your rules. All of this FDM work is done on local Maildir subdirs (local disk). I also have FDM move (ie. archive) all email over 30 days to a local, “old mail” folder.

WARNING: Be extra cautious as you develop your FDM rules. If you make a typo (eg. regex typo), you could accidentally delete incoming mail. Use the -n switch (test syntax) and -v (verbose output) to check your rules carefully.

#unmatched keep default. A lot of these, just to be safe.
set unmatched-mail keep
# Delivery actions.
action "INBOX" maildir "%h/Maildir/work/INBOX"
action "widgets-list" maildir "%h/Maildir/work/widgets-list"
action "ix_email" maildir "%h/Maildir/work/ix_email"
action "inbox_overflow" maildir "%h/Maildir/work/inbox_overflow"
#Bread and Butter INBOX account only operates on one folder
account "WORK" maildir "%h/Maildir/work/INBOX"
# Match regex's are *not* case-sensitive by default.
match account "WORK" {
 match "^subject:.*\\[ubuntu-widgets\\].*" in headers {
 match all action "widgets-list"
 # Negative match regex.
 match "^(to:|cc:)(.*|.*tech-l@ams-ix.*)" in headers {
 match "^From:(?!.**)" in headers {
 # If not from linx admins, move it.
 match all action "ix_email"
 # Move older msgs to overflow box so fdm doesn't have to process them over and over.
 # My email is donn @
 match age > 1 months {
 match "^To:(?!.*donn@.*)" in headers {
 # If not to-donn, archive the old mail to overflow box.
 match all action "inbox_overflow"
 # Last catchall match term.
 match unmatched action keep
# Send all mail to inbox.
match unmatched action keep

Paste view:

Step 6: Setup mutt basics


set hostname=""
set realname="Donn Lee"
set from=""
set envelope_from=yes
set hidden_host=yes
set use_domain=yes
set pager_stop=yes
my_hdr From:
set attribution="%n <%a> wrote on %{%a} [%{%Y-%b-%d %H:%M:%S %Z}]:"
set strict_threads = no
set date_format="!%a, %b %d, %Y at %I:%M:%S%p %Z"
set index_format="%4C %Z[%[%a %m/%d %H:%M]] %-16.16L%?X? [%X]? (%?l?%4l&%4c?) %s"
set pager_index_lines=20
# Change default subject format for fwd'd msgs.
set forward_format="Fwd: %s"
# Use emacs to compose mail.
set editor = "/usr/bin/emacs -nw %s"
# Create a nice status bar.
set status_format=" %r %b %f New:%n Del:%d TotMsgs:%m Tag:%t Flag:%F Size:%l %> Lim:%V (%P)"
# Ignore all lines by default
ignore *
# Set what I want to see
unignore from to cc subject date reply-to mail-followup-to x-url organisation organization x-mailer user-agent xmail-sent-to
hdr_order from to cc subject date reply-to mail-followup-to x-url organisation organization x-mailer user-agent xmail-sent-to
# Create a cache for performance.
set header_cache="~/.mutt_cache"
set maildir_header_cache_verify="yes"
set header_cache_pagesize="65536"
# And all your other mutt goodies...

Paste view:

Step 7: Read messages that are html formatted

Add to muttrc:

# View html email
# Must also add these two lines to ~/.mailcap
# text/html; links %s; nametemplate=%s.html
# text/html; links -dump %s; nametemplate=%s.html; copiousoutput
auto_view text/html

Paste view:

Step 8: Opening email attachments

I use Outlook Web App (OWA) to open an attachment (eg. a Powerpoint file). I always have a browser up and OWA is usually loaded in one of the tabs.

Continue reading Using mutt instead of MS Outlook on an Exchange server