Home

17 Mar 2014

iCal attachments and emailing woes

Calendar notifications are becoming more common in collaborative applications. Scheduling systems are easy to build but difficult to integrate as there are large number of clients and service providers. Our product went through a painful time regarding integrating with various email clients and online services regarding iCalendar integration.

This post is going to make sure, I document them for anybody who is facing similar problems. If you have any questions regarding specifics, I will be happy to respond on a stackoverflow question or comment below.

I am not going to create a HOW-TO list of articles here - but document things I bumped against. So if you’re looking for a detailed tutorial, you’ll need to research on your own.

Brief about our solution

Our application is built on PHP 5.4 + Symfony 1.4. We are using iCalcreator library for the generation of iCal files. In my research, this is the most comprehensive standard compliant library out there for PHP. It supports RFC2445 and RFC5545. If you’re on PHP platform and looking for iCal generating library, you’re search ends here.

Our calendar system generated events between only two parties; organiser and attendee. So the event generation is pretty straightforward. We generate iCal files and just dispatch as attachment. Fairly simple. Still I managed to bump against couple of nasty issues on the way.

Issue 1 - Email clients won’t welcome us

iCal files are creates a tricky issue for client compatibility. You can respond to users of your website depending upon their environment. You can define custom CSS classes, generate responses based on browser, location etc. When you’re dealing with iCal files, you don’t get to do any of that. Your file is attached and you let the chips fall wherever they might be. So there is no alternative to extensive testing with various browsers, email clients and email service providers. Our focus was on -

  • Microsoft Exchange, Outlook (Desktop), Outlook Web Access (OWA) with version 2007, 2010, 2013
  • Thunderbird with iCal plugin
  • Google Mail
  • Yahoo Mail
  • Outlook.com

These clients provided a broad spectrum of variation which gives us enough comfort in our solution.

We wanted email clients to recognise calendar emails and show action buttons to respond to the event. Usually you’ll see “Yes”, “Maybe” or “No” buttons with additional context information in a special bar by these email clients.

Our emails were not able to achieve this feature. We investigated a lot but didn’t happen. My investigations told me you’ll need to do few things while attaching iCal file to email before clients will recognise it. These were -

  • Email body header should be “multipart/alternative”
  • Optionally attach the ics file as well.
  • Attach iCal file to email with following specifications -
    • Encoding base64 or 7-bit
    • Content type “text/calendar”
    • Header parameter “method=REQUEST”

For Swift mailer sample code might be like this -

$messageObject = Swift_Message::newInstance();
$messageObject->setContentType("multipart/alternative");
$messageObject->addPart("Email message body goes here", "text/html");

$messageObject->setSubject("Subject line goes here")
  ->setFrom("Sachin Dharmapurikar");

$messageObject->setTo(array('johnsmith@example.com'));
$ics_content = file_get_contents("/tmp/sample.ics");
$ics_attachment = Swift_Attachment::newInstance()
  ->setBody(trim($ics_content))
  ->setEncoder(Swift_Encoding::get7BitEncoding());
$headers = $ics_attachment->getHeaders();
$content_type_header = $headers->get("Content-Type");
$content_type_header->setValue("text/calendar");
$content_type_header->setParameters(array(
  'charset' => 'UTF-8',
  'method' => 'REQUEST'
));
$headers->remove('Content-Disposition');
$messageObject->attach($ics_attachment);

$mailObject = Swift_Mailer::newInstance($transportObject);
$mailObject->send($messageObject);

If you look carefully, the message is marked with headers appropriately and the ical attachment is inline rather than as file.

If you look at the source of email generated with above type of code -

Delivered-To: johnsmith@example.com
Received: by 10.220.78.135 with SMTP id l7csp244293vck;
        Thu, 13 Mar 2014 04:04:02 -0700 (PDT)
X-Received: by 10.224.151.130 with SMTP id c2mr1291334qaw.67.1394708642049;
        Thu, 13 Mar 2014 04:04:02 -0700 (PDT)
Return-Path: <00000144bb1cf5e9-fd06-4945-b4fc-386af484b339-000000@amazonses.com>
Received: from a8-35.smtp-out.amazonses.com (a8-35.smtp-out.amazonses.com. [54.240.8.35])
        by mx.google.com with ESMTP id f6si1004430qap.152.2014.03.13.04.04.01
        for <johnsmith@example.com>;
        Thu, 13 Mar 2014 04:04:02 -0700 (PDT)
Received-SPF: pass (google.com: domain of 00000144bb1cf5e9-fd06-4945-b4fc-386af484b339-000000@amazonses.com designates 54.240.8.35 as permitted sender) client-ip=54.240.8.35;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of 00000144bb1cf5e9-fd06-4945-b4fc-386af484b339-000000@amazonses.com designates 54.240.8.35 as permitted sender) smtp.mail=00000144bb1cf5e9fd06-4945-b4fc-386af484b339-000000@amazonses.com
Return-Path: 00000144bb1cf5e9fd06-4945-b4fc-386af484b339-000000@amazonses.com
Message-ID: <00000144bb1cf5e9fd06-4945-b4fc-386af484b339-000000@email.amazonses.com>
Date: Thu, 13 Mar 2014 11:04:01 +0000
Subject: Calendar appointment
From: Support <noreply@serviceprovider.com>
To: johnsmith@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="_=_swift_v4_1394708669_02698d0808f167b864966fe96816eec3_=_"
X-SES-Outgoing: 2014.03.13-54.240.8.35


--_=_swift_v4_1394708669_02698d0808f167b864966fe96816eec3_=_
Content-Type: multipart/alternative;
 boundary="_=_swift_v4_1394708669_9fe42676490416322d81ca192ce8c5f7_=_"


--_=_swift_v4_1394708669_9fe42676490416322d81ca192ce8c5f7_=_
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<div>Hello World!</div>

--_=_swift_v4_1394708669_9fe42676490416322d81ca192ce8c5f7_=_--


--_=_swift_v4_1394708669_02698d0808f167b864966fe96816eec3_=_
Content-Type: text/calendar; charset=UTF-8; method=REQUEST
Content-Transfer-Encoding: 7bit

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Service Provider Inc//NONSGML kigkonsult.se iCalcreator 2.18//
CALSCALE:GREGORIAN
METHOD:REQUEST
X-WR-TIMEZONE:America/New_York
BEGIN:VEVENT
UID:20140313T070425EDT-3998irwScV@Service Provider Inc
DTSTAMP:20140313T110425Z
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE
 ;CN=John Smith:MAILTO:johnsmith@example.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=Jane Smith:MAILTO:janesmith@example.com
DESCRIPTION:
DTSTART:20140316T103000Z
DTEND:20140316T110000Z
LOCATION:Meeting venue
ORGANIZER:MAILTO:johnsmith@example.com
SEQUENCE:0
SUMMARY:Meeting about discussion of future
END:VEVENT
END:VCALENDAR

--_=_swift_v4_1394708669_02698d0808f167b864966fe96816eec3_=_--

This works with all the popular email clients. This was the first problem we encountered.

Issue 2 - Despite of writing above code, email clients still didn’t respect us

Now thats a bummer. I spent lot of time and made sure that all the required parameters are satisfied but still no luck with showing calendar invitations properly. Spent lot of time reading original messages sent to the users and figured out that the email headers and everything is stripped out by our vendor.

Our email sending API vendor used to parse message and headers, and convert them to generic message for god knows reasons. Then I decided to switch email sending API from current vendor to Amazon SES. SES allows you to send raw messages which makes life extremely easy. That solved our problem completely.

So if you’re using above code and your email headers are not appearing correctly, try a different way to send your email. It might work for you.

Most of the modern email service providers allow you read the source of the email received. Keep checking that for the headers. That is incredebly helpful.

Issue 3 - Occasionly our users received error in Outlook

Some of outlook users started receiving “not supported calendar attachment.ics” attachment with the emails. This was weird as nobody else receive this error.

Again, I started investigating this issue and figured out that there is some issue with RRULE directive in Outlook if the iCal file is created from specific clients e.g. Lotus Notes. Our case didn’t have that issue as we don’t deal with RRULE (recurring appointments) or using Lotus Notes to generate iCal files.

Later I figured out that due to a bug in our code, we were occasionly sending 0 byte attachments. This was treated like no attachment in modern email clients but Outlook gave this error message to the users.

Issue 4 - Outlook used to show 3-4 hour differences in meeting timings

This one was the most weird issue I faced. Google, iCal on mac or Yahoo used to show correct meeting timings but Outlook used to show timings with 3-4 hour time difference. This one was a huge problem.

The reason for this problem lies in formatting the dtstart and dtend attributes of vevent.

There is a good example here. You’ll need to format time in the appropriate timezone. You’ll also need to specify X-WR-TIMEZONE property in the vcalendar section of the file.

Tags - | programming | php | symfony | iCalcreator | iCal | calendar | attachment |

29 Sep 2012

Symfony Worker Tasks

Every large application needs background worker tasks for sending emails, cleaning up files or archiving content. We are using symfony 1.4 for our current project. In symfony, this can be achieved by creating tasks. Tasks can be created by generator. You can trigger these using cron, hooked with a queuing system or invoked manually.

In my experience, writing a symfony task in general is very easy and fun. When you want to run a task for long time, that’s tricky! We created a task to send emails asynchronously. Our solution involves three components.

  1. Web Application this is where we decide to send a notification email. Create email text and then save it in a persistent store. Notify access identifier to RabbitMQ to dispatch using worker.
  2. sfRabbit this plugin allows to create consumer class, invoked by a symfony task. Each event consumer has a callback. When qualifying event is received, execute method of the callback is invoked.
  3. Email Dispatcher we use Amazon SES for sending emails. We wrote a dispatcher class which handles the responsibility of fetching the content from persistent storage and then dispatch email.

Looks like a simple process which shouldn’t be difficult to implement. At least I thought so.

I quickly created a database backed symfony task. It will use doctrine to connect to MySQL and fetch email information, dispatch emails and I can go have a beer. Not so easily.

Issue 1

After some time, I saw an error message in log file -

MySQL server has gone away

So this means, there was some connection idle for more than wait_time configured in MySQL. In my case it was set to 28800 (8 hours). This was long enough duration to hit at least one query. It didn’t happen over the night and the task stopped sending emails. Snippet from my task code -

protected function execute($arguments = array(), $options = array())

$databaseManager = new sfDatabaseManager($this->configuration);

$connection = $databaseManager->getDatabase($options['connection'])->getConnection();

// more code with database goes here

}

When you instantiate sfDatabaseManager in symfony task, it will create a connection using provided configuration. This connection will be killed if it idles for wait_time. If you kept issuing ping queries, then you don’t have that issue. Upon using the connection to query database wait_time will keep resetting to 0 and you’ll reuse the connection again and again.

This is exactly what happens when symfony is processing a web request. Only 1 connection is opened and that is reused throughout the lifecycle of the request. At the end of the request, the connection is closed and that’s how no memory leaks or connection leaks happen.

To tackle this connection getting killed, I just started creating sfDatabaseManager every time I received a message. Just like the symfony web request procedure.

$databaseManager = new sfDatabaseManager($this-&gt;configuration);
// Use database to do operations
$databaseMangare->shutdown();

$databaseManager->shutdown() is actually supposed to close all the active connections and then shut itself down. Sounds reasonable and worked for me until I hit second issue.

Issue# 2

That trick worked for a while and I bumped across another MySQL error -

Too many connections

I was surprised. Database manager is getting shutdown, I debugged code and I saw that each connection is issued close call. Why there is database connection leak?! I kept investigating.

What I found was very interesting.

Symfony task used sfDatabaseManager to manage database activity. I used doctrine so sfDatabaseManager internally used Doctrine_Manager. Doctrine uses PDO to finally connect to the database server. I found that right way to manage a PDO connection is simple -

$dbh = new PDO('mysql:host=localhost;dbname=test', $user, $pass);
$dbh = null;

Setting a PDO reference to null will close the connection. Simple enough.

From documentation -

Upon successful connection to the database, an instance of the PDO class is returned to your script. The connection remains active for the lifetime of that PDO object. To close the connection, you need to destroy the object by ensuring that all remaining references to it are deleted -- you do this by assigning NULL to the variable that holds the object. If you don't do this explicitly, PHP will automatically close the connection when your script ends.

My interpretation says, you assign null to the PDO (which symfony code did) and then PDO will close connection immidiately. If you didn’t assign null then connection will be held until script ends. This is not how it happens. If you assign null it doesn’t get freed immediately. When the script ends, that connection will be actually closed. Till that time, MySQL will think of it as an idle connection. Bummer!

I didn’t find any easy way to force close the connection immediately.

How I solved this issue?

After lot of Googling, reading I didn’t find anything which can be used as solution. I had two options:

  • Increase wait_time – Increasing wait_time to something like 72 hours. No matter what happens, I will send 1 email in 72 hours and that will keep the connection alive. I didn’t prefer this solution as it will just delay the problem instead of fixing it. This will allow me to use my first solution of instantiating sfDatabaseManager only once and keep re-using the connection.

  • **Reduce wait_time ** – Reduce wait_time and use second solution of creating sfDatabaseManager on each message receipt. This is as bad as first solution. Just different approach so I really resisted this solution as well.

Finally, I found the solution which is not my favorite but doing the trick –

  • Create another task which will carry out unit work for you e.g. in my case sending 1 email. This task will use database connection and will be good for only 1 email processing.
  • Make the sfRabbit callback task as lightweight as possible. Just receive message and then invoke the unit worker task using exec() function.

How does it solve the problem?

sfRabbit consumer task will keep running forever, and it will not use any database connections. This allows us not to worry about database connection leaks.

Using exec() function it will invoke the unit worker task which actually initiates the sfDatabaseManager which will create a connection. Unit worker will be destroyed once the task is carried out. As script ends, PDO will close the connection automatically.

Not the most elegant solution but seems like the only option to me.

Tags - | programming | php | symfony | mysql |

15 Jul 2012

Programming language and Developer's Perspective

In last October, I left ThoughtWorks and joined a startup in Pune. The new company is developing a product for advising health professionals in United States.

The product is a web-application developed in PHP and Symfony framework. Not the cool stack for cool kids (I am referring to Ruby developers here). I never worked on PHP before and believe me, it is not the most awesome language to work when you have worked on Ruby, Java and C#. I hated many things about PHP in the beginning and I still do. But, as a geek at core, slowly I started getting comfortable and found lot of awesome things in it. I can live with it.

I started working with it, and faced huge challenges. We didn't have any tests (despite of PHPUnit). The code wasn't very modularized and object paradigm wasn't followed at all. So I just kept thinking, we should just rewrite entire thing in Ruby and move to a new world. That is when I realized that, I am not going to solve anything by moving to Ruby.

Yes, Ruby is a better language, it will reduce a lot of code and generally Rails will take care of plenty of the best practices without breaking a sweat. I just needed developer's agreement and their comfort level to move to Rails. The team was not comfortable at all. They were comfortable in PHP and not Ruby. It means, you'll be spending time in ramping up team in Ruby and migrating application. Bad way to move. Getting team to sign up for that, it was another challenge. Never forget that even if everybody agreed, its not very difficult to write bad code in ruby and create a royal mess.

So, in the end, we decided to stay in the PHP world, and do the right thing. Clean up the code and make it a better place. We started writing tests using PHPUnit. Used phactory for database factories. Introduced Jenkins as CI server. Started using jasmine for JS unit testing. Selenium was setup for functional tests. Got everybody licenses of PHPStorm :).

While all of this action takes place, we performed merciless refactoring and we deleted thousands of lines of duplicate/unnecessary code. This effort was happening in parallel to regular feature development. After nearly two months of effort, we reached our goal. We had a codebase which sucked less and worked better. Code was easier to manage and extend.

During this time the team learned a lot. Instead of switching to a different tool, we cleaned our own mistakes. That taught entire team to do things differently and in a better way. Now all we needed were few more helping hands like this and we are good to go!

We faced another huge challenge here.

When we put in word to recruit PHP developers in Pune, we got hundreds of resumes. It felt incredibly easy to hire PHP talent in Pune. We were clearly wrong!

We observed 90% of the resumes got rejected simply because those guys were not application/web developers. They were plugin developers for Wordpress, Drupal or Joomla! So that never exposed them to real programming at all!

So I started asking first question to them, what is a web application for you? If they say Drupal, they just got rejected. I might have lost some talent in this process but who cares?! I want a guy who can define an application which is not Drupal!

Then I tried to see if other people (Java, Ruby or .Net developers) want to work for us. Clearly they had their reservations to put PHP on their resume. Why?! Just because they think PHP is not a language which real developers use. I am sorry but thats not how it works!

It doesn't matter what language do you use, what matters is what you did using that language.
Clearly, it was impossible to convey this thought to plenty of real developers.

Finally, we started hunting for talent which was right for us. People who don't give a damn about programming language and focus entirely on programming. They like to solve problems and take pride in code they wrote.

If you belive you're one of those who care about what code you write and it doesn't matter which language, talk to me. My email address is - sachin at dharmapurikar dot in.

Tags - | programming | php | pune | startup |

26 Oct 2011

Mac OS: netstat and pid

I am a Linux user by large. Lately I migrated to Mac as it worked better for me. (…and they look beautiful!) Although, I found from @hyfather that there is significant difference between Mac based utilities vs Linux-based. This difference exists as Linux has GNU coreutils and Mac has BSD based.


<div style="float: left"> </div> <div style="float: left;line-height: 123px">VS</div> <div style="float: left"> </div>

Many switches are different, output is different and it is very frustrating to experienced users.

I ran into netstat command recently. I needed to find out pid of memcached server on my mac. Spend few mins but no luck. Mac based netstat command won't understand -p switch. Aaargh! I had to use lsof to get my job done.

Here is how I did it - 

sudo lsof -i -P|grep memcache

Worked great!

Tags - | linux | bsd | mac | netstat | tips |

15 Oct 2011

Ruby: Insecure World Writable Directory

I keep getting following warning lately on my mac.

ruby warning: Insecure world writable dir /usr/local/bin, mode 040777

Little research and I found, Ruby warns you about any world writeable directory in your PATH. Not only writeable directories but parents as well.

Fix is super quick.

chmod o-w /usr/local/bin

You can replace /usr/local/bin with any directory which ruby complains about. This fixes the issue.

Tags - | ruby | tips | console | warning |