I’ve been a terribly inconsistent writer on MyOwnPirateRadio over the years. I’ve written sporadically. I’ve written snotty rants, mind-numbing trivia, and esoteric technology crib notes. I’ve written about completely random and unrelated topics. In short, I’ve written so poorly that only my most patient, masochistic, and excitement-starved readers are still around. (Hi, mom! Oh, wait, she doesn’t have an internet connection.) So, dear reader(s), I’m sorry. I apologize. I’ve been a shit.

But I do want to write — it makes me happy — and so I’m going to start again.

I will write about one topic:

Life trying to get startups off the ground.

This will not be “How I succeeded at getting my startup off the ground”, or “My genius recipe for gobsmacking success in startup life”, or, ”How I built and sold my company for $10M in just 3 months with only 8 toothpicks, a pair of tweezers, and a tenth of my staggeringly massive brainpower, bitch”. This is not one of those blogs. Gloating will be kept to an absolute minimum. I intend to write more in the vein of “How I’ve worked like crazy on too many projects at once for 5+ years with little to show for it except a much smaller bank account, a bunch of hard-earned lessons, and a smug sense of satisfaction. Please send help.”.

I will write as openly and honestly as I can.

I will try really hard to write regularly, at least once a week.

I will spend no more than 30 minutes writing a post.

I will avoid the temptation to portray this lifestyle as glamorous. ‘Cuz it ain’t. Read me?

And I want to feel good about doing this. I have a lot of “priority guilt”… why write blog posts when I should be writing code, helping out at home, doing some exercise, or becoming a better husband and dad? I also fear the embarassment of screwing up in public. But I’ve come to grips with a few realities: I need to write; my privacy is largely an illusion anyway; and I’m much less interesting to everyone else than I like to think.

I’ve put off starting this for a very long time now, so I’d better post this before I manage to stop myself again.

So here goes: Startup Fever. Wish me luck.

P.S. How Canadian of me, to start off with an apology. Represent!

Unlike Bamboo, Cedar does not offer Varnish as a reverse proxy cache, nor does it automatically gzip content. You need to do it yourself. Heroku recommends:

1. Use memcached, with Dalli as the memcached client. Make sure to follow the Rails 3 section.
2. Use Rack::Cache as a substitute for Varnish. Heroku links to this article which explains how to integrate Heroku with Rack::Cache. I couldn’t get it to work, so I hunted around and pieced together the folllowing riff.

require 'rack-cache'
My::Application.configure do
...
  # Enable Rack::Cache
  config.middleware.use Rack::Cache,
   :metastore => "memcached://#{ENV['MEMCACHE_SERVERS']}/meta",
   :entitystore => "memcached://#{ENV['MEMCACHE_SERVERS']}/body"

You can also set HTTP headers in production.rb as follows. (3600 is an example value, make this whatever you want.)

# Add HTTP headers to cache static assets for an hour
config.static_cache_control = "public, max-age=3600"

And you may want to add this to your config.ru to get gzip working:

use Rack::Deflater

References:

I just finished watching the videos of you launching Pressly at TechCrunch Disrupt. Well done. You and your team should be really proud of what you’ve achieved.

I wish you’d been given more opportunity to elaborate on your strategy to win. Instead you had to spend most of the Q&A time defending Pressly’s raison d’etre to Dustin Moskovitz et. al. That was unfortunate; deck stacked against you. But you did a nice job staying the high road and giving solid answers to the skeptical questions.

Jeff, I loved your comment about publishers being great at telling stories, and not so great at building technology innovations to deliver those stories (in compelling new ways, with compelling profitability). It’s true. And it’s good, I think, that Pressly is joining a cadre of other players in this same space. Existence of multiple players is proof that the market is ready. And publishers need lots of options right now, especially ones that let them do fast, cheap experiments. Pressly can help them do that.

I suppose controversy-seekers could frame this as “walled garden versus open web, round 2″. That was my very first thought after watching the video. But that isn’t really the case, is it? Neither the iPad/App Store ecosystem nor HTML is going away anytime soon. There will be multiple winners in this market, with multiple technology bets. Consumers will buy many different kinds of devices, and consume content in many different places and ways. It’s probably more accurate to compare Pressly’s space to the blogging services market back when it was just getting going: Blogger, WordPress, Movable Type, and so on. Lots of experimentation and diversity, with consolidation down the road.

My biggest takeaway on all this is that Pressly makes a lot of sense from a publisher’s point of view. Publishers are losing sleep over how to follow their audiences to digital devices without abandoning all the assets they hold dear: their brand, destination websites, exclusive content, and UX. And with limited capital and time/runway remaining for technology investment (or investment of any kind) they have to be brutally frugal and thoughtful about what bets they make. Pressly has good answers on the economics (very little cash up front), the technology (more open), the user experience (niiice), and control issues. Clearly The Economist and Toronto Star think so, and I bet many others will reach the same conclusion.

I hope Pressly does really well.

Let me know when I can buy some shares.

osh

I couldn’t find a way to do it with Facebook’s Graph API, but there’s good support for it in the FQL API. You can craft up queries by hand, as shown below, and issue them right from your web browser. The response is an XML document with the statistics for the URLs you want to know about. There are also FQL wrapper libraries in several languages… for Ruby I had some success with the fb_graph gem.

Multiple URLs example:

https://api.facebook.com/method/fql.query?query=select  url,like_count, total_count, share_count, click_count from link_stat where url in(“http://bing.com”, “http://google.com”, “http://facebook.com”, “http://twitter.com”)

Single URL example:

https://api.facebook.com/method/fql.query?query=select  url,like_count, total_count, share_count, click_count from link_stat where url = “http://bing.com”

Within the XML results, the “total_count” attribute is the number shown within the Facebook Like widget.

Docs:
http://developers.facebook.com/docs/reference/fql/link_stat/
http://developers.facebook.com/docs/reference/fql/

Context: MongoHQ, which I’m using with one of my work projects right now, is a service that hosts MongoDB databases. It integrates with Heroku, allowing you to get started on a Heroku app using MongoDB very quickly. MongoHQ supports hourly and daily backup of their databases to Amazon S3 buckets, but alas, not yet for databases provisioned automatically via Heroku. So if you want to step up to a backed-up database you’ll have to set up your own. Here are the steps I followed to do it.

Secret decoder ring:
S3 = Amazon Web Services Management Console, S3 tab.
IAM = Amazon Web Services Management Console, IAM tab.
MHQ = Mongo HQ console.
HKU = Heroku.

Recipe:

1. S3 : Create a bucket for your app’s backups, e.g. “my-app-backups”.
2. IAM: Create an IAM User account for MongoHQ to use, e.g. “mongohq”. Download the credentials and store them safely.
3. IAM: Create an IAM Group to contain all users with backup permission, e.g. “myapp-backup-writers”.
4. IAM: Add the mongohq User account to the newly created Group.
5. IAM: Using the Permissions tab of the Group’s properties attach a new security policy, granting permission to write to the myapp-backups bucket. See below for a sample policy.
6. MHQ: Sign up for your own MongoHQ account.
7. MHQ: Create the database you wish to use with Heroku.
8. MHQ: Select the “Backups” tab and enter the IAM User credentials from step 2.
9. MHQ: Click “Save Settings”. If all is well you’ll see a message, “Setting Updated”.*
10. MHQ: Create a new user on the Database Users tab.
11. MHQ: Copy down the Mongo URI string from the Database Info tab, substituting your user name and password from step 10.
12. HKU: In the root directory of your Heroku app do “heroku config:add MONGOHQ_URL=’URI’”. URI is the Mongo URI from step 11. (Normally Heroku sets this variable for you when it provisions a MongoHQ database. You’re overriding it with a link to your own database.)

* Step 9 is where I got stuck. It worked fine with my own personal security credentials, but not with the IAM account I created specifically for MongoHQ to use. (I wanted more security… giving your main account credentials to a 3rd party site isn’t smart.) I fiddled around with the security policy for quite a while, and eventually discovered I needed the “ListAllMyBuckets” permission. Here’s the security policy that worked for me:

    {
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [ "s3:ListAllMyBuckets" ],
          "Resource": "arn:aws:s3:::*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:PutObject",
            "s3:PutObjectAcl"
          ],
          "Resource": [
            "arn:aws:s3:::MY-APP-BACKUPS/*"
          ]
        }
      ]
    }

Treat the world as flat, and find a defensible niche to start in. E.g. think of the world as the Risk game board, and pick an initial territory where your chances of getting established are high. If you start in Europe, you have lots of borders and the battle is much harder than, say, starting in Australia. (Or Kamchatka. Remember Kamchatka? I think I learned more geography in Risk than in school.)

Follow your own rules. In response to an audience question about mistakes he has made in his career, Howard explained that many of them stemmed from breaking his own rules when investing, e.g. (1) investing in the market when all signs still clearly pointed to down, (2) investing in companies whose CEOs did not have deep domain experience. (Howard likes such companies best, as a rule, for his investments.)

Have a sense of humor, especially about yourself. Howard’s self-deprecating humor made him a real pleasure to listen to, and he explained that his habit of poking fun at himself is part of why people like to follow him in social media. “I make fun of my own mistakes first, before anyone else has a chance to.”  He talked openly about passing on a few investments that would have been incredibly profitable, e.g. a chance to invest $25K in Twitter when it was valued at only $20M.

Spot and leverage the big waves. He spoke to only one slide during the 30-or-so minutes, and it showed two graphs depicting the meteoric rise in Twitter and Facebook usage. His main point was that when you can get in front of a huge market-changing trend (like social media) you are maximizing your business opportunities to do well. “You don’t need to catch every wave. Just one amazingly great wave in your lifetime is all you really need”. Also, fighting a paradigm-changing trend is futile. So, in this specific example, which Howard calls “the social leverage trend”, figure out which of Facebook and Twitter it makes best sense to align your business with first, then integrate and try to ride the wave.

Use your dashboards. He talked about dashboards and his view that they’re essential to business success. From memory, “What’s your dashboard? It’s never too early to start building your dashboard. … Figure out the one or two pages you need to look at every day to understand both the macro market picture and the micro [i.e. the key signals/metrics from your own business]. For me it’s http://techmeme.com, http://angellist.com, and market all time highs.” These dashboards give him “the very early market signals [from AngelList] and the loudest public market signals [from all-time highs]“.

Thanks Howard for a fun and frank talk.

Ruby on Rails normally provides pretty good debugging information for dealing with errors, but I ran into an odd error today that had no debug info other than the exception itself. Here’s what I did to find and fix the problem.

Symptoms: “So, tell me, how long have you been feeling this way?” I had just added a new controller, model, views, and helpers to my app. Requesting a route to any of the new controller’s actions caused Rails to die with the exception “ActionController::RoutingError (undefined method `sub’ for nil:NilClass)”. No stack trace. No filename / line numbers. I knew one of the 20-odd new files I had just created must be causing the problem, but I didn’t know which one.

Diagnosis: “Breathe in. Breathe again. Again. OK, now cough. Ah, yes, now I see the problem.”: The only way I know of to diagnose these kind of problems is brute force. I added a debugger statement to the top of application_controller.rb and restarted the app. Then I requested a URL that was causing crashes, and from the breakpoint, stepped forward in the debugger (next, next, next…) until the app died. Studying the code that executed just before the app died didn’t yield an obvious result, but about 10 frames up the stack I could see a call to ActiveSupport::Inflector#constantize. I edited that method to print (puts) the parameters it was being called with, then restarted the server and tried again. Voila: the last parameter spit out by #constantize before the server died was the culprit.

#constantize PlaceLinksHelper for name PlaceLinksHelper

The Cure: “Take three of these and call me in the morning.” : It turns out I had created a helper module with the file name “places_helper.rb”, but within that file I mistakenly defined “module PlaceLinksHelper”. For some reason that shall remain a mystery to me, this caused things to fall-down-go-boom. The fix was simply to give the module the name Rails expected to find: “PlacesHelper”.

Upon Reflection… “I sure hope that doesn’t happen to me again. But it probably will.” I don’t know how you do this sort of diagnosis efficiently in a closed-source or compiled environment.  This is about the 5th time I’ve edited Rails source code to find and fix an oddball problem. I guess you could try attaching a debugger to binaries and then study the gobbledy-gook, but… no thanks. The only other thing I  could have done in a compiled/closed source setup is study each of my own files line by line. That would have worked, but it would have taken far more time.

Source code good.

First, an account of the nefarious deeds

Case #1: Sally got phished. Sally received a phishing email claiming to be from “Google Account Management”, asking to reconfirm her password.* She gave them the password, only to discover a few hours later, that (a) her password had been changed, locking her out of her gmail account; (b) the phishers had sent email to all of her contacts with a “Help, I need cash fast, send me money via Western Union” scam; (c) she couldn’t get back into her gmail account because she hadn’t set the password recovery information. It took her days to get everything back to normal.

Case #2: Jane dated a creep. Jane dated a guy who turned out not to be a good fit for her. Unfortunately they moved in together before she figured it out. She had to force a breakup, eventually kicking him out. A few days later she realized he was logging on to her email account and reading her email, and in some cases deleting emails. She changed her password, but to no avail: it seems he had installed spyware on her home computer that monitored what she was typing and informed him of password changes. Yikes.

Second, some advice

For starters, both the Canadian and US government offer comprehensive web sites that tell you what to do to protect yourself against identity theft. Here’s the Canadian site, and here’s the US site. Check them out. They say roughly the same thing. I find the Canadian one a little easier to read.

I gave Sally and Jane some advice specific to their scenarios, as follows:

1.  Never reply to an email request for a password. No legitimate business will ever ask you for a password via email. If you’re wondering whether an email is “real” or not, you might try searching on Snopes to see if someone else has reported the same thing.

2. Call Transunion, Experian, and Equifax (the credit reporting agencies) and (a) ask for a copy of your credit reports, and (b) have them flag your file with a “fraud alert” so that nobody can open an account under your name without confirmation. See the US identity theft web page for phone numbers for the credit reporting agencies and a  description of how fraud alerts work.

3. From a secure network, log in to your email provider (gmail, hotmail, whatever) and do two things:

(a) Change the password you use to login to email. Here’s good advice on how to create a strong password that’s also easy to remember.

(b) Fill in the “Password Recovery” information in your account settings. This is they info they use to confirm your identity when you click “I forgot my password”… typically, a secondary email address and/or phone number. This step is really vital because if you get hacked without filling in the recovery information you may not be able to get your account back at all.

Notice I suggest doing this from a secure network. In Jane’s case, she couldn’t safely change her password from home because her ex-boyfriend put spyware on her machine. She went to a friend’s house to do this step.

4. Protect your computer and home network.

(a) If you suspect your computer has spyware or a virus on it then install and run a virus scanner. For this I recommend Avast. I’ve had several experiences with Norton products crippling computers that were otherwise fine.

(b) If you think your home network has been compromised (like Jane’s), change the password on your home network router and cable modem / DSL modem.

It’s best to do 4a and 4b in unison. Get a computer-savvy person to help you if you haven’t done this before.

5. Report what’s happening to the police. Tell them you are concerned you’ve been or may become a victim of identity theft. Collect any evidence you can and give it to them.

Since identity theft is still relatively new, you may find it difficult to get police to pay attention to you. In Sally’s case, the local authorities refused to help. The US identity theft site has suggestions on what to do if local police won’t handle your report, and it’s applicable to Canada (and probably other places) as well.

That’s it. There’s heaps more you can do, but these basic steps will help you protect yourself and recover if you run into similar problems. Here’s hoping you manage to stay problem-free.

* I’ve changed names to protect my peeps. They don’t need any more hassles.

We use Macs almost exclusively as our work machines these days, and we don’t have a lot of spare hardware lying around. Going without a machine for even a few hours is a big productivity hit. So the awful grinding noise that began emanating from Katrin’s Macbook last week was not a welcome sound. Luckily it was clearly coming from the fan, not the hard drive.

We were having an extremely busy week, and having waited days in the past for support at the local Apple store my first instinct was to find a local repair shop that could do it quickly. After all, it’s just a fan, so it should be fast and cheap, right?

Silly me.

After calling several repair shops I realized that the fix was going to run between $150 and $200. Pretty steep for a simple part swap — I can get parts in my car fixed for less than that — so I decided to do it myself.

Here’s what I discovered:

1. I couldn’t find any stores in Toronto who were willing to sell me the part. They all want to do full repairs.
2. More generally, it seems Apple doesn’t allow local service providers to resell parts. They are full-service only.
3. Laptop fans are commodity even in Macs, according to one of the folks at laptopexpert.ca, so you’re better off shopping for the fan by its physical dimensions rather than an Apple part number. (I.e. you pay more for an “Apple fan” than a “laptop fan”, but they’re the same thing.) This may help in the future, but in my case I didn’t know the fan size and couldn’t find it online.
4. Darryl at iRepair.ca was most helpful on the phone and pointed me at several web sites for finding the part. He also had the most reasonable repair prices, about $50 less than everyone else I called.
5. I couldn’t find any Canadian wholesalers who had the part. Perhaps I didn’t search hard enough, but I was surprised, nevertheless.
6. IFixIt.com and PowerBookMedic.com are two US-based businesses that sell parts online. Their shipping options are slightly different, and the part prices in this case were about the same. It’s worth checking both and deciding how important delivery speed is to you.
7. There are fans for sale on eBay, but when I searched I only found used parts. A used fan is a pretty silly thing to buy, in my opinion, so… no joy there.

End result: I bought a fan from IFixIt for $40, they shipped it overnight (FedEx: $40), and I paid another $5 and change in HST. I used the excellent IFixIt repair guide and had the repair complete in under an hour. If I wasn’t in such a rush I could have chosen a $4.00 delivery option instead, and waited for US Post.

Thank you, IFixIt, and Darryl at iRepair.ca. Darryl, you’re first on my list the next time I need repair help.

P.S. I did also try an app called smcFanControl, to no avail. Worth a try.

If you run a web site you’ll want to periodically verify that your site is being crawled properly by search engines. Both Google Webmaster tools and Bing Webmaster give you a handy “crawl errors” list which shows URLs the search engine fails to index, and the corresponding “origin” pages that link to the problem URLs.

This weekend I checked on Google Webmaster and was surprised to find a few hundred 404 errors in recent crawls of 5 Blocks Out. (There were no matching errors on Bing, and I’m still not sure why.)

Warning: from here on, this blog post gets ugly. Skip to “Lessons Learned” below if you just want the punchline.

All the URLs looked similar to this:

5blocksout.com/q5E3Q1Db7SpV45DINIfRr28pnGRt5G/ujfe3SPV36M=

In other words, each URL had a path consisting of a long string of alphanumeric characters, and ending with an equals sign.

We don’t support URLs like this on 5 Blocks Out, hence all the 404 errors. But where were they coming from?

The crawl errors report listed many different 5 Blocks Out web pages as the origins for these problem URLs. The next step was obvious, then: look at the source code for each origin page containing a problem URL, and then fix the problem. Well, it turned out that none of the origin pages actually contained any of these problem URLs. Googlebot had discovered these URLs at crawl time, but I couldn’t find any them myself. Phantom URLs.

I cleared my browser cache and tried again, with the same results. Then I tried wget, to more closely mimic what Googlebot would see. Again, same results: I couldn’t find any of these supposed 404 URLs on the origin pages.

My hypotheses at this point were not pretty:

• A bug in the application is intermittently generating bad URLs.
• The pages contain poorly formed HTML, which is causing Googlebot (but not Bing) to parse them incorrectly.
• A virus or some sort of malware is corrupting the web pages.
• A bug in our web server software or Rails is corrupting the pages.

Not an appealing list to investigate, especially without a way to reproduce the problem. The “bad HTML” one was easy to eliminate by running the page through the W3C Validator, so that left only gnarly possibilities.

After a while of casting about for clues I eventually returned to looking at the URLs themselves, hoping to find patterns in their structure, the crawl dates, the origin pages… anything. And a few things struck me:

1. The 404 errors all began around Nov 26, which is the date I upgraded our web server to the latest versions of nginx and Phusion Passenger. This later turned out to be a false lead, but it seemed useful at the time.

2. The strings of alphanumeric characters in these phantom URLs looked similar to the authentication tokens that Rails generates as a countermeasure against cross-site request forgery. Rails inserts these tokens into every page that contains an HTML form.

A little voice reminded me, “Hey smartypants, we have some javascript that appends authentication tokens to AJAX requests made from our web pages.” Now, that code shouldn’t have affected search engines, because it was specific to POST requests, and everyone knows that well-behaved search engine crawlers only do GET request. But still… new hypothesis: “Our javascript is mistakenly appending authentication tokens to Googlebot GET requests.”

How to prove it? Well, search engines have caches of crawled pages. I checked Google’s page cache and found a cached version of an origin page that supposedly contained a 404 URL. The HTML source showed… drum roll… no such URL. But the HTML did contain javascript with an authentication code that exactly matched the path portion of the 404 URL. Bingo!

From there, the fix was easy. I did indeed have a bug in my javascript. The code was screening out AJAX requests with a type of “GET”, but it didn’t check for the lowercase “get”, and so it appended auth tokens in cases where it shouldn’t have.

Why did this start happening all of a sudden, you might ask? I don’t know. Perhaps something changed in the way Googlebot issues its HTTP requests. Maybe we’d always had this problem, and Googlebot just hadn’t yet crawled pages that exhibited it. Maybe web fairies temporarily stopped sprinkling magic pixie dust on our site. Who knows.

Lessons learned:

• Check webmaster tools and web server logs once in a while for unexplained crawl errors.
• Remember that search engines allow you to search for cached versions of web pages. This can help with diagnosing crawl problems. (In fact, crawl error reports really ought to link directly to the cached versions of crawled pages.)
• String comparisons don’t work very well if you don’t make them case-insensitive.

// Insert authenticity token into all non-GET AJAX requests
function insertAuthToken(elm, xhr, s) {
-    if (s.type == "GET") {
-        return;
-    }
+    if (s.type == 'GET' || s.type == 'get' || typeof(window.auth_token) == "undefined") return;