Learning To Code

Join me as I take the plunge!

Make Your Own URL Shortener With Rails 4 and Heroku

Ever wanted to make your very own URL shortener? Know a little bit of rails? Then you’re good to go!

My goal was to make my own personal bit.ly. I wanted a home page with a text box that I could paste in a URL, press a button, and get a shortened link back. I also wanted to track the number of times each of my links was clicked. I also wanted a very visual leaderboard of my top links as well as a tabular page of all my links. I also wanted to deploy the app live to the internet and use my own custom short domain.

Here is what I ended up making: acal.io

Set Up

The very first step was to a create new rails app with postgres since I knew that I would later be deploying to heroku

1

rails new url-shortener --database=postgresql

The next thing I do to any new rails app is to add pry to gemfile. Pry is an amazing gem that really helps me develop. If you are not sure about using pry, or how it can help, you I suggest you watch this talk about REPL Driven Development from Ruby Conf 2013.

1

gem 'pry'

I knew that I wanted the app to look at least somewhat nice, so not being a designer I defaulted to using bootstrap. I download the latest copy of bootstrap and unzipped it.

I added the file ‘bootstrap.min.js’ to the folder vendor/assets/javascripts. Then modified application.js to indicate this

1

//= require bootstrap.min

I add the file ‘bootstrap.min.css’ to the folder vendor/assets/stylesheets. Then I modified application.css to indicate this

1

 *= require bootstrap.min

Models

Now I reached the ‘heart’ of the application - the link model. I first generated the scaffolding for the link

1

rails g scaffold link given_url:string slug:string clicks:integer snapshot:string title:string

I then went into the actual migration file and modified the default value for clicks BEFORE I migrated. This is because you can’t increment a non-integer value (nil), which is the original default.

1

t.integer :clicks,  :default => 0

I then migrated the database

1

rake db:migrate

Now that I had my model in place, I still needed to actually generate a short link for each URL that I pasted in.

I made a method ‘generate_slug’, which creates the short part of the link after the base url. All this method does is take the unique id for a link, and converts it into it’s Base 36 equivalent. Base 36 allows really large numbers to be shown in just a few characters. For example the 1 Billionth link in the database will have a slug of “gjdgxs” - just 6 characters as opposed to the 10 characters in the number’s ID (1,000,000,000).

1
2
3
4

  def generate_slug
    self.slug = self.id.to_s(36)
    self.save
  end

But the slug alone is not enough. I want to show the user the entire full URL. To do this I want to create an environmental variable, and knowing that I will be storing some keys down the line anyway, I want to keep it just to my machine. I added the figaro gem to accomplish this, but there are other ways to do this, which I’ve written about before.

1

gem "figaro"

I then used figaro’s generate command to create the application.yml file that git will ignore (so your passwords don’t get committed to the rest of the world)

1

rails generate figaro:install

Then I add the base URL as an environmental variable to the application.yml file

1

BASE_URL: 'http://acal.io/'

With this I was able to make a convenience method to display the entire shortened URL

1
2
3

  def display_slug
    ENV['BASE_URL'] + self.slug
  end

In order to make the trending links section really pop visually, I figured getting a screenshot would do the trick. I also wanted to grab the title of each page, in order to give a bit more context to the end user. I used IMGKit to capture the screenshot, CarrierWave and Fog to associate a screenshot with a model and upload it to Amazon S3, and Mechanize for getting the page title. I also knew upfront that I want to do this asynchronously, so I’ll also threw in SideKiq.

1
2
3
4
5

gem 'imgkit'
gem 'carrierwave'
gem "fog", "~> 1.3.1"
gem 'mechanize'
gem 'sidekiq'

I got my key’s from Amazon, and created a bucket on S3. I went back to my application.yml file and added these all as environmental variables, so I could call them elsewhere in the application.

1
2
3

AWS_ACCESS_KEY_ID: 'REDACTED'
AWS_SECRET_ACCESS_KEY: 'REDACTED'
AWS_BUCKET: 'REDACTED'

First I made a method for actually getting the screenshot and page title. For now, these are just Sidekiq Workers which I’ll actually build later. Of note is that I had to pass in the ID of a link, and not the full object, in order to get SideKiq to work.

1
2
3
4

  def screenshot_scrape
    Screenshot.perform_async(self.id)
    Scrape.perform_async(self.id)
  end

I already had a method generate a slug, and now I had a method to both capture the screenshot and capture the page title. I decided to use after_create callbacks to make sure these things all happend after each new link was created.

1

  after_create :generate_slug, :screenshot_scrape

I then made a screenshot worker, which captures a screenshot of each link. Because I had to pass in an ID, I first had to lookup the object. I then needed to create a temporary file of the screenshot as an intermediate step, to ensure it worked with CarrierWave. Finally I associated the screenshot with the link, which then allows triggers a call to Fog to upload the screenshot to S3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class Screenshot
  include Sidekiq::Worker

  def perform(link_id)
    link = Link.find(link_id)
    file = Tempfile.new(["template_#{link.id.to_s}", '.jpg'], 'tmp', :encoding => 'ascii-8bit')
    file.write(IMGKit.new(link.given_url, quality: 50, width: 600).to_jpg)
    file.flush
    link.snapshot = file
    link.save
    file.unlink
  end

end

Then I made a scrape worker, which scrapes the link and finds the page title.

1
2
3
4
5
6
7
8
9
10
11
12

class Scrape
  include Sidekiq::Worker

  def perform(link_id)
    link = Link.find(link_id)
    agent = Mechanize.new
    page = agent.get(link.given_url)
    link.title = page.title
    link.save
  end

end

Next I made a configuration file for CarrierWave, and made sure it would work on Heroku (which has permissions issues - more on that later)

1
2
3
4
5
6
7
8
9
10
11

CarrierWave.configure do |config|
  config.root = Rails.root.join('tmp')
  config.cache_dir = 'carrierwave'

  config.fog_credentials = {
    :provider               => 'AWS',
    :aws_access_key_id      => ENV['AWS_ACCESS_KEY_ID'],
    :aws_secret_access_key  => ENV['AWS_SECRET_ACCESS_KEY']
  }
  config.fog_directory  = ENV['AWS_BUCKET']
end

I then made the uploader that CarrierWave needs. the store_dir method tells S3 the naming convention of the screenshots, and I just left it at the default. The cache_dir method is a way to get around Heroku’s file writing permissions issues. Basically Heroku won’t let you write files except in this one place, so I made sure to do this.

1
2
3
4
5
6
7
8
9
10
11

class SnapshotUploader < CarrierWave::Uploader::Base
  storage :file
  storage :fog
  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  def cache_dir
    "#{Rails.root}/tmp/uploads"
  end
end

I finished up by adding the mount_uploader call to the link model.

1

  mount_uploader :snapshot, SnapshotUploader

Controllers

First up I make some modifications to the auto generated links controller. I decided to delete the index, edit, update, and destroy actions. These actions get into permissions issues, which could lead naturally to building out users and all sorts of other features that are not core to this app. I’m not a fan of feature creep, and recommend that you also limit the scope of your own projects whenever possible. I also knew that I wanted the create action to respond asynchronously. I didn’t want to have to do a hard page reload when you shortened the link, and like bit.ly does, it just appends your new short link back to the screen. To do this I added the ability for the create action to respond to format.js (more on this later).

I also added the ability to route a slug on a naked url to the show action. This is so the URL http://acal.io/slug will naturally get routed to the long URL that the corresponding slug was made for. It also increments every time the link is clicked. If the user instead goes to http://acal.io/links/id, no incrementing is done, and they are taken to the show page for that link - same as normal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

class LinksController < ApplicationController
  before_action :set_link, only: [:show]

  def index
    @links = Link.all
  end

  def show
    if params[:slug]
      @link = Link.find_by(slug: params[:slug])
      if redirect_to @link.given_url
        @link.clicks += 1
        @link.save
      end
    else
      @link = Link.find(params[:id])
    end
  end

  def create
    @link = Link.new(link_params)

    respond_to do |format|
      if @link.save
        format.html { redirect_to root_path, notice: 'Link was successfully created.' }
        format.js
        format.json { render action: 'show', status: :created, location: @link }
      else
        format.html { render action: 'new' }
        format.json { render json: @link.errors, status: :unprocessable_entity }
      end
    end
  end

  private
    def set_link
      @link = Link.find_by(slug: params[:slug])
    end

    def link_params
      params.require(:link).permit(:given_url)
    end
end

I then updated the routes file to map the naked url with a slug to the show action.

1

  get ':slug' => 'links#show'

I generally make a home controller for my apps, and it’s a common design pattern I’ve seen others do, so I generated a home controller.

1

rails g controller home

I modified the routes file to make the index action on the home controller the root

1

  root 'home#index'

I decided to only have the top 12 trending items, since it’s allows for bootstrap to gracefully degrade from 4 to 3 to 2 to 1 columns.

1
2
3
4

  def index
    @link = Link.new
    @top_links = Link.order(clicks: :desc).first(12)
  end

I also wanted a page that just listed all the links created as a table, so first made a route.

1

  get '/all' => 'home#all'

I wanted to be sure not to overwhelm the server in case this app received a lot of link shortening requests. I figured pagination would do the trick and I opted to use the will_paginate gem.

1

gem 'will_paginate', '~> 3.0'

Then I filled out the all action, and decided to just put 4 rows per page as the app is still small.

1
2
3
4

  def all
    @link = Link.new
    @links = Link.paginate(:page => params[:page], :per_page => 4)
  end

Front End

I first made a new partial ‘layouts/_new_link’ for a new link that would persist on top of all the pages in the app. I included a blank div called ‘result’ so I could append the result later on.

1
2
3
4
5
6
7

<div class="row">

<%= render 'links/form' %>
</div>

<div class="result">
</div>

I then modified the ‘layout/application’ file. I changed the auto generated title, added the new partial I just created and wrapped everything in a bootstrap container.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<!DOCTYPE html>
<html>
<head>
  <title>Url Shortener</title>
  <%= stylesheet_link_tag    "application", media: "all", "data-turbolinks-track" => true %>
  <%= javascript_include_tag "application", "data-turbolinks-track" => true %>
  <%= csrf_meta_tags %>
</head>
<body>
  <div class="container">
    <%= render 'layouts/new_link' %>
    <%= yield %>
  </div>
</body>
</html>

I then modified the link form partial ‘links/_form’. I wanted to make this submit asynchronously so added a remote:true option to the form. I also added a logo, some placeholder text, and some bootstrap styling.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<%= form_for(@link, remote: true) do |f| %>
  <% if @link.errors.any? %>
    <div id="error_explanation">
      <h2><%= pluralize(@link.errors.count, "error") %> prohibited this link from being saved:</h2>

      <ul>
      <% @link.errors.full_messages.each do |msg| %>
        <li><%= msg %></li>
      <% end %>
      </ul>
    </div>
  <% end %>
  <div>
    <%= link_to image_tag("logo.png"), root_path %>
  </div>
  <div class="field">
    <%= f.text_field :given_url, :class => "form-control", :placeholder => "Paste a link to shorten" %>
  </div>
  <div class="actions">
    <%= f.submit 'Shorten Me', :class => "btn btn-primary"%>
  </div>
<% end %>

I then wanted to append the short link result back onto the page once it was processed. I referred to this earlier in the post when I was making the links controller. In order to do this I made a new file called create.js.erb in the ‘views/links’ folder.

1

$('.result').append('<p class="short_url">Your Shortened Link: <%= link_to @link.display_slug, @link.display_slug, target: "_blank" %></p>')

I then made the ‘links/show’ page. I added in a bit of formatting, both from bootstrap, and from my own css.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<p id="notice"><%= notice %></p>

<% if @link.snapshot %>
<div class="screenshot-outer-individual"><%= link_to image_tag(@link.snapshot.url, class: "screenshot"), @link %></div>
<% end %>
<p class="title"><%= @link.title %></p>
<p class="short_url"><%= link_to @link.display_slug, @link.display_slug, target: '_blank' %></p>
<ul class="click_time">
  <li><span class="badge"><span class="clicks"><%= @link.clicks %></span></span> clicks</li>
  <li><span class="time"><%= time_ago_in_words(@link.created_at) %> ago</span></li>
</ul>
<p class="long_url"><%= link_to @link.given_url, @link.given_url %></p>

<%= link_to 'Back', root_path %>

I also wanted to make sure that non-valid URLs could not be submitted. There are a lot of ways to do this, but I settled with doing client side validation.

I first installed the jquery-validation-rails gem, which is just an easy way to get the JQuery Validation Plugin into a rails app.

1

gem 'jquery-validation-rails'

I then indicated that these files were part of my application in application.js

1
2

//= require jquery.validate
//= require jquery.validate.additional-methods

I then made the actual javascript validation that required both the presence of a URL, and it’s validity as a URL according to a regular expression. I put this in ‘application.js’.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

$.validator.addMethod(
        "regex",
        function(value, element, regexp) {
            var re = new RegExp(regexp);
            return this.optional(element) || re.test(value);
        },
        "Please enter a valid URL (using http)"
);

$('#new_link').ready(function() {
  var password_validator = $('#new_link').validate({
    rules: {
      'link[given_url]': {
        required: true,
        regex: "^(http|https):\/\/[a-z0-9]+([\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5}"
      },
    },
    messages: {
      'link[given_url]': {
        required: 'Please put in a URL',
      }
    }
  });
});

I then made the ‘home/index’ page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<div class="title">
  <h1>Trending links</h1>
</div>

<div class="row">
  <% @top_links.each do |link| %>
    <div class="col-lg-3 col-md-4 col-sm-6 col-xs-12 link">
      <% if link.snapshot %>
      <div class="screenshot-outer"><%= link_to image_tag(link.snapshot.url, class: "screenshot"), link %></div>
      <% end %>
      <p class="title"><%= truncate(link.title, length: 60) %></p>
      <p class="short_url"><%= link_to link.display_slug, link.slug, target: '_blank' %></p>
      <ul class="click_time">
        <li><span class="badge"><span class="clicks"><%= link.clicks %></span></span> clicks</li>
        <li><span class="time"><%= time_ago_in_words(link.created_at) %> ago</span></li>
      </ul>
      <p class="long_url"><%= link_to truncate(link.given_url, length: 60), link.given_url %></p>
    </div>
  <% end %>
</div>

<p class="bottom_nav"><%= link_to 'See all of the links', '/all' %></p>

I finally made the ‘home/all’ page

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

<div class="table-responsive">
  <table class="table">
    <thead>
      <tr>
        <th>Given url</th>
        <th>Short Link</th>
        <th>Title</th>
        <th>Clicks</th>
        <th></th>
        <th></th>
      </tr>
    </thead>

    <tbody>
      <% @links.each do |link| %>
        <tr>
          <td><%= link.given_url %></td>
          <td><%= link_to link.display_slug, link.slug, target: '_blank' %></td>
          <td><%= link.title %></td>
          <td><%= link.clicks %></td>
          <td><%= link_to 'Show', link %></td>
        </tr>
      <% end %>
    </tbody>
  </table>
</div>



<%= will_paginate @links %>

<p class="bottom_nav"><%= link_to 'See just the trending links', root_path %></p>

Along the way I added a lot of custom css to ‘application.css’. Of note is an animated yellow flash of the new shortened link as it is appended to the page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157

/*
 * This is a manifest file that'll be compiled into application.css, which will include all the files
 * listed below.
 *
 * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
 * or vendor/assets/stylesheets of plugins, if any, can be referenced here using a relative path.
 *
 * You're free to add application-wide styles to this file and they'll appear at the top of the
 * compiled file, but it's generally better to create a new file per style scope.
 *
 *= require_self
 *= require bootstrap.min
 *= require_tree .
 */
body
{
background-color:yellow;
padding-bottom: 100px;
}

span.clicks {
  font-size: 50px;
}

span.time {
  font-style: italic;
}

img.screenshot {
  width:100%;
  /*position: absolute;*/
  /*clip: rect(0px,200px,300px,0px);*/
}

form div {
  /*float:left;*/
  display:inline-block;
}

form {
  text-align: center;
  margin-top:20px;
}
form div.field {
  margin-top:12px;
  margin-left:10px;
}

form div.actions {
  margin-top:12px;
  margin-left:10px;
}

h1 {
  text-align: center;
}
div.title {
  margin-bottom: 20px;
}
div.result {
  height: 60px;
}

@keyframes myfirst
{
  from {background: yellow;}
  to {background: none;}
}

@-webkit-keyframes myfirst /* Safari and Chrome */
{
  from {background: yellow;}
  to {background: none;}
}

div.result p {
  margin:0;
  padding:0;
  line-height: 60px;
  animation: myfirst 1s;
  -webkit-animation: myfirst 1s; /* Safari and Chrome */
}

div.link {
  margin-bottom: 60px;
}


p.bottom_nav {
  text-align: center;
}

p.short_url {
  text-align: center;
  font-size: 20px;
  margin-top:10px;
}

p.title {
  text-align: center;
  height: 36px;
}

p.long_url {
  text-align: center;
  color:#DCDCDC;
  font-size: 10px;
  padding-left: 10px;
  padding-right: 10px;
  overflow: hidden;
  height: 36px;
}

ul {
  padding:0px;
  list-style:none;
  height:400px;
}

ul.click_time li:first-child{
  float:left;
}

ul.click_time li:last-child{
  margin-top:55px;
  float:right;
}

ul.click_time {
  height:70px;
  margin-left: 20px;
  margin-right: 20px;
}

li {
  margin-top:20px;
}

div.screenshot-outer {
  height:200px;
  overflow: hidden;
  border:1px solid;
  border-radius:5px;
  border-color:#DCDCDC;
}

div.screenshot-outer-individual {
  height:340px;
  overflow: hidden;
  border:1px solid;
  border-radius:5px;
  border-color:#DCDCDC;
}

div.screenshot-outer img {
  padding:10px;
}

Deploying to Heroku

Heroku makes deploying rails apps incredibly easy, and I use it for all my side projects. I first had to add the ‘rails_12factor’ gem and a line indicating that I was using ruby 2.0.0. In order to get SideKiq to work I had to use unicorn, so included the ‘unicorn’ gem too.

1
2
3

gem 'rails_12factor', group: :production
gem 'unicorn'
ruby "2.0.0"

Heroku does not play nicely with the IMGKit gem, and in order to make the screenshots work I needed to download a compiled version of IMGKit.

https://wkhtmltopdf.googlecode.com/files/wkhtmltoimage-0.10.0_rc2-static-amd64.tar.bz2

I unzipped it and put it in the ‘bin’ directory of my application. I then made a configure file for IMGKit so that this would actually take effect.

1
2
3

IMGKit.configure do |config|
  config.wkhtmltoimage = Rails.root.join('bin', 'wkhtmltoimage-amd64').to_s if ENV['RACK_ENV'] == 'production'
end

I also had to modify config.ru to get CarrierWave to work. This again was a permissions issue with Heroku not allowing file write access, except for the one tmp directory.

1
2
3
4
5

# This file is used by Rack-based servers to start the application.

require ::File.expand_path('../config/environment',  __FILE__)
use Rack::Static, :urls => ['/carrierwave'], :root => 'tmp'
run Rails.application

I then made a Procfile, which is all Heroku needs to run redis and SideKiq for your application automatically. I put it in the base directory (same place as the gemfile).

1
2

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq

I then had to make a configure file for unicorn.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 15
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

I was finally ready to go live! I created, pushed, and migrated the app on Heroku

1
2
3

heroku create
git push heroku master
heroku run rake db:migrate

I then configured the environmental variables on Heroku (remember, these are in application.yml which is ignored by git and thus not already on Heroku).

1
2
3
4

heroku config:set AWS_ACCESS_KEY_ID=REDACTED
heroku config:set AWS_SECRET_ACCESS_KEY=REDACTED
heroku config:set AWS_BUCKET=REDACTED
heroku config:set BASE_URL=http://acal.io/

Then I added redistogo (free) and added another dyno so that SideKiq would run.

1
2

heroku addons:add redistogo
heroku ps:scale worker+1

Finally I bought a custom domain through hover, who I try to use for all my domain purchasing needs (no affiliation, they are just awesome). I had to add a CNAME record called ‘www’ with the target being ‘cryptic-shore-8548’, since my heroku domain is https://cryptic-shore-8548.herokuapp.com.

I also set up domain forwarding so that a naked url (http://acal.io) would just redirect to www (http://www.acal.io). Heroku does not support naked domains out of the box. There are other solutions out there (they do cost money) but this works for now.

I then associated the domain with my app and opened it up in the browser.

1
2

heroku domains:add www.acal.io
heroku open

It worked!

Conclusion

That seems like a lot of work, but it actually didn’t take me that long to do. It’s a great learning project for those of you just starting out as a developer. Let me know if you can actually use this to make your own URL shortener, or if there is something I missed. Best of luck!