Workaround for File Input on Rails Remote Forms

Workaround for File Input on Rails Remote Forms

Recently, I worked on a project which included a form that was being submitted remotely using jQuery-Rails. The main problem was that the form needed to have a file input on it and jQuery-Rails won’t work if you have one. Now, there are workarounds that work if you need to actually keep the file input. In my case, I was processing the file with JavaScript and then just submitting the parsed data, so the file input wasn’t really needed.

Read more

Building a Schedule with D3

Building a Schedule with D3

The One Acre Cafe, a non-profit community café, recently opened in Johnson City. But due to budget constraints, they’re without software to manage their volunteer staffing.

I thought I would do a simple rails app (live) to try and address this problem. For rendering the calendar, I decided to do it client-side usingD3 . My initial version was in SVG, based on the Calendar View example. The SVG layout relied on static block sizes, so I reimplemented it in HTML.

Read more

Using the Twitter Gem in your Rails Application

Using the Twitter Gem in your Rails Application

Recently, a fellow developer I know expressed his frustrations regarding Twitter’s update to its API. “It seems the days of loading up a simple jQuery to fetch a user’s Tweets are gone,” he lamented. Not to worry, because savvy Ruby on Rails developers can remedy this situation in about 15 minutes while gaining some new functionality “for free.”

Start by opening your Gemfile and adding the Twitter gem.

Read more

Using Amazon’s Simple Workflow Service from Rails

Using Amazon’s Simple Workflow Service from Rails

One of the projects I’m working on involves scraping information from Amazon product listings. Currently it uses delayed job, but there’s an issue with long-running processes dying off.
To deal with this issue, we are considering a switch to Amazon’s Simple Workflow Service. It moves the task queue to Amazon’s servers and client programs, then polls for tasks.
For the purpose of this article, I’m going to do a simplified version of the application that just pulls down prices. The data structures for this are pretty straightforward:

  • rails new swf_scraper
  • rails generate scaffold Product asin:string
  • rails generate scaffold Record price:float product_id:integer

Connect the records to the products by adding has_many and belongs_to to the product and record models respectively.

Accessing the workflow service requires adding aws-sdk and aws-flow gems to the Gemfile.

SWF uses workflows to define the order of activity execution. The workflow for this project is:

class ScrapeWorkflow extend AWS::Flow::Workflows
workflow :queue_scrape do
{
:version => "1.1",
:task_list => SWF_WORKFLOW_TASK_LIST,
:execution_start_to_close_timeout => 10 * 60,
}
end

activity_client(:activity){ {:from_class => "ScrapeActivity"} }

def queue_scrape(asin)
scrape_future = Future.new.set
scrape_future = activity.send_async(:scrape, asin)
# wait_for_all(scrape_future)
end
end

There is just one task in this workflow: scrape the asin. The activity is where the actual processing takes place:

class ScrapeActivity
extend AWS::Flow::Activities

activity :scrape do
{
:version => "1.1",
:default_task_list => SWF_ACTIVITY_TASK_LIST,
:default_task_schedule_to_start_timeout => 10 * 60,
:default_task_start_to_close_timeout => 30,
}
end

def initialize
@count = 0
end

def scrape(asin)
begin
@count += 1

url = "http://www.amazon.com/dp/" + asin
response = HTTParty.get(URI.encode(url))
doc = Nokogiri::HTML(response)

price_div = doc.at_css('.priceLarge')
price = (price_div.nil? or price_div.text[/[0-9\.,]+/].nil?) ? nil : price_div.text[/[0-9\.,]+/].gsub(/,/, '').to_f

unless price
price_div = doc.at_css('.a-color-price.a-size-large')
price = (price_div.nil? or price_div.text[/[0-9\.,]+/].nil?) ? nil : price_div.text[/[0-9\.,]+/].gsub(/,/, '').to_f
end

if price
product = Product.find_by_asin(asin)
product.records.create( price: price )
end

puts "#{@count} Scraped: #{asin}: #{price}"
rescue => e
puts "Error: #{e.message}"
end
end
end

The last piece of the puzzle is actually queuing the jobs and running the workflow and activity. This is accomplished with a rake task:

require "#{Rails.root}/app/helpers/application_helper"
include ApplicationHelper<

require "#{ENV['GEM_HOME']}/gems/aws-flow-1.0.0/lib/aws/decider.rb"
require "#{Rails.root}/config/initializers/swf.rb"
require "#{Rails.root}/lib/scrape_activity.rb"
require "#{Rails.root}/lib/scrape_workflow.rb"

namespace :swf do
desc 'Start activity worker'
task :activity => :environment do
swf, domain = swf_domain
activity_worker = AWS::Flow::ActivityWorker.new(swf.client, domain, SWF_ACTIVITY_TASK_LIST, ScrapeActivity) { {:use_forking => false} }
activity_worker.start
end

desc 'Start workflow worker'
task :workflow => :environment do
swf, domain = swf_domain
worker = AWS::Flow::WorkflowWorker.new(swf.client, domain, SWF_WORKFLOW_TASK_LIST, ScrapeWorkflow)
worker.start
end

desc 'Queue activities'
task :scrape => :environment do
swf, domain = swf_domain
my_workflow_client = AWS::Flow::workflow_client(swf.client, domain) { {:from_class => "ScrapeWorkflow"} }

Product.all.each do |product|
$workflow_execution = my_workflow_client.start_execution(product.asin)
end
end
end

Setting up the client and domain is done by a helper method:

module ApplicationHelper
def swf_domain
config_file = File.open("#{Rails.root}/config/aws.yml") { |f| f.read }
AWS.config(YAML.load(config_file))

@swf = AWS::SimpleWorkflow.new
begin
@domain = @swf.domains.create(SWF_DOMAIN, "10")
rescue AWS::SimpleWorkflow::Errors::DomainAlreadyExistsFault => e
@domain = @swf.domains[SWF_DOMAIN]
end

return @swf, @domain
end
end

Parsing Excel Files in Rails

Parsing Excel Files in Rails

I’m doing a project where I need to import a spreadsheet, parse it, and then insert the data into the database.  Now my first concern for finding a gem was that it could support more than one format of spreadsheet file.  My second concern was that I wanted to be able to easily iterate over sheets, rows, and columns.  My third and last concern was if it’s still being actively developed.  While doing research for reading spreadsheet files I found 2 primary candidates.

The first candidate is the Spreadsheet gem.  This was the first gem I tried and could easily iterate over sheets, rows, and columns.  It was also still being actively developed.  But the primary disadvantage is that it will only work with .xls files.  Here is an example of how to use this spreadsheet gem to iterate over sheets and rows.

Read more

Should a developer be a jack of all trades… or not?

Should a developer be a jack of all trades… or not?

As a freelancer or small development group, a common piece of advice is to focus and not try to be a jack-of-all-trades, as that hinders clients from finding you. My advice is to take that one step further: try to calculate an expected value from a technology in order to determine where to focus. But keep in mind the size of the market: while it is helpful to be a big fish in a small pond, if the pond is too small, you’ll starve from lack of food.

In probability, an expected value is found by multiplying the probability of an event with the value of that event to determine an expected ‘payout’. For software development, I find the following equation helpful: number of jobs/size of competition pool * average hourly rate, i.e., with all other things being equal, what is an expected payout for a given technology.

Read more