RightGrid Basic Example

Objective

The purpose of this tutorial is to introduce you to some of the key concepts about RightGrid and demonstrate how to set up a RightGrid application of your own. 

In this example, you will build a "Hello World" RightGrid application that demonstrates the batch processing functionality of the RightGrid system.

NOTE: This tutorial only applies to Grid or Premium accounts.  If you have a Developer account and would like to upgrade, please contact sales@rightscale.com.

Step 1 - Create SQS Queues (Input and Output)

The first step is to create your SQS Queues.  You'll need to create an SQS Input Queue to receive work unit messages from the job producer and an SQS Output Queue to receive the result messages after the work unit has been processed. Optionally, you can create an SQS Audit Queue where RightGrid will send its audit entires. 

Go to Manage -> Queues.  Click the New Queue button. 

• Queue Name - a nickname for the SQS queue.  NOTE:  You will need to create unique names.
• API Generation - the generation of the SQS Queue API. Select 2. 
  NOTE:  All SQS Queues for RightGrid must use the second generation API.
• Visibility Timeout - after a worker instance takes a work unit from the input queue, the input queue will "hide" the job in order to prevent multiple worker instances from taking the same job.   The visibility timeout specifies the time in seconds that a job will remain hidden before it becomes available to other worker instances.  If a worker instance successfully receives a work unit, it will send a message to delete the work unit from the input queue.  The visibility timeout ensures that if a worker is unsuccessful receiving a work unit, the work unit will become visible and available again to other worker instances.  (Default = 30)

Click the Create button.  A confirmation window will appear where you can add a message to the queue as a test.  

Go back to Clouds -> AWS -> Queues and repeat the process by creating an SQS Output Queue.

You should now have an input queue and output queue.

NOTE: Similar to S3 bucket names, SQS Queue names must be unique.

Step 2 - Create an S3 Bucket

You will need to create an S3 bucket to store work unit data.  You can use the same S3 bucket for storing input and output (result) data.  

Go to Manage -> Storage -> S3 Browser and click the New Bucket button.

• Bucket Name - a nickname for your S3 bucket.  NOTE:  You will need to create a unique bucket name.
• Location - the geographic location of your S3 bucket.  Select default.

Step 3 - Create a Job Producer

The job producer does not have to be running on EC2.  It can be located anywhere on the Internet.  The code for the job producer and job consumer can be written in any programming language provided that you can upload/download data to S3 and send/receive work units from SQS queues.

A job producer performs the following tasks:

• It breaks down data-sets into individual work units.
• It upload a work unit's matching input data files to a specified bucket on S3.
• It sends a message about a work unit to the SQS Input Queue.  NOTE: Since the RightGrid daemon on a worker instance will automatically grab work units that become available in the input queue, it's important that the associated input data files for the work unit are already available on S3.

To create a job producer follow the steps below.  

1. Install the RightScale AWS interface gem (right_aws).
2. Add code to upload the input data files to the specified bucket on S3.  (See '# Get S3 and SQS handle')
3. Add code to generate and encode a work unit.  (See '# Get S3 and SQS handle')
4. Add code to send a work unit's message to the SQS input queue.

Similar to SQS messages, input queue messages are limited to 256KB. In most cases, the input queue messages contain only the work unit meta-data while the actual input data files for the worker application are uploaded to S3.

require 'yaml'
require 'rubygems'
require 'right_aws'

def upload_file(bucket, key, data)
  bucket.put(key, data)
end

def enqueue_work_unit(queue, work_unit)
  queue.send_message(work_unit)
end

# Load jobspec
jobspec = YAML::load_file("oneshotspec.yml")

# Get S3 and SQS handle
s3 = RightAws::S3.new(jobspec[:access_key_id], jobspec[:secret_access_key])
bucket = s3.bucket(jobspec[:bucket], false)
sqs = RightAws::SqsGen2.new(jobspec[:access_key_id], jobspec[:secret_access_key])
inqueue = sqs.queue(jobspec[:inputqueue], false)

# Generate work units
for id in 1...(jobspec[:number_of_units]+1)
  puts "Generating work unit #{id}"
  filename = "in/Log#{id}.log"
  text = "HelloWorld!"
  
  work_unit = {
    :created_at => Time.now.utc.strftime('%Y-%m-%d %H:%M:%S %Z'),
    :s3_download => [File.join(jobspec[:bucket], filename)],
    :worker_name => jobspec[:worker_name],
    :id => id,
  }

  wu_yaml = work_unit.to_yaml
  upload_file(bucket, filename, text)
  enqueue_work_unit(inqueue, wu_yaml)
  puts wu_yaml
end

--- 
:name: OneshotJob 
:worker_name: RGHelloWorld
:number_of_units: 5000 
:bucket: dw_rightgrid_demo
:inputqueue: RG-Inputs
:outputqueue: RG-Outputs
:access_key_id: <AWS_ACCESS_KEY> 
:secret_access_key: <AWS_SECRET_ACCESS_KEY>

Step 4 - Create a Job Consumer

Similar to the job producer, the job consumer can be located anywhere on the Internet. The job consumer typically parses a work unit's result message and data files.  It can also update a central database, if necessary. 
 

To create a job consumer that's compatible with the RightGrid framework:

1. Install the RightScale AWS interface gem (right_aws).
2. Add code to receive messages from the SQS output queue and decode the message.
3. Add code to download the result files from S3.
4. (Optional) Update a central database with the result data files.

require 'rubygems'
require 'yaml'
require 'right_aws'


def download_result(bucket, key)
    bucket.get(key)
end


def dequeue_entry(queue)
   queue.pop
end

# Load jobspec
jobspec = YAML::load_file("oneshotspec.yml")

# Get S3 and SQS handles
s3 = RightAws::S3.new(jobspec[:access_key_id], jobspec[:secret_access_key])
bucket = s3.bucket(jobspec[:bucket], false)
sqs = RightAws::SqsGen2.new(jobspec[:access_key_id], jobspec[:secret_access_key])
outputqueue = sqs.queue(jobspec[:outputqueue], false)

# Continually Pop messages off the result queue
while true do
  msg = dequeue_entry(outputqueue)

  #Here is where you would:
  #  1. Decode msg
  #  2. Download result files from s3
  #  3. Update a central database/update job statistics
end

Step 5 - Create a RightGrid configuration file (rightworker.yml)

The rightworker.yml configuration file is the heart of a RightGrid application. The config file sets variables needed by the RightGrid worker daemon in order to call the user's application with the correct parameters.

The rightworker.yml file contains the following parameters:

• Defines the environment (ex: development, staging, production).
• Defines how to upload the results back to S3.
• Defines how to send result messages to the corresponding queues.
• Includes your AWS access and secret access keys.

 The sample code below was written in Ruby.  Use this code as a template for creating your own job producer.

rightworker.yml

development: 
RightWorkersDaemon: 
        aws_access_key: <AWS_ACCESS_KEY>
        aws_secret_access_key: <AWS_SECRET_ACCESS_KEY> 
        log: RGHelloWorld.log
        email: yourName@yourSite.com
        halt_on_exit: true
        workers: 1
        user: 
            custom_entry_a: user_entry_1
            custom_entry_b: user_entry_2
        queues:
            rg_example_inputs:
                invocation_model: oneshot
                result_queue: rg_example_results
                audit_queue: rg_example_audit
                message_decoder: RightYamlDecoder
                s3_log: dw_rightgrid_demo/log/%{DATE}/%{MESSAGE_ID}
                s3_out: dw_rightgrid_demo/out/%{DATE}-%{TIME}-%{MESSAGE_ID}
                receive_message_timeout: 3600
                default_worker_name: RGHelloWorld
                life_time_after_fault: 7200
                s3_in: /tmp/s3_in
                s3_in_delete: false
                s3_in_overwrite: false
                s3_in_flat: true

In the sample code above, the parameters are defined in three sub-sections.

• # Environment
  
  • # RightWorkers Daemon
    
    • # User
    • # Queue

Step 6 - Create your application

RightGrid supports two ways of invoking the user's application: one shot or persistent.  In this example, we are using the 'one shot' invocation model where a new process is created for each new work unit that is received by a worker.

RGHelloWorld.rb

class RGHelloWorld
  
  def do_work(message_env, message)
    starttime = Time.now
    
    for j in 0...10000 do
      2 + 2
    end  

    finishtime = Time.now  

    result = {
      :result => 0,
      :id => message_env[:id],
      :starttime => starttime,
      :finishtime => finishtime,
      :created_at => message_env[:created_at],
      :output => "Goodbye World!"
    }
  end

end

Where message_env is given by the following hash:

message_env = { 
  'tmp_dir' => “Directory for temp files”, 
  's3_in' => "Directory with the files downloaded from S3", 
  'output_dir' => "Directory where the app or kicker should put output files to be 
uploaded to S3", 
  'log_dir' => "Directory where the app or kicker should put log files to be 
uploaded to S3", 
  'log_file' => "File for right_worker logs', 
  'message_id' => "SQS message id", 
  'controller' => "RightWorkersDaemon", 
  'worker_name' => "name of user worker" 
  'logger' => "handle to the RightGrid logger object" 
      ‘s3_downloaded_list’ => {‘bucket/key1’=> ‘local_file1’, ..., ‘bucket/keyN’=> 
‘local_fileN’} 
}

Step 7 - Configure an EC2 'Worker' ServerTemplate

Each worker instance must be created with the same ServerTemplate.  RightScale provides the RightGrid Worker ServerTemplate, which is already configured to install RightGrid and automatically start the rightworker daemon.  Simply configure the ServerTemplate with the appropriate input parameters such as your SVN repository and add any custom RightScripts for your application and its dependencies.

Go to Design -> ServerTemplates.  Under the RightScale tab, filter by 'worker' and select the latest version of the RightGrid Worker template.

Since you will probably customize the ServerTemplate, click the Clone button and rename the ServerTemplate "Worker."

Now go to the cloned ServerTemplate's Inputs tab and click the Edit link.

Each instance will need to be able to access your SVN repository in order to download your application code.

Define the the following input parameters. 

• MON_PROCESS - Set to Ignore (default).
• OPT_GEMS_LIST - Set to right_aws (default).
• RAILS_ENV - Select the environment where you would like to run the RightGrid daemon.  (ex: development, staging, production).  You can only run a RightGrid daemon on one environment at a time.  Enter development to match the environment that's defined in rightworker.yml.
• SVN_PASSWORD - the password to your SVN repository.
• SVN_URL - the URL to your SVN repository.
• SVN_USER - the username to your SVN repository.
• SYSLOG_SERVER - Enter syslog.rightscale.com (default).

Step 8 - Create a Queue-based Server Array

The last step is to create a scalable server array for all worker instances.

You can create two types of server arrays, based on how you want the server array to resize.  You can either create a server array that will resize based on the number of jobs in the queue or based on the amount of time a job is in the queue.  For more information, see Server Arrays.

In this example we will create a server array based on the number of jobs in the queue. 

Go to Clouds -> AWS -> Queues and click the New button.

• Nickname - 
• Server template - select the template that will be used to create the worker instances in the server array. Select: Worker from your Private ServerTemplates.
• SSH key - select the SSH key that will be used by all worker instances in the server array. Select your SSH key.
• Security Group - select the security group that will be used by all worker instances in the server array. Select your Security Group.
• Deployment - select a deployment for the server array.  In this example, we've already created a "RightGrid Example" deployment. 
• Default min count - the minimum number of worker instances that must be operational in the server array.  You can set this value to zero to avoid idle server usage costs.  But, if you have time-sensitive tasks that need to be processed immediately, you might want to use a value of at least one worker instance.  Default = 1
• Default max count - the maximum number of worker instances that can be operational at any given time in the server array. Default = 20
• Elasticity function - defines how the server array should grow/shrink. Select sqs_queue_size. 
• Elasticity params - defines the elasticity parameters that will be used to determine when a server array should be resized.
  
  • Items per instance - defines the ratio of worker instances per items in the queue.  Ex: If there are 50 items in the queue and "Items per instance" is set to 10, the server array will resize to 5 worker instances.  Enter: 20
• Indicator - select the input SQS queue that contains all of the job tasks.  (ex: RG-Inputs)
• Audit - (optional) select the audit SQS queue that will store audit entries.  Set this value to -none-.

Click Save.

The next step is to activate your server array.  Click the enable link.

Step 9- Launch a Worker Instance and Test the Results

You're almost ready to launch your RightGrid application. 

The first step is to start the job producer to put work units into the SQS Input Queue.  Now go to your input queue to see the number of work units.  Go to Manage -> Queues.

Next, go to your RightGrid deployment (ex: RightGrid Example).  Notice that the "RG Worker Array" is active and running. 

Click the Launch button.  An input confirmation window will appear.  Confirm that you are using the correct launch inputs and click the Launch button.

Since we have 500 work units sitting in the input queue and we specified "20 items per instance" 25 instances would be launched, but since we defined a maximum of 20 instances in the server array, only 20 instances were launched.

Now check your SQS Queues.  You should now see the number of jobs in the output queue slowly increase as more jobs from the input queue are successfully processed.

NOTE:  The job producer and job consumer can run continuously.

Eventually all of the jobs will be processed and the server array will eventually  resize down to 1 worker instance. 

Congratulations!  You've just created a basic RightGrid application and have seen how powerful and easy it is to use cloud computing resources for more efficient batch processing tasks.

 ----------------------

Did you find this document helpful?  Please feel free to leave us a comment below so that we'll know how we can improve our documentation.  Thanks!