Using EC2 on-demand with Jenkins

Amazon Elastic Compute Cloud on-demand

In order to maintain the quality of your software, the best way would be to start with automating the regression suite. Automated tests can be executed as frequently as we wish to, run very fast, and at any time.

Using CI tools such as Jenkins can facilitate the execution of your automated test suite. All of this can bring you another obstacle: how to run a regression suite without leaving your machine on 24/7? Regression suite usually runs from about 30 minutes to a few hours, depending on a number of the test cases and complexity of the corresponding project. Execution time can be reduced using execution in parallel, so the time needed for execution will decrease. One of the solutions for this problem is to use the Amazon Elastic Compute Cloud (Amazon EC2).

Amazon EC2

Let’s start with the description from the AWS official page:

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use. As described above, Amazon EC2 enables you to use its instances on-demand. That means that you can scale the desired capacity, use as many server instances as you need and run each instance exactly when you need it. Considering all of this, using EC2 on-demand would be a perfect solution for creating an isolated test environment where automation test suite can be executed. Another benefit is cost since you are paying the instance only when using it.

As described above, Amazon EC2 enables you to use its instances on-demand. That means that you can scale the desired capacity, use as many server instances as you need and run each instance exactly when you need it. Considering all of this, using EC2 on-demand would be a perfect solution for creating an isolated test environment where automation test suite can be executed. Another benefit is cost since you are paying the instance only when using it.

On-Demand Instances

To start using on-demand instances, the very first step would be to launch an Amazon EC2 instance using your AWS account. Launch screen gives you the freedom to choose any configuration that satisfies your needs. Launch wizard on AWS is pretty detailed and will not be a part of this article.

After you are done with configuration and installation on your new EC2 instance, the next step would be to create a new image from the existing EC2 instance. Creating a new image for the selected EC2 instance is displayed below:

1. Create image

    2. Set configuration values

3. Track AMI’s status

As soon as the status turns to available, Amazon Image (AMI) is ready for use. This means that you can stop the EC2 instance and run it on-demand anytime you need it.

Using EC2 on-demand with Jenkins CI

Integrating EC2 with Jenkins requires the Amazon EC2 plugin.

4. Installing Amazon EC2 plugin on Jenkins

After the new plugin is installed, go to the Configure system and add a new Cloud Amazon EC2 section.

5. Amazon EC2 configuration on Jenkins

The picture above represents a sample configuration for Amazon EC2. Jenkins uses access key ID and secret access key to interface with Amazon EC2. EC2 Key Pair’s Private key is a key generated when creating a new EC2 image on AWS.

Configure new AMI on Jenkins

The next step is to add a new AMI to the Jenkins configuration. As displayed in the picture below, it is necessary to set an AMI ID from AWS. Then, choose the instance type and set Idle termination time. Idle termination time is a value that determines how long slaves can remain idle before being stopped or terminated.

6. AMI configuration on Jenkins

Route 53 – scalable DNS

Each time a new image starts, it gets different DNS. To solve the problem of scalable DNS, Amazon has Route53 Hosted zone. Let’s start with a brief description from the Amazon website:

Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. It is designed to give developers and businesses an extremely reliable and cost-effective way to route end users to Internet applications by translating names like www.example.com into the numeric IP addresses like 192.0.2.1 that computers use to connect to each other.

The first step is to create a new Amazon hosted zone. When having a new zone, create a new record and set it’s value to AMI’s current DNS.

7. Create a new record for Hosted Zones on AWS

The next step is to add init script to AMIs configuration. As displayed in picture 6, init module runs ruby route53.rb script with three parameters: SERVER_HOSTNAME, HOSTED_ZONE_ID, and RECORD. HOSTED_ZONE_ID and RECORD have been discussed above, while creating a new record and hosted zone. For the previous example, RECORD=“slave_instance.abhapp.com”.

SERVER_HOSTNAME uses cURL tool to get metadata for a specific instance, which will be used to manage the running instance. The entire script is displayed below:

require 'fog'

hosted_zone_id = ARGV[0]
record = ARGV[1]
new_value = ARGV[2]

dns = Fog::DNS.new({:provider => 'AWS'})

zone = dns.zones.get(hosted_zone_id)
record_name = zone.records.get(record)

change_batch_options = [
  {
    :action => "DELETE",
    :name => record,
    :type => "CNAME",
    :ttl => 5,
    :resource_records => [ record_name.value.first ]
  },
  {
    :action => "CREATE",
    :name => record,
    :type => "CNAME",
    :ttl => 5,
    :resource_records => [ new_value ]
  }
]

dns.change_resource_record_sets(hosted_zone_id, change_batch_options)

8. route53.rb script

After configuring AMI is all done, Jenkins will automatically start instances when the load goes up, and the instance will be terminated automatically if its idle time is more than the time configured in AMIs configuration.

When should we use EC2 on-demand with Jenkins?

EC2 on-demand should be used to reduce costs in the case when our Jenkins job does not require a server that is up 24/7. It’s ideal for running the regression suite and other jobs that do not execute frequently. On the other hand, if we need a server to be up most of the time (for e.g. running the same Jenkins job for 10-15 times in one day), EC2 on-demand is definitely not a good choice.