I recently presented at an Amazon (AWS) EC2 user group meet-up in Toronto, Canada. The idea was to demo some popular tools being used today rather than only provide a Ravello product demo. It turned out to be a very technical dynamic group with lots of interactive discussion and technical questions. As promised, I put together a detailed blog that described the talk and provides some technical details for anyone who wants to try it out. In the end I hope this blog is detailed enough for anyone interested in learning the various technologies to integrate some cool tools like Jenkins and Chef with Ravello Systems.
Here is a link to the presentation, most of the real good content is in the blog
First off "Why should you be looking at Continuous Deployment?" and "How does Ravello's solution get this concept off the ground so quickly?". With Continuous Deployment, you need to move from big, slow, and expensive processes to small, frequent and continuous ones. You need to know that this will happen in an automated fashion and be able to receive quick feedback on the test results. This gives everyone involved in the pipeline quick feedback and in most cases accurate results. Generally people move in this direction to be "Agile" or drive "Agility" in you project. If also a requirement is you ever want to move even closer to Continuous Deployment where every change goes through the pipeline and automatically gets put into production.
In most cases you need a dedicated virtual environment for this testing and lots of custom scripts and tools just to build the underlying framework or infrastructure to run the initial tests. (Compute, Network, Software stack and so on…). Ideally this is all kicked off with a single call, (Ideally via an API). You will see in the blog below that Ravello's solution makes this super easy and also leverages the agility and seamless capacity of the public cloud. Your able to pay for the capacity you need and also move to a cost model where you pay for the running time of your test(s). You will see below the concept of a known "blueprint" is the perfect starting point to drive "Continuous Deployment.
Feel free to check out more about "Continuous Deployment / Integration" concepts.
Here is some information on how to get it all working and learn how you can completely automate application deployment via Jenkins and Chef. This can be a first step in getting to Continuous Deployment and Continuous Integration with an application of your choosing.
Let's start by creating a new application inside your Ravello Organization. First sign up for a free trial. Here is a link to our developer centre and API.
- Drag your virtual machine into your new application canvas (I am using a base Ubuntu 12.04 I uploaded based on a .qcow2 file)
- We provide a base image in your organization that will get you started.
- Go ahead and configure your ubuntu server to you liking… you can see the VM Editor on the right after you click on the vm.
- You will also want to create a key pair so you can ssh and login to this vm when it is booted up.
This is the base image you can use from your new ORG:
You can create the key pair here.
Go ahead and rename you vm to jenkins-01
My base ubuntu vm does not use a key pair for example (FYI - we also support installing from an .ISO)
I am going to use a 20GB file system to get started.
I will keep the networking simple and use the DHCP service provided by the Ravello Service.
Let's make sure we open up some service ports so we can access the virtual machines from our local client.
Jenkins will come up on port 8080 so we need to open that up, also port 22 so we can ssh to the virtual machine.
Alright we are ready to publish our (soon to be) Jenkins virtual machine.
In this example I want to make sure I deploy to Amazon - Virginia, I choose performance optimized and will run for 6 hours.
* Note - When Ravello auto stop's your application we take a full snap shot / backup so you can simply come back and start it up at any time you like.
You will see our your new virtual machine will start to deploy - you can get the connection information from the bottom left corner.
When you deploy to the public cloud you can choose to have a public IP Address, You can also choose to have an Elastic IP Address.
You can make these selections under the VM Editor I described above.
External Access Section
You will see you machine boot-up in a few minutes - you click on the console option in the bottom left corner.
If you are quick you can watch the machine boot.
You can connect to the console using the button in the bottom right corner.
*if you configured your vm with a key pair you will have to ssh to the public ip using your ssh key.
example - ssh -i
Now it is time to make sure we can ssh to our new virtual machine. You can get the connection information from the bottom righ corner after you click on the vm.
ssh to your virtual machine (note I don't have a key pair configured)
ssh ubuntu@85.190.182.100
I like to set my host name right away (if we had cloudinit installed in this virtual machine Ravello would look after that for us).
ubuntu@ubuntu:/$ sudo su
root@ubuntu:/# nano /etc/hostname
root@ubuntu:/# reboot
Here is what my filesystem looks like
ubuntu@jenkins-01:/$ sudo su
root@jenkins-01:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 786M 18G 5% /
udev 998M 8.0K 998M 1% /dev
tmpfs 401M 204K 401M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1002M 0 1002M 0% /run/shm
root@jenkins-01:/#
Time to install Jenkins
Before we can install Jenkins, we have to add the key and source list to the vm.
This is done in 2 steps, first we'll add the key.
root@jenkins-01:/# wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
OK
Next run this..
echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list
I like to make sure I update my ubuntu server before I move on...
apt-get update
We can proceed installing Jenkins. Note that Jenkins has a whole bunch of dependencies, so it might take a few moments to install them all.
apt-get install jenkins
you will see lots going on here…..
Adding debian:Go_Daddy_Class_2_CA.pem
done.
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
root@jenkins-01:/#
Not it's time in install the Ravello SDK, there are a few pre requisites so let's look after those...
Install the following packages
sudo apt-get install python-pip python-dev build-essential
$ sudo pip install --upgrade pip
Install pip
sudo easy_install pip
Then install Ravello-SDK
sudo pip install ravello-sdk
Ravello SDK get's installed int the following location:
root@jenkins-01:/usr/local/bin# ls
pip pip2 pip2.7 python-scripts ravello-create-nodes ravello-set-svm ravello-set-uuid
root@jenkins-01:/usr/local/bin#
Let's first test all this out manually before we move onto the Jenkins piece.
I like to create a directory in here for my scripts - I called it "python-scripts"
Jenkins is installed in /var/lib/jenkins
Create a file and drop in the script below
sudo nano python-scripts.py
For our test we need a blueprint - let's just make a blueprint out of our existing application we are using.
From the Ravello UI - Click "Save as Blueprint" - and give it a name.
Optional - If you want to test via the python script before you do the Jenkins integration.
Here is the python script… we have some sections commented out as we are just testing to see if this all works...
If you are wondering where you get the blueprint ID - choose a blueprint from inside the Ravello UI - look at the URL showing - the ID is in that URL.
Go under Library - then blueprints - click on your new blueprint.
Example - 52396048
https://cloud.ravellosystems.com/#!blueprint;appid=52396048;pkey=p3
#here is our manual script - this will deploy to a cost optimized cloud
#manual python script
import os
from ravello_sdk import *
import time
def createClient():
client = RavelloClient()
client.connect()
client.login ('your_userid', 'your_password')
return client
client = createClient()
#gets blueprint design, sets the name to be that of the application we intend to publish and creates+publishes application
bp = client.get_blueprint(int('your_blueprint_id'))
bp['name'] = 'your_application_name'
del bp['id']
app = client.create_application(bp)
client.publish_application(app)
#gets the application and waits until all vms are published
app_id = app['id']
app = client.get_application(app_id)
numOfVms = len(app['deployment']['vms'])
started = [False for i in xrange(numOfVms)]
allStarted = False
while not allStarted:
app = client.get_application(app_id)
for it in xrange(numOfVms):
started[it] = (app['deployment']['vms'][it]['state'] == "STARTED") #note that this assumes all vms start and there is no error
allStarted = reduce(lambda x, y: x and y, started)
time.sleep(30)
#logs out of the ravello client
client.logout()
client.close()
#we will use this section later on
#writes the names and dns addresses, the format is arbitrary for readability and can be changed as you see fit
#filename = "{0}{1}-{2}.params".format(os.getenv('JOB_NAME'), os.getenv('BUILD_NUMBER'), 'dns')
#with open(filename, 'w+') as parFile:
# parFile.write("Name - DNS pairs of the machines in the application {0} \n".format(app['name']))
# for vm in app["deployment"]["vms"]:
# parFile.write("{0}={1} \n".format(vm['name'], str(vm["networkConnections"][0]["ipConfig"]["fqdn"])))
# for service in vm["suppliedServices"]:#writes the external port of any external service (for the case of port forwarding)
# if service["external"]:
# parFile.write(" {0}={1} \n".format(service['name'], service["externalPort"]))
Save that file and let's give it a try...
root@jenkins-01:/usr/local/bin/python-scripts# Is
python_manual.py
Run this after you saved your changes...
root@jenkins-01:/usr/local/bin/python-scripts# python python_manual.py
f you flip back to you Ravello UI - you should see your application get created and then publish to the public cloud
If yes you are good and we can move on to the Jenkins integration…
Let's open a web browser and go to the Jenkins URL - Again you get this from the bottom right section of the UI after you click on the vm. -
example - http://85.190.182.100:8080 - you can also just click the open link.
In my case - here...
http://85.190.182.100:8080/
You should see Jenkins ready to go… We don't have any security setup for this example...
Let's first install the "python" plugin...
Make sure you click on the "Available" tab and search for python.
*If nothing comes up here, click on "Advanced" and update at the bottom of that screen.
than - "download and install after reboot"
Now let's create a new item or project
Choose a freestyle project
You can follow along on the sections I make...
Since we want to automate this and not have to edit the script each time - we want this to be a "parameterized" project.
I included the parameters below for your reference.
Here are the "parameters"
Below we want to select "Execute Python Script" this comes from our new plugin...
Here is the Script we want to run… This time you don't need to edit the script - we will run it with "parameters".
#this will be the script we use for this one - note we deploy to Amazon Virginia in the example
#jenkins python script
import os
from ravello_sdk import *
import time
def createClient():
client = RavelloClient()
client.connect()
client.login( os.getenv('ravello-username'), os.getenv('ravello-password') )
return client
client = createClient()
#gets blueprint design, sets the name to be that of the application we intend to publish and creates+publishes application
bp = client.get_blueprint(int(os.getenv('blueprint_id')))
bp['name'] = os.getenv('application_name')
del bp['id']
#app = client.create_application(bp)
#client.publish_application(app)
#gets blueprint design, sets the name to be that of the application we intend to publish and creates+publishes application
#bp = client.get_blueprint(int('52134098'))
#bp['name'] = 'application_name23123456'
#del bp['id']
app = client.create_application(bp)
req={"preferredCloud": "AMAZON",
"preferredRegion": "Virginia",
"optimizationLevel": "PERFORMANCE_OPTIMIZED",
"startAllVms": "true"}
client.publish_application(app,req)
#app = client.create_application(bp)
#client.publish_application(app)
#gets the application and waits until all vms are published
app_id = app['id']
app = client.get_application(app_id)
numOfVms = len(app['deployment']['vms'])
started = [False for i in xrange(numOfVms)]
allStarted = False
while not allStarted:
app = client.get_application(app_id)
for it in xrange(numOfVms):
started[it] = (app['deployment']['vms'][it]['state'] == "STARTED") #note that this assumes all vms start and there is no error
allStarted = reduce(lambda x, y: x and y, started)
time.sleep(30)
#logs out of the ravello client
client.logout()
client.close()
#writes the names and dns addresses, the format is arbitrary for readability and can be changed as you see fit
filename = "{0}{1}-{2}.params".format(os.getenv('JOB_NAME'), os.getenv('BUILD_NUMBER'), 'dns')
with open(filename, 'w+') as parFile:
parFile.write("Name - DNS pairs of the machines in the application {0} \n".format(app['name']))
for vm in app["deployment"]["vms"]:
parFile.write("{0}={1} \n".format(vm['name'], str(vm["networkConnections"][0]["ipConfig"]["fqdn"])))
for service in vm["suppliedServices"]:#writes the external port of any external service (for the case of port forwarding)
if service["external"]:
parFile.write(" {0}={1} \n".format(service['name'], service["externalPort"]))
Now let's run our new Jenkins job!
Go back to main Jenkins screen and find your new job - click on it and then select "Build with Parameters".
You will be asked to input you information - these are the items we edited manually earlier.
After you hit build - you can look click on the running job on the left and then look at the Console Output.
Here is the output
You will notice your application has been deployed to the cloud...
Look inside you will see all the vm's being published...
In my example I had a windows vm deployed in my blueprint also
Have a look at the Jenkins console - this is what Success looks like...
After a couple minutes you will see all the vm's are started and are accessible.
Now we can move on to doing some additional activities with our new application environment.
But we need to know all the information about the virtual machines.
Our script used the API to obtain all the information after it was deployed and started.
We wrote that information here on the Jenkins virtual machine.
/var/lib/jenkins/jobs/ravello-pythonsdk-demo/workspace
If you look inside the job file - all your vm information is there...
root@jenkins-01:/var/lib/jenkins/jobs/ravello-pythonsdk-demo/workspace# cat ravello-pythonsdk-demo3-dns.params
Name - DNS pairs of the machines in the application awsdemoapp
jenkins-01=jenkins01-awsdemoapp-3cuoxeci.srv.ravcloud.com
web=8080
ssh=22
Windows2k8r2-01=windows2k8r201-awsdemoapp-0ur6h6st.srv.ravcloud.com
rdp=3389
In conclusion, this was a quick example on how to get started with Ravello Systems and also integrate Jenkins to start to drive some nice automation and start to embrace the inner devop's in all of us...
In my next blog I will show you how to use this same process but integrate your virtual machines with Chef. This will allow you to dynamically build you application components each and every time after a blueprint is deployment.
Good luck, let us know how you get along...
The post Step-by-step guide: Setting up CI/CD with Ravello, Jenkins and Chef on Amazon EC2 appeared first on The Ravello Blog.