User Tools

Site Tools


bt-tutorial:real-deployment

Preparing and deploying experiments on CONFINE testbed

Preamble

This tutorial is the fourth part of a series of four part tutorial. Many concepts and ideas shown here are described on previous parts. So, make sure that you has finished the third part of this tutorial before start this. Again, we assume that all the next commands will be executed from the ~/confine/utils/vct directory.

After completing this tutorial, we will be able to deploy and execute any experiment on CONFINE testbed (real or virtual). However, as CONFINE is under development, some topics may not be covered in detail in this tutorial. If you have more questions and/or suggestions, take a look at CONFINE Wiki webpage.

What have we missed?

Well…. we have some CONFINE research nodes configured, two with a transmission client and one more with a tracker. We tested them and now we know that are properly configured and run our experiment without problems. However, there are two basic points missed:

  1. Scalability: We configured each node manually, who served us to verify our idea but is not exportable to larger (say, 1000 nodes) slices. we need to create a CONFINE sliver template sefl-deployable. Additionally, we have to include some mechanism to execute the experiment automatically without human intervention.
  2. Logging: We ran our experiment, but we did not measure anything on it. We need some method to get information about nodes/network behavior, maybe in real-time.

We are going to address both problems in next sections.

Preparing the sliver template

One of CONFINE current limitations is that, on each sliver, a researcher can only allocate and deploy a single template. Is not posible to have connectivity from one slice to another, or deploy two different OS images inside the same slice. It is consistent with SFA concept of intra-experiments' connectivity and extra-experiments' isolation.

Technically speaking, a sliver template is nothing more, nothing less, than a filesystem directory structure that contains a complete OS. There are a lot of tutorials on Intenet that explains how to create this templates from scratch, but the fastest way to do it is taking advantage of our previous created slivers.

However, our fd02 and fd03 slivers are almost full. So, before start working we need a largest research node template - slices' size are only limited by nodes' free space. If you don't know how to generate it, we suggest you to visit our short tutorial at the end of this page.

We increased the fd01 node size up to 1G, enough for our purpose; but not too large.

:!: Remember that all slices on your sliver shares the same template. So, you have to assure that you can find the desired number of CONFINE research devices with the enough space to allocate both templates: sliver plus experiment.

Add the extra software.

root@rdfd01:~$ exit
Connection to fdbd:e804:6aa9:0002:5254:00ff:fe00:fd01 closed.
 
~$ ./vct_sliver_start 0123456789ab fd01
 
~$ ./vct_slice_attributes update all
 
~$ ./vct_sliver_ssh 0123456789ab fd01
 
root@0123456789ab_fd01:~$ apt-get install vim wget transmission-cli transmission-common transmission-daemon

:!: If at some point, it is impossible to reach the sliver try to stop it and start again (with the proper attributes update between each operation). It happens some times when the node has been stopped before stop the sliver.

In our case, we need to install wget, vim and transmission packages. The original file to be downloaded by clients will be configured on the experiment template instead been provided by the sliver or experiment template. It makes more sense, because is an experiment-specific configuration parameter (e.g. we can imagine that some researchers wants to reproduce the experiment using other kind of files, with different sizes or maybe test corrupted files) and we are saving slivers space.

Configure, as before. Stop the service.

root@0123456789ab_fd01:~$ service transmission-daemon stop
service transmission-daemon stop
 
root@0123456789ab_fd01:~$ vim /var/lib/transmission-daemon/info/settings.json

Edit the /var/lib/transmission-daemon/info/settings.json.

"rpc-whitelist-enabled": false,

Start the service, and exit the sliver.

root@0123456789ab_fd01:~$ service transmission-daemon start
Starting bittorrent daemon: transmission-daemon.
 
root@0123456789ab_fd01:~$ exit
logout
Connection to fdf5:5351:1dfd:fd01:0123:4567:89ab:01 closed.

Our sliver is now complete. It has an opentracker (that we will execute only on one sliver) and a transmission client that will start as a daemon by default on all slivers. It is time to create the template.

Stop all the slivers and enter on fd01 node.

~$ ./vct_sliver_stop 0123456789ab fd01
 
~$ ./vct_slice_attributes update all
 
~$  ./vct_node_ssh fd01

CONFINE slivers are simple linux containers filesystems, which means that can be packed and exported whenever you want simply compressing its rootfs directory as a tgz file. We will call our template debianbt32.tgz

root@rdfd01:~$ cd /var/lib/lxc/
 
root@0123456789ab_fd01:/lxc/images$ ls
01  7d
 
root@0123456789ab_fd01:/lxc/images$ cd 01/rootfs
 
root@0123456789ab_fd01:/lxc/images/01/rootfs$ tar czfv debianbt32.tgz *

Finally, you have to “download” it from the sliver to some accesible place. In our case, we uploaded the sliver on CONFINE's images server.

Preparing the experiment template

Experiment templates preparation are formally explained in details on Node system hack page. Here we are just desacribe our experiment preparation.

The experiment template is an archive file that is extracted on top its sliver root file system. The archieve should provide at least an init script and links to start the related init function in the corresponding OS format. Basic directory structure

~$  mkdir etc
 
~$  cd etc
~/etc$  mkdir rc.d init.d

(NEED MODIFICATIONS)

A researcher must provide an experimentation archive file that is extracted on top its sliver root file system. The archieve should provide at least an init script and links to start the related init function in the corresponding OS format. For example for OpenWrt this would be:

./etc/rc.d/S94confine-experiment (which is a link to ../init.d/confine-experiment) ./etc/init.d/confine-experiment start() Function stop() Function During livetime of the sliver, all slice attributes could be accessed either in uci format or bash environment variables:

/root/confine/uci/confine-slice-attributes /root/confine/bash/confine-slice-attributes (not supported yet) The SLIVER_DESCRIPTION field exp_data_url defines the URL of the experimentation archieve. A simple hello world example is given by the following URL: http://distro.confine-project.eu/misc/exp-data-hello-world-openwrt.tgz

This example experiment simply pings the public IP of all other slivers of its slice and stores measured round trip times in the directory /root/confine/data/ . It contains of two files:

./etc/rc.d/S94confine-experiment (which is a link to ../init.d/confine-experiment) ./etc/init.d/confine-experiment

Running the experiment

Last step is to assemble all the pieces together and run our bitTorrent experiment. First of all, we need to modify some configurations on VCT. We are going to create a new file named vct.conf.overrides on vct directory to easely track the changes.

VCT_TEMPLATE_URL="http://images.confine-project.eu/misc/CONFINE-owrt-310812_1626.img.gz"
 
vct_bittorrent_debian() {
    cat <<EOF
config sliver $SLICE_ID
    option user_pubkey     "$( cat $VCT_KEYS_DIR/id_rsa.pub )"
    option fs_template_url "http://images.confine-project.eu/misc/debianbt32.tgz"
 
    option exp_data_url    "file://home/<username>/confine/utils/vct/exp-data-hello-world-debian.tgz"
    option exp_name        "bittorrent-experiment"
    option vlan_nr         "f${SLICE_ID:10:2}"    # mandatory for if-types isolated
    option if00_type       internal
    option if00_name       priv
    option if01_type       public   # optional
    option if01_name       pub0
    option if01_ipv4_proto $VCT_NODE_SL_PUBLIC_IPV4_PROTO   # mandatory for if-type public
    option if02_type       isolated # optional
    option if02_name       iso0
    option if02_parent     eth1     # mandatory for if-types isolated
    option if03_type       isolated # optional
    option if03_name       iso1
    option if03_parent     eth2     # mandatory for if-types isolated
    option if04_type       isolated # optional
    option if04_name       iso2
    option if04_parent     wlan0     # mandatory for if-types isolated
    option if05_type       isolated # optional
    option if05_name       iso3
    option if05_parent     wlan1     # mandatory for if-types isolated
EOF
}

First line defines the research node template to use. As you can see, we are downloading a public CONFINE image as before, but with 1GB hard disk space. You can fins below how to create larger CONFINE research node images.

Next, we have been defined a new template based on previous debian hello-world experiment:

fs_template_url is a public bitTorrent sliver template available on our images server. You can use it if you want.

exp_data_url is the new experiment template.

exp_name is the new identification number for this experiment. It is not necessary to change this iD, but will help us to track the slice status.

If you followed the whole tutorial, probably you noticed that the slice creation process is straigh. So, it's a good idea to create your own slice creation script.

~$ vim create-slice.sh

It takes three arguments and allocates, deploys and executes the desired experiment.

#
# These are the steps to create a new sliver to join an experiment
#
 
# Parameters:
#  1: Sliver identifier (any) 
#  2: Node to allocate the sliver
#  3: Experiment template (defined in vct.conf)
 
 
# Allocating the sliver to the specified node (configuration files mainly)
./vct_sliver_allocate $1 $2 $3 
./vct_slice_attributes update $1
 
#   
# Multiple slivers can also be allocated using this syntax:
#    /vct_sliver_allocate 111111111111 f101-f104 bitorrent_experiment
#   
 
# Deploying sliver into the node (downloads and installs experiment template)
./vct_sliver_deploy $1 $2
./vct_slice_attributes update $1
 
# Starting the sliver and joining the slice (which contains the experiment)
./vct_sliver_start $1 $2
./vct_slice_attributes update $1
 
# Show slice/slivers info
./vct_slice_info

Give read/write permission and just execute it with biTorrent experiment parameters. In our case we are executing the experiment on ten (10) nodes, with the slice uID 0123456789bb.

~$ chmod r+w create-slice.sh
 
~$ sh ./create-slice.sh 0123456789bb ab01-ab10 vct_bittorrent_debian

Generating larger CONFINE research node images

Creating a new image

This method is simpler than the other, but it is also less flexible because you have to do it at the begining. The idea is generate a new CONFINE OpenWRT image, with the desired final space and use it as VCT user template.

We start, of course, with our SONFINE SDK. Go to confine directory and backup the configuration files.

~$ cd confine-dist/configs 
 
~/confine-dist/configs$ cp kernel_config kernel_config.old
 
~/confine-dist/configs$ cp owrtt_config owrt_config.old

Execute OpenWRT configuration tool on root SDK directory.

~$ cd ..
 
~/confine-dist$ make menuconfig

This command executes the OpenWRT image configuration tool that will be used for CONFINE SDK to generate the research image used inside real hardware and virtual research nodes.

Click on Target images and move your cursor downward until Root filesystem partition size (in MB) and then press Enter.

Next creen allow us to set the desired image size. We selected a 1GiB image.

Remember to save the configuration file when you Exit. Upcoming compilations will create research device images with the desired size.

~/confine-dist$ make all V=99 J=4

Done! As a result you can find your new image on images subdirectory. Now substitute the VCT template configuration URL.

~/confine-dist$ cd utils/vct
 
~/confine-dist/utils/vct$ cp vct.conf.defaults vct.conf
 
~/confine-dist/utils/vct$ vim vct.conf
# 
# the format must look like:
# <url-type><path/to/dir>/<file-to-download>.<image-type>.<compresson-type>
# valid url types: http://, https://, ftp://, file://, ssh:[port]:<user@domain>://
# valid image types: raw, img, or vmdk (vmdk only for virsh --version >= 0.9.9)
# valid compressions: tgz or gz
 VCT_TEMPLATE_URL="file://home/<your name>/confine-dist/images/CONFINE-owrt-current.img.gz" 

Finally, the VCT has to be reconfigured any time the template has been changed.

~/confine-dist/utils/vct$ ./vct_system_cleanup
 
~/confine-dist/utils/vct$ ./vct_system_install

Increasing previous image

This method (inspired by [1]) is prefereable if you already have a slice with something installed on it and you want to save some time. We are going to increase the size of a CONFINE node (named fd01) from 512MB to 1024MB.

First, stop the node.

~$ ./vct_node_stop fd01

VCT images are inside VCT_VIRT_DIR/images directory.

Notice that when images are stopped, VCT system returns the user and groups ownership to root user (you can change the user on vct.conf file). It is a good thing, because allow us make any change desired. ;)

Before continue, backup the image file.

~$ cd /var/lib/vct/images
 
~$ cp openwrt-x86-generic-combined-ext4-40-rdfd01.img openwrt-x86-generic-combined-ext4-40-rdfd01.img.bak

As you can see, images' filename coincide with VCT nodes' id.

Create a blank qemu file, this is the file we will be adding to our VM image. Create it with a size equal to the amount of extra disk space you want added to your VM. In our case we want to add an additional 512MB of space.

/var/lib/vct/images$ qemu-img create -f raw addon.raw 512M

This is what the addon file looks like after it is created.

VCT is very rigurous with image filenames, so we need to rename the original image file febore change it. After the whole process, we will delete all this extra files. Then, we wil combine both files into a single one.

/var/lib/vct/images$ mv openwrt-x86-generic-combined-ext4-40-rdfd01.img openwrt-x86-generic-combined-ext4-40-rdfd01.img.save
/var/lib/vct/images$ cat openwrt-x86-generic-combined-ext4-40-rdfd01.img.save addon.raw >> openwrt-x86-generic-combined-ext4-40-rdfd01.img

Now we have an image with the desired size.

The partitions, however, are not partitioned to take advantage of this extra space. We are going to fix-it loading a live gparted image (because we need to work from an unmounted filesystem) and resize the partition using that tool. First, download the live image and start our node using qemu tool.

/var/lib/vct/images$ wget http://sourceforge.net/projects/gparted/files/gparted-live-stable/0.13.1-2/gparted-live-0.13.1-2.iso/download
/var/lib/vct/images$ mv download gparted-live-0.13.1-2.iso
 
/var/lib/vct/images$ sudo qemu-system-x86_64 -hda openwrt-x86-generic-combined-ext4-40-rdfd01.img -cdrom gparted-live-0.13.1-2.iso -boot d

Now, simply follow the screen instructions to resize the /dev/sda2/ partition until fills all the felt-most free space.

:!: Remember delete all extra files .bak and .save created during the resize process.

Return to our working directoy, start the node and check that everything went ok.

/var/lib/vct/images$ cd ..
/home/dvladek/confine/utils/vct
~/confine-dist/utils/vct$ ./vct_node_start fd01
 
~/confine-dist/utils/vct$ ./vct_node_ssh fd01

Walkthrough video

The following video shows a the setup, configuration and execution of the experiment with an earlier version of the software.

References

bt-tutorial/real-deployment.txt · Last modified: 2012/11/16 12:28 by amin