User Tools

Site Tools


soft:node-upgrade

Node upgrade

Image types

There are two kinds of CONFINE operating system images:

  1. The generic image is the raw image which contains the CONFINE distribution with no customization. It can be found in the CONFINE distribution repository
  2. The customized image is the raw image but with the specific configuration files from the target node. It is generated by your CONFINE testbed controller, and can be download from there.

First-time installation

Both types of images have the same format: a GZip-compressed file containing a plain dump of a disk with an MBR (i.e. MS-DOS) boot loader and partition table holding partitions for the boot manager and root file system. This means that for a host not yet running the CONFINE firmware:

  1. If the host is running OpenWrt you may use the web interface or upload the image to the host (e.g. using scp) and run gunzip IMAGE.img.gz && sysupgrade -n IMAGE.img in the command line (see sysupgrade documentation).
  2. If the host is running another Unix-like system you may upload the image to the host (e.g. using scp), switch to single user mode (e.g. telinit S), stop all possible processes, unmount or remount all physical filesystems in read-only mode (e.g. mount -o ro,remount FILE_SYSTEM), dump the file using gunzip -c IMAGE.img.gz | dd of=/dev/sdX bs=1M ; sync (where /dev/sdX is your computer's main hard drive) and force a reboot.

However, if you are able to boot your host using an external USB drive, we strongly recommend that you generate a customized USB image in the CONFINE controller, download it to some computer and write it to the USB drive as root using gunzip USB_IMAGE.img.gz -c | dd of=/dev/sdX bs=4M ; sync (where /dev/sdX is the USB drive's device, see blkid). Then plug the USB drive in the node and boot it, the installation will start automatically.

The rest of this document assumes that the node is already running CONFINE firmware.

Important note: The next examples assume that the CONFINE node is using the overlay scheme. If once the node has finished all the required installation and upgrade reboots, it is still not using it (you may check it with mount | grep overlay), please execute confine.disk-parted to create the overlay and /home partitions in the remaining space of the node's hard drive. Please note that some testbeds like VCT and Community-Lab may enable automatic partitioning on installation, so please be patient and wait until the node becomes stable before proceeding to check partitions.

Note: Whenever NODE_ADDRESS is mentioned you may use any node address reachable from your computer – see Logging into a node for more information.

Preparation of the node

Nodes with invalid certificates

Due to the already fixed bug #625 in the Controller's firmware generator, your node may have an invalid API certificate. If the node is working correctly, it is recommended that you replace the certificate before upgrading so that you may keep your configuration files and deployed slivers during node upgrade using a generic image. If your node is not working, the best option anyway is to use a customized image and dump all local changes and configuration, so you may skip this section.

To check the validity of your certificate, go to the testbed controller web interface (e.g. in Community-Lab's controller) and click on the State button in the node page. If the current state appears striked out (e.g. PRODUCTION) then your certificate needs to be replaced before upgrading. To do it:

  1. Back in the node page, scroll down to Node keys and show the section. If there is a key with path /etc/uhttpd.crt.pem (i.e. the HTTPS certificate used for the node API), check its Delete? box and click on Save and continue editing. Please note that the private key is kept.
  2. Click on the Download firmware… button and generate a new firmware for the node. If the upgrade works well, you will not actually be using the image, but you may use it if something fails.
  3. Back in the node page, you will see a new key under /etc/uhttpd.crt.pem (which happens to use the same private key as before). Copy the whole key to the clipboard or some temporary file.

Now connect via SSH to your node and run the following commands:

# cp -a /etc/uhttpd.crt* /tmp
# cat > /etc/uhttpd.crt.pem << 'EOF'
… paste your key here …
EOF
# openssl x509 -in /etc/uhttpd.crt.pem -outform DER -out /etc/uhttpd.crt

The next time uhttpd starts it will use the new certificate.

Nodes with old firmware

If your node is running a CONFINE firmware version based on the Master branch before June 2015, it is highly recommended that you use current versions of the confine.* scripts mentioned below to avoid several issues during the upgrade.

To know the firmware version your node is running you may:

  • Use the testbed controller web interface (e.g. in Community-Lab's controller) by clicking on the State button in the node page of and looking at the soft_version member.
  • Query the node REST API directly with a command like wget -qO- 'http://NODE_ADDRESS/confine/api/node/' | grep soft_version.
  • As a last resort, log into the node using SSH and run cat /etc/confine.version (this version is not as reliable as the one reported by the node API).

To download the latest stable scripts to a temporary directory in your node and use them instead of the existing scripts you may run:

# in your computer
ssh root@NODE_ADDRESS
# in the node
SDIR=$(mktemp -d)
for SCRIPT in disk-parted sysupgrade remote-upgrade keep-files; do
    wget --no-check-certificate -qO "$SDIR/confine.$SCRIPT" "https://media.confine-project.eu/node/upgrade/confine.$SCRIPT"
    chmod +x "$SDIR/confine.$SCRIPT"
done
export PATH="$SDIR:$PATH"

Please note that https://media.confine-project.eu/ is also available in Community-Lab's management network as http://[fdf5:5351:1dfd:0:2000::d4]/. You may prefer this if your node has no access to the Internet.

When done downloading, please run the rest of commands below under this same shell session in the node.

How to preserve changes

If your node is using the overlay scheme, just log into it and touch the files you want to preserve (they will be added to the overlay). For instance, to explicitly preserve the contents of /etc/config:

find /etc/config -type f -exec touch {} \;

Automatic upgrade of an already customized node

This is the recommended way of upgrading a node.

For upgrading the system when the node is already customized, the latest generic node firmware image can be used. For facilitating the process the confine.remote-upgrade and confine.sysupgrade scripts have been developed. To upgrade a node execute the next commands:

# in your computer
ssh root@NODE_ADDRESS
# in the node
confine.remote-upgrade

confine.remote-upgrade uses a default set of URLs to download the image from. You can override them by setting a space-separated list of URLs in the confine.node.latest_image_uri UCI option, or you can override both defaults and the UCI option by setting the list of URLs in the IMG_URLS environment variable. For instance, to retrieve the image from a host in the testbed's management network, you may run this in the node:

export IMG_URLS="http://[MGMT_ADDR_OF_TESTBED_HOST]/path/to/generic-node-image.img.gz"
confine.remote-upgrade

Customized firmware generated by the Community-Lab Controller uses i686 images by default, built for Pentium Pro-compatible processors. If your node has a Pentium-compatible processor instead, you may want to permanently configure a i586 image before upgrading, like this:

uci set 'confine.node.latest_image_uri=http://[fdf5:5351:1dfd:0:2000::d4]/node/CONFINE-owrt-master-i586-current.img.gz'
uci commit confine
confine.remote-upgrade

Note on non-matching partitions: If you get an error saying that “partitions do not match”, it means that your node cannot be upgraded preserving the /overlay and /home partitions. In this case you may want to use the -n option of confine.remote-upgrade to upgrade anyway but this will destroy the current node configuration, deployed slivers and any data which is not in the ROM, so you will need to manually backup and restore them. Since this requires deep OpenWrt and CONFINE knowledge, we do NOT recommend this method, and instead of it you should use the one explained in the following section that uses an already customized image.

Upgrade a node (non-automatic)

First, download from the CONFINE image repository the generic node firmware image that you want to upgrade to. Then, upload the image to the /tmp directory of the node (e.g. using scp), log into it and execute the confine.sysupgrade tool provided by the CONFINE node system:

confine.sysupgrade /tmp/<generic_image.gz>

If you get an error saying that “partitions do not match” you may still force an upgrade with the -n option of confine.sysupgrade, but you will need to manually backup and restore all node configuration and data (see the note above on non-matching partitions). In this case we recommend that you download an already customized node firmware image by clicking on the Download firmware… button in the node's page in the testbed controller web interface (e.g. in Community-Lab's controller), making sure it is a plain non-USB image, and then upload it to the node's /tmp directory, log into it and run:

confine.sysupgrade -n /tmp/<custom_image.gz>

Note: Using wget or similar to download a custom image straight from the controller into the node is not recommended because you must enter your authentication tokens in the node (which may be unsafe) and it may not be able to establish a (secure) connection to the controller's web interface without resorting to the management network. If you still want to download the image to the node using this procedure instead of uploading it using scp or similar, you may try the following command while in the node's /tmp directory:

cd /tmp
wget --user <user> --ask-password --auth-no-challenge --no-check-certificate <custom_image_url>

or, if you prefer to use Curl:

cd /tmp
curl -k -O -u <user> <custom_image_url>

How to remove changes

If your node is using the overlay scheme, just log into it and remove the /overlay directory before upgrading:

rm -rf /overlay
confine.sysupgrade <custom_image.gz>

Another method is using the -n option.

confine.sysupgrade -n <custom_image.gz>

If you are removing the overlay scheme, after the routine installation and upgrade reboots finish, please remember to check that the node has the proper partitions. In not, you may create them by means of running:

confine.disk-parted

Important note: Do not remove changes if you use a generic firmware image, it will result in a deconfigured node that will need a clean install with a custom firmware again. Be careful if you remove changes remotely because the network can be deconfigured too and then you will lose the access.

Quick confine-opkg upgrade

This method is only documented for completeness. It is unreliable and insufficiently tested, please use one of the methods described above.

This procedure will temporarily suspend interaction with the CONFINE controller and update only the confine-system package (i.e. the CONFINE daemon) to a specific new version. Afterwards the daemon will continue based on the updated code. All node configuration and currently deployed and active slivers will remain. Most recent confine-system opkgs can be found in the CONFINE OpenWrt packages repository.

# in your computer
ssh root@NODE_ADDRESS
# in the node
confine_daemon_update 'http://repo.confine-project.eu/x86/packages/confine-system_r20140213.0946-1_x86.ipk'
soft/node-upgrade.txt · Last modified: 2016/10/28 16:19 by ivilata