User Tools

Site Tools


d27-core

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
d27-core [2015/08/14 14:06]
ivilata Limit issues to appearing ones, sort numerically.
d27-core [2015/08/17 11:22] (current)
ivilata Minor changes after review.
Line 7: Line 7:
 ===== Introduction ===== ===== Introduction =====
  
-The [[soft:​node|CONFINE Node software]] and [[soft:​server|CONFINE Controller software]] have received a great amount of deep changes during the last year.  Besides the usual fixes to several problems ​arised ​during the usage of [[https://​community-lab.net/​|Community-Lab]] and other CONFINE-based testbeds, the main changes have been oriented towards the //​stabilization// ​on the CONFINE architecture,​ the Node and Controller implementations themselves, and the //​maintainability//​ and ease of adoption of the software by future users. ​ To this point, new [[soft:​utilities|utilities]] and [[testing:​start|tests]] have been developed to complement the usage and development of the software. ​ As usual, all the aforementioned software is available with a full history of changes and issues in the [[https://​redmine.confine-project.eu/​|CONFINE Redmine server]].+The [[soft:​node|CONFINE Node software]] and [[soft:​server|CONFINE Controller software]] have received a great amount of deep changes during the last year.  Besides the usual fixes to several problems ​arisen ​during the usage of [[https://​community-lab.net/​|Community-Lab]] and other CONFINE-based testbeds, the main changes have been oriented towards the //​stabilization// ​of the CONFINE architecture, ​and the Node and Controller implementations themselves, and the //​maintainability//​ and ease of adoption of the software by future users. ​ To this point, new [[soft:​utilities|utilities]] and [[testing:​start|tests]] have been developed to complement the usage and development of the software. ​ As usual, all the aforementioned software is available with a full history of changes and issues in the [[https://​redmine.confine-project.eu/​|CONFINE Redmine server]].
  
-From July 15th 2014 to the present day (August 2015), around 150 new issues have been created in Redmine'​s issue tracker, with less than 50 remaining open, some of them being long-term feature requests, and less than 10 of them having a high priority. ​ This is an indication of the high effort put into the stabilization of the testbed and its reference implementation provided by the CONFINE Project. ​ Issues related with the topics discussed in this chapter will be indicated as ''​#NUMBER'' ​and linked ​at its end.+From July 15th 2014 to the present day (August 2015), around 150 new issues have been created in Redmine'​s issue tracker, with less than 50 remaining open, some of them being long-term feature requests, and less than 10 of them having a high priority. ​ This is an indication of the high effort put into the stabilization of the testbed and its reference implementation provided by the CONFINE Project. ​ Issues related with the topics discussed in this chapter will be indicated as #NUMBER and linked ​in [[#Issues]].
  
 The resulting CONFINE testbed architecture,​ which is implemented by the new stable releases of the Node software (Master ''​20150715-1002''​),​ Controller software (''​1.0.1''​) and [[soft:​vct-container|VCT container]] (''​201507291401''​),​ comprises the [[milestones:​confined|"​Confined"​ milestone]],​ planned for the stability and maintainability objectives mentioned above. ​ The node architecture offered by this release can be seen in <imgref node-architecture>,​ and the associated object/data model in <imgref data-model>​. ​ This model is reflected by versions ''​v1''​ of the Registry and Node [[arch:​rest-api|REST APIs]], which along the Controller API constitute the programmable interface to CONFINE testbeds. The resulting CONFINE testbed architecture,​ which is implemented by the new stable releases of the Node software (Master ''​20150715-1002''​),​ Controller software (''​1.0.1''​) and [[soft:​vct-container|VCT container]] (''​201507291401''​),​ comprises the [[milestones:​confined|"​Confined"​ milestone]],​ planned for the stability and maintainability objectives mentioned above. ​ The node architecture offered by this release can be seen in <imgref node-architecture>,​ and the associated object/data model in <imgref data-model>​. ​ This model is reflected by versions ''​v1''​ of the Registry and Node [[arch:​rest-api|REST APIs]], which along the Controller API constitute the programmable interface to CONFINE testbeds.
Line 19: Line 19:
 ===== Architectural updates ===== ===== Architectural updates =====
  
-The main changes to the CONFINE architecture from the previous [[milestones:​bare-bones|"​Bare ​Bones" milestone]] are:+The main changes to the CONFINE architecture from the previous [[milestones:​bare-bones|"​Bare ​bones" milestone]] are:
  
   * Support for per-testbed and per-node **resources and their allocation** for slices and slivers, respectively (#46, #​602). ​ Maximum and default requests can be set for each resource in the testbed or node (by superusers and node administrators,​ respectively). ​ Explicit requests of disk space and memory are allowed for slivers, as well as implicit requests of public IPv4 and IPv6 addresses via sliver interfaces. ​ Implicit requests of VLAN tags are allowed for slices via the activation of isolated interfaces.   * Support for per-testbed and per-node **resources and their allocation** for slices and slivers, respectively (#46, #​602). ​ Maximum and default requests can be set for each resource in the testbed or node (by superusers and node administrators,​ respectively). ​ Explicit requests of disk space and memory are allowed for slivers, as well as implicit requests of public IPv4 and IPv6 addresses via sliver interfaces. ​ Implicit requests of VLAN tags are allowed for slices via the activation of isolated interfaces.
Line 25: Line 25:
   * Unification of testbed server and gateways so that **multiple servers** can be defined (#​236). ​ The main difference of servers with nodes and hosts is that the former are maintained by testbed operators instead of normal users, and they can publish several API endpoints (like Registry and Controller).   * Unification of testbed server and gateways so that **multiple servers** can be defined (#​236). ​ The main difference of servers with nodes and hosts is that the former are maintained by testbed operators instead of normal users, and they can publish several API endpoints (like Registry and Controller).
   * **Configurable API endpoints** for Node, Registry and Controller APIs in testbed nodes and servers (#245). Instead of the previous implicit API base URIs, this sticks to REST's «Hypermedia as the Engine of Application State» (HATEOAS), and allows for flexible delegation on per-island caching proxies, front ends for groups of nodes, etc.  Since the certificate is now also indicated for each API endpoint, there is no longer need for a centralized Certificate Authority run by testbed operators.   * **Configurable API endpoints** for Node, Registry and Controller APIs in testbed nodes and servers (#245). Instead of the previous implicit API base URIs, this sticks to REST's «Hypermedia as the Engine of Application State» (HATEOAS), and allows for flexible delegation on per-island caching proxies, front ends for groups of nodes, etc.  Since the certificate is now also indicated for each API endpoint, there is no longer need for a centralized Certificate Authority run by testbed operators.
-  * Definition of **sliver defaults** in the slice which can be overridden by slivers and include resource requests, sliver template and data, and set state (#​234). ​ Along with the slice'​s own set state, this allows sophisticated ​cominations ​like all-slivers-running-but-some or all-slivers-stopped-but-some. +  * Definition of **sliver defaults** in the slice which can be overridden by slivers and include resource requests, sliver template and data, and set state (#​234). ​ Along with the slice'​s own set state, this allows sophisticated ​combinations ​like all-slivers-running-but-some or all-slivers-stopped-but-some. 
-  * Make **network backend configuration** (tinc or native) independent from management network configuration in testbed hosts, nodes and servers (#​157). ​ This allows the backend ​to be reused by other features like a hypothetical VPN to access the Internet from certain community networks.+  * Make **network backend configuration** (tinc or native) independent from management network configuration in testbed hosts, nodes and servers (#​157). ​ This allows the back end to be reused by other features like a hypothetical VPN to access the Internet from certain community networks.
   * Unification of tinc server and client into **tinc host** to resemble the underlying mesh model (#​157). ​ Any such host (i.e. a testbed host, node or server) can act as a gateway to the testbed'​s management network if trusted by the host connecting to it.  This does away with the need of dedicated gateways and enables more resilient overlay setups in islands without intervention from testbed superusers.   * Unification of tinc server and client into **tinc host** to resemble the underlying mesh model (#​157). ​ Any such host (i.e. a testbed host, node or server) can act as a gateway to the testbed'​s management network if trusted by the host connecting to it.  This does away with the need of dedicated gateways and enables more resilient overlay setups in islands without intervention from testbed superusers.
   * Relate hosts and nodes directly to islands (#264), as well as server API endpoints and tinc addresses (#236, #245, #​157). ​ This eases the client API code needed to locate hosts that offer a specific API or service in an island.   * Relate hosts and nodes directly to islands (#264), as well as server API endpoints and tinc addresses (#236, #245, #​157). ​ This eases the client API code needed to locate hosts that offer a specific API or service in an island.
Line 55: Line 55:
 A big development and testing effort has been put in upgrading the Node software to **OpenWrt 14.07 "​Barrier Breaker"​**,​ the current stable OpenWrt release, and supporting it in the Controller. ​ Adopting this release increases the chances of adoption of CONFINE software by OpenWrt users and developers, as well as improving hardware support (among others). A big development and testing effort has been put in upgrading the Node software to **OpenWrt 14.07 "​Barrier Breaker"​**,​ the current stable OpenWrt release, and supporting it in the Controller. ​ Adopting this release increases the chances of adoption of CONFINE software by OpenWrt users and developers, as well as improving hardware support (among others).
  
-With the new stable releases of Node and Controller software, a new [[soft:​vct-container|VCT container]] (version ''​201507291401''​) has been packaged that includes them both, along with other CONFINE tools, in a Debian Jessie base system that can be run under [[https://​linuxcontainers.org/​|LXC]] to get a complete virtual CONFINE testbed with nodes and a Controller, for learning and testing. ​ Initial work has also been done to create a [[soft:​vct-docker|VCT Docker]] ​container.  We expect to leverage the software used to create these containers in order to prepare plain Controller containers in the future.+With the new stable releases of Node and Controller software, a new [[soft:​vct-container|VCT container]] (version ''​201507291401''​) has been packaged that includes them both, along with other CONFINE tools, in a Debian Jessie base system that can be run under [[https://​linuxcontainers.org/​|LXC]] to get a complete virtual CONFINE testbed with nodes and a Controller, for learning and testing. ​ Initial work has also been done to create a [[soft:​vct-docker|VCT Docker ​container]].  We expect to leverage the software used to create these containers in order to prepare plain Controller containers in the future.
  
 Node administrators now have a greater control over who can access their nodes as ''​root''​. ​ They have the possibility of setting a closed set of authorized keys, having the node poll the registry for new administrators in the group, or even accepting additional keys for centralized remote maintenance. ​ Also, for an easier administration of the community devices where research devices (i.e. nodes) use to be attached, some implementation limitations on the arrangement of sliver interfaces have been coordinated between Node and Controller software (#633). Node administrators now have a greater control over who can access their nodes as ''​root''​. ​ They have the possibility of setting a closed set of authorized keys, having the node poll the registry for new administrators in the group, or even accepting additional keys for centralized remote maintenance. ​ Also, for an easier administration of the community devices where research devices (i.e. nodes) use to be attached, some implementation limitations on the arrangement of sliver interfaces have been coordinated between Node and Controller software (#633).
  
-Finally, several new scripts have been added to the utilities repository. ​ The REST API client library included there has also been updated to the stable API specification. ​ Some integration code for creating [[http://​www.linux-kvm.org/​page/​Main_Page|KVM]] nodes under the [[http://​cloudy.community/​|Cloudy distribution]] can also be found there, along with the software used to package the LXC-based VCT container as mentioned above.+Finally, several new scripts have been added to the utilities repository. ​ The REST API client library included there has also been updated to the stable API specification. ​ Some integration code for creating [[http://​www.linux-kvm.org/​|KVM]] nodes under the [[http://​cloudy.community/​|Cloudy distribution]] can also be found there, along with the software used to package the LXC-based VCT container as mentioned above.
  
 ==== Node ==== ==== Node ====
Line 72: Line 72:
  
   * The node's local and direct **interfaces can be VLAN-tagged**. ​ This allows hybrid nodes using DLEP to communicate with external network interfaces. ​ On slivers with isolated interfaces, this implies VLAN stacking (802.1ad or QinQ, which requires Barrier Breaker).   * The node's local and direct **interfaces can be VLAN-tagged**. ​ This allows hybrid nodes using DLEP to communicate with external network interfaces. ​ On slivers with isolated interfaces, this implies VLAN stacking (802.1ad or QinQ, which requires Barrier Breaker).
-  * Node administrators can permanently customize the location of the firmware used in remote upgrades (#653, #654), e.g. for choosing a customized image for some nodes, or a closer image repository.+  * Node administrators can permanently ​**customize the location of the firmware** used in remote upgrades (#653, #654), e.g. for choosing a customized image for some nodes, or a closer image repository.
   * The **on-demand CPU governor** has been enabled by default to save power on idle nodes (#59).   * The **on-demand CPU governor** has been enabled by default to save power on idle nodes (#59).
   * Community-Lab specific customizations have been moved to a separate package (#655).   * Community-Lab specific customizations have been moved to a separate package (#655).
Line 89: Line 89:
  
   * Some scalability issues with the growing size of the database have been fixed (#475, #448).   * Some scalability issues with the growing size of the database have been fixed (#475, #448).
-  * Compatibility with the latest Debian Jessie (#685) and its Apache 2.4 packages (#684) has been added.+  * Compatibility with the latest Debian Jessie (#685) and its Apache ​''​2.4'' ​packages (#684) has been added.
   * The standard Controller installation now includes an NTP server which can be used by nodes over the management network to keep their clocks synchronized (#404).   * The standard Controller installation now includes an NTP server which can be used by nodes over the management network to keep their clocks synchronized (#404).
  
Line 103: Line 103:
   * An interesting use of notifications is part of the new **sliver'​s journal application**,​ which reports a history of sliver usage for the nodes of a group. ​ This may improve the awareness of community network members about the utility of their nodes to researchers.   * An interesting use of notifications is part of the new **sliver'​s journal application**,​ which reports a history of sliver usage for the nodes of a group. ​ This may improve the awareness of community network members about the utility of their nodes to researchers.
   * A new **customizable dashboard** with information on the user's hosts has been added (#47).   * A new **customizable dashboard** with information on the user's hosts has been added (#47).
-  * **Better ​online ​documentation** is provided on firmware installation instructions (#674) and unavailable sliver interfaces (#216).+  * **Better ​on-line ​documentation** is provided on firmware installation instructions (#674) and unavailable sliver interfaces (#216).
   * Better access to the configuration of tinc addresses (#603).   * Better access to the configuration of tinc addresses (#603).
   * A new [[admin:​start|CONFINE administrator'​s guide]] has been written that mostly covers Controller usage for testbed operators, and which complements the already existing [[usage:​start|CONFINE user's guide]].   * A new [[admin:​start|CONFINE administrator'​s guide]] has been written that mostly covers Controller usage for testbed operators, and which complements the already existing [[usage:​start|CONFINE user's guide]].
Line 109: Line 109:
 Ongoing updates: Ongoing updates:
  
-  * The firmware generator is being rewritten to use **OpenWrt'​s Image Generator** (#​645). ​ This is the new standard way of customizing precompiled OpenWrt images, including the ability ​of adding ​or removing ​whole packages. +  * The firmware generator is being rewritten to use **OpenWrt'​s Image Generator** (#​645). ​ This is the new standard way of customizing precompiled OpenWrt images, including the ability ​to add or remove ​whole packages. 
-  * Work on porting the Controller to **newer versions of Django** is in progress (#​582). ​ Mainstream support for Django 1.6 ended in April 2015.+  * Work on porting the Controller to **newer versions of Django** is in progress (#​582). ​ Mainstream support for Django ​''​1.6'' ​ended in April 2015.
   * Preliminary code using [[http://​docs.seleniumhq.org/​|Selenium]] for testing the web interface has been written.   * Preliminary code using [[http://​docs.seleniumhq.org/​|Selenium]] for testing the web interface has been written.
   * A new [[devel:​controller|Controller developer'​s guide]] has been started (#677) to help with its release and testing.   * A new [[devel:​controller|Controller developer'​s guide]] has been started (#677) to help with its release and testing.
d27-core.txt · Last modified: 2015/08/17 11:22 by ivilata