Monitoring across operator boundaries is tricky, since it requires one operator to gather information from resources or services that maybe abstracted by the other operator’s management and orchestration system. The solution we propose is an intelligent monitoring subsystem in the multi-domain orchestration system that can account for this abstraction.
A Multi-provider Multi-domain Orchestrator (MdO) coordinates resource and/or service orchestration at multi-domain level, where multi-domain may refer to multi-technology (orchestrating resources and/or services using multiple Domain Orchestrators) or multi-operator (orchestrating resources and/or services using Domain Orchestrators belonging to multiple administrative domains).
The implementation architecture of the MdO as presented in D3.1  is shown in Figure 1. The architecture is derived as a more focused version of the detailed architecture presented in the project deliverable D2.1 . While working on the architecture presented in  we realized that the architecture presented in WP2 was missing the monitoring subsystem that would coordinate the monitoring across the different MdO. The Intelligent Monitoring Subsystem (IMoS) was therefore added in the implementation architecture.
Each MdO as shown in Figure 1 consists of a Topology Abstraction and Discovery Subsystem (TADS), see Figure 1. The function of this subsystem is to abstract the internal topology of a single domain to be exposed to other domains to enable the other domains to make service and resource slice placement decisions over a global understanding of the topology. Once the service is deployed it needs to be monitored across the various domains. This is done by the Intelligent Monitoring Subsystem (IMoS) in Figure 1. This information capsule presents the high level concept of the IMoS.
The Concept of the IMoS
The IMoS is responsible for collecting monitoring information of every deployed service across the different domains with regards to the KPIs expressed by : i) the SLA of the respective service ii) the KPI requirements of the local administrator of the MdO. The IMoS must be able to combine these requirements and efficiently collect KPI relevant information from the operators’ domains where the service is deployed.
Figure 2.: Functioning the IMoS
To do so it must evaluate the most efficient deployment of “probes” to reduce the probe effect on the overall system. A probe in this context is not a real entity but a concept that refers to the need of collection of monitoring data, typically, related to a KPI. The need for this concept is due to the fact that an IMoS can only request collection of monitoring data from another operator’s MdO over the topology that is exposed to it by the TADS, see Figure 2.
Figure 3.: Concept abstract probes and the IMoS recursive design
The relevant monitoring data over the real topology must be collected by the other MdO itself and reported back to the requesting IMoS. This translation of “real” collected monitoring data to abstracted topology monitoring data is also the responsibility of IMoS in MdO 2 in our architecture in Figure 2. This whole procedure makes the IMoS recursively stackable as shown in Figure 3. The last level of this recursion is named the local monitoring system or LMS which collects domain specific information to report it up.
Design of the IMoS
The Intelligent Monitoring Subsystem receives the detailed service request (via the interface with the SLA Management Subsystem) and references to resource allocation and VNF deployment (via the interface with the RO in the NFVO) over the (possibly abstracted) physical topology view that the MdO has. The Intelligent Monitoring Subsystem comprises two main components called Intelligent Resource Monitoring and Monitoring Database (DB) as well as an interface I3 toward the Local Monitoring Subsystems deployed in each technological domain and interface I2 towards the other operator IMoS.
The Intelligent Resource Monitoring is responsible for the following functionalities:
- To select the appropriate probes for each service request according to the service specification minimizing the overhead due to probe effect.
- To create a DB entry in the Monitoring DB for each service request.
- To interact with the Local Monitoring Subsystem(s) and other MdO’s IMoS via the I3 and I2 interface, respectively, to activate the probes on the desired entities for each service request.
- To interact with the SLA Management Subsystem in order to support services lifecycle management.
- To receive probes deployment request from other operator IMoS on the abstracted topology and to further translate them to probe deployment requests over what it sees as the real topology and accordingly to report back an appropriate abstracted monitoring data. This enables the IMoS to be stacked indefinitely as shown in Figure 3.
The Monitoring DB is a persistent storage element where all the monitoring measurements coming from different Local Monitoring Subsystems (in a given administrative domain) as well as the other IMoS are stored. The Monitoring DB is then used by the SLA Manager to check for conformation of the deployed service to the SLA.
Further details on the design of MdO are presented in the recently published project deliverable D3.1 . Some earlier concepts of the IMoS are presented in  and the generic concept of stack-ability of the entire MdO was earlier presented in .
 The 5GEx Prject deliverable D3.1: “Description of protocol and component design”. Available at https://www.5gex.eu/wp/?page_id=55
 Perez-Caparros, David, et al. “An architecture for creating and managing virtual networks.” Personal Indoor and Mobile Radio Communications (PIMRC), 2013 IEEE 24th International Symposium on. IEEE, 2013.
 Vaishnavi, Ishan, Riccardo Guerzoni, and Sergio Beker. “An architecture for co-ordinated monitoring for multi-provider cloud platforms.” Globecom Workshops (GC Wkshps), 2014. IEEE, 2014.
 The 5GEx Project deliverable D2.1: “Initial System Requirements and Architecture” Available at https://www.5gex.eu/wp/?page_id=55