Hein Meling

This paper presents the design and implementation of Jgroup/ARM, a distributed object group platform with autonomous replication management along with a novel measurement-based assessment technique that is used to validate the fault-handling capability of Jgroup/ARM. Jgroup extends Java RMI through the group communication paradigm and has been designed specifically for application support in partitionable systems. ARM aims at improving the dependability characteristics of systems through a fault-treatment mechanism. Hence, ARM focuses on deployment and operational aspects, where the gain in terms of improved dependability is likely to be the greatest. The main objective of ARM is to localize failures and to reconfigure the system according to application-specific dependability requirements. Combining Jgroup and ARM can significantly reduce the effort necessary for developing, deploying and managing dependable, partition-aware applications. Jgroup/ARM is evaluated experimentally to validate its fault-handling capability; the recovery performance of a system deployed in a wide area network is evaluated. In this experiment multiple nearly coincident reachability changes are injected to emulate network partitionsseparating the service replicas. The results show that Jgroup/ARM is able to recover applications to their initial state in several realistic failure scenarios, including multiple, concurrent network partitionings.

Keywords: fault tolerance, fault treatment, replication and recovery management, measurement-based assessment, middleware, remote method invocation, group communication

This paper presents an architecture enabling developers to easily and flexibly assign replication protocols simply by annotating individual server meth- ods. This avoids using costly replication protocols for all object methods, e.g. read-only methods can use less costly protocols, reserving the costly replica- tion protocols for update methods. The architecture has been implemented in the Jgroup/ARM middleware, and enables addition of new replication protocols with- out modifying the core toolkit. It also supports runtime selection of replication protocol for individual methods. This can be used to support self-optimization of protocol selection by optimizing for the most appropriate configuration under a given system load.

This paper presents the design and implementation of the Distributed Autonomous Replication Management (DARM) framework built on top of the Spread group communication system. The objective of DARM is to improve the dependability characteristics of systems through a fault treatment mechanism. Unlike many existing fault tolerance frameworks, DARM focuses on deployment and operational aspects, where the gain in terms of improved dependability is likely to be the greatest.

DARM is novel in that recovery decisions are distributed to each individual group deployed in the system, eliminating the need for a centralized manager with global information about all groups. This scheme allows groups to perform fault treatment on themselves. A group leader in each group is responsible for fault treatment by means of replacing failed group members; the approach also tolerates failure of the group leader. The advantages of the distributed approach is: (i) no need to maintain globally centralized information about all groups which is costly and limits scalability, (ii) reduced infrastructure complexity, and (iii) less communication overhead. We evaluate the approach experimentally to validate its fault handling capability; the recovery performance of a system deployed in a local area network is evaluated. The results show that applications can recover to their initial system configuration in a very short period of time.

Jgroup/ARM is a middleware for developing and operat- ing dependable distributed Java applications. Jgroup integrates the dis- tributed ob ject model of Java RMI with the object group paradigm, en- abling construction of replicated servers that offer dependable services to clients. ARM aims to improve the dependability characteristics of systems through fault treatment, focusing on operational aspects where the gain in terms of improved dependability is likely to be the greatest. ARM offers two core mechanisms: recovery from node, object and net- work failures and distribution of replicas. ARM identifies failures and reconfigures the system according to its dependability requirements. This paper proposes an enhancement of the ARM framework in which replica placement is performed in a distributed manner, eliminating the need for a centralized manager with global information about all ob ject groups. Instead each autonomous ob ject group handles their own replica placement based on information from nodes. Assuming that multiple ob- jects groups are deployed in the system, this constitutes a distributed replica placement scheme. This scheme enables the implementation of self-healing ob ject groups that can perform fault treatment on them- selves. Advantages of the approach: (a) no need to maintain global infor- mation about all ob ject groups which is costly and limits scalability, (b) reduced infrastructure complexity, and (c) less communication overhead.

The emergence of short-range wireless communications hold the promise of realizing the grand vision of the next generation communication networks in which devices follow a always best-connected pattern for anybody, to anything from anywhere at anytime. Short-range wireless communications offers easy access to the global information and communications infrastructure, facilitates seamless connectivity amongst a host of computing, communications and sensing devices that collaborate to form a supporting ambient and pervasive computing environment.

In this paper, we describe the IS-Home project where a we propose a novel autonomic communications middleware platform enabling a wide variety of integrated services to be installed, including health monitoring services. The motivation for the project is to take advantage of the opportunities that open and standardized protocols and technologies can represent in the future home environment.

Performing experimental evaluation of fault tolerant distributed systems is a complex and tedious task, and automating as much as possible of the execution and evaluation of experiments is often necessary to test a broad spectrum of possible executions of the system to obtain good coverage. The confidence of the results obtained from an experimental evaluation depends on the degree of control over the environment in which experiments are being executed. Typically, an uncontrolled environment is exposed to numerous sources of external influence that can affect the obtained results. Automated and repeated executions can be used to reduce the impact of such influences.

In this paper, a framework for experimental validation and performance evaluation of fault management in a fault tolerant distributed system is presented. The framework provides a facility to execute experiments in a configured target system. It is based on injecting faults or other events needed to test the fault handling capability of the system. Relevant events are logged and collected for post-processing and analysis, e.g. to construct a single global timeline of events occurring at different nodes in the target system. This timeline of events can then be used to validate the behavior a system, and to evaluate its performance.

Protocol composition is a common approach to structure generic protocols used by networked applications, and typically a vertically layered approach is taken. This paper presents an alternative approach, where the protocol composition is a weakly-coupled set of protocol modules organized in a non-hierarchical structure. Protocol modules are dynamically constructed at runtime. The approach is designed for systems that involves multiple communicating entities and multicast style interactions are supported, making the approach suitable for building reliable network applications. The main advantage of the approach is that modules in the same composition communicate by direct interaction, whereas other frameworks typically use a vertically layered protocol stack, forcing all messages/events to pass through all intermediate layers introducing unnecessary delays.

Networked computer systems are prevalent in most aspects of modern society, and we have become dependent on such computer systems to perform many critical tasks. Moreover, making such systems dependable is an important goal. However, dependability issues are often neglected when developing systems due to the complexities of the techniques involved.

A common technique used to improve the dependability characteristics of systems is to replicate critical system components whereby the functions they perform are repeated by multiple replicas. Replicas are often distributed geographically and connected through a network as a means to render the failure of one replica independent of the others. However, the network is also a potential source of failures, as nodes can become temporarily disconnected from each other, introducing an array of new problems.

The majority of previous projects have focused on the provision of middleware libraries aimed at simplifying the development of dependable distributed systems, whereas the pivotal deployment and operational aspects of such systems have received very little attention. This thesis extends on previous works and emphasize the deployment and operational aspects, where the gain in terms of improved dependability is likely to be the greatest.

The main contribution of this dissertation is an architecture for autonomous replication management, aimed to improve the dependability characteristics of systems through a self-managed fault treatment mechanism that is adaptive to network dynamics and changing requirements. Consequently, the architecture also improves the deployment and operational aspect of systems, and reduces the human interactions needed. The architecture has been implemented as a proof of concept prototype by extending the Jgroup object group system.

In addition, numerous supporting contributions are also included in this work: (i) an architecture for dynamic protocol composition that avoids the delays of event processing in intermediate layers of a strictly vertical protocol stack; (ii) adaptive protocol selection is also made possible on a per method/invocation basis, by annotating server methods with the replication protocol to be used; (iii) client-side membership handling is also implemented aimed to improve the load balancing and failover properties of systems when exposed to failures; (iv) online upgrade management of operational services is also implemented as an extension to the replication management architecture.

Finally, the dissertation provides extensive experimental evaluation of the fault treatment capabilities of the autonomous replication management architecture, with emphasis on testing complex failure scenarios. The first experiment examines the ability of clients to maintain correct membership when servers crash and recover. The second experiment investigates the behavior of services when exposed to multiple nearly-coincident node crash failures. In conjunction with this experiment, a novel technique has been developed to estimate various service dependability characteristics. In the third experiment the recovery performance of a system deployed in a wide area network is evaluated. In this experiment multiple nearly-coincident reachability changes are injected to simulate network partitions separating the service replicas.

To support the experimental evaluation, a set of generic tools have also been developed to aid the execution and analysis of the experiments.

Jgroup/ARM is a middleware framework for operating dependable distributed applications based on Java. Jgroup integrates the distributed ob ject models of Java RMI and Jini with the object group communication paradigm, enabling the construction of groups of replicated server ob jects that provide dependable services to clients. ARM provides automated mechanisms for distributing replicas to host processors and recovering from replica failures. This paper describes an approach based on stratified sampling combined with fault injections for estimating the dependability attributes of a service deployed using the Jgroup/ARM middleware framework. A first experimental evaluation is performed focusing on a service provided by a triplicated server, and indicative predictions of various dependability attributes of the service are obtained. The evaluation shows that a very high availability and MTBF may be achieved for services based on Jgroup/ARM.

In a distributed fault-tolerant server system realized according to the open group model, inconsistency will (temporarily) arise between the dynamic membership of the replicated service and its client-side representation in the event of server failures and recoveries. The paper proposes techniques for maintaining this consistency and discuss their performance implications in failure/recovery scenarios where clients load balance requests on the servers. Comparative performance measurements is carried out for two of the proposed techniques. The results indicate that the performance impact of lacking consistency is easily kept small, and that the cost of the technique is small.

In this paper, we discuss issues related to maintaining consistency (and freshness) between the dynamic membership of a replicated server, and its representation in a naming service storage. We propose two solutions based on leasing and notification, and evaluate the suitability of each solution based on measurements of performance impact and failover delay. Furthermore, we discuss several approaches to ensure that the dynamic membership of the replicated server is also reflected (refreshed) in the client-side representation of the server group membership.

Change management is indispensable in most distributed software systems, which are continuously being modified throughout their life cycle. Managing the changes at runtime in highly available distributed systems is especially challenging as upgrade of a running system should not deteriorate its availability characteristics. We present a distributed algorithm that allows to dynamically upgrade an actively replicated server so that the server is operational, even during the upgrade process. The algorithm makes use of the core functionality of an underlying Group Communication System that has been extended with a recovery mechanism. Its design enables dependable upgrades of replicated software in the presence of replica crashes. The presented mechanisms are part of the Dynamic Upgrade Management Framework aiming at supporting and managing dependable upgrades of distributed systems on the fly.

Peer-to-peer (P2P) systems are characterized by decentral- ized control, large-scale and extreme dynamism of their environment. Developing applications that can cope with these characteristics requires a paradigm shift that puts adaptation, resilience and self-organization as primary concerns. Complex adaptive systems (CAS), commonly used to explain the behavior of many biological and social systems, could be an appropriate response to these requirements. In order to pursue these ideas, this paper presents Messor, a decentralized load-balancing algo- rithm based on ideas such as multi-agent systems drawn from CAS. A novel P2P grid computing application has been designed using the Mes- sor algorithm, allowing arbitrary users to initiate computational tasks.

Recent peer-to-peer (P2P) systems are characterized by decentralized control, large scale and extreme dynamism of their operating environment. As such, they can be seen as instances of complex adaptive systems (CAS) typically found in biological and social sciences. In this paper we describe Anthill, a framework to support the design, implementation and evaluation of P2P applications based on ideas such as multi-agent and evolutionary programming borrowed from CAS. An Anthill system consists of a dynamic network of peer nodes; societies of adaptive agents travel through this network, interacting with nodes and cooperating with other agents in order to solve complex problems. Anthill can be used to construct different classes of P2P services that exhibit resilience, adaptation and self-organization properties. We also describe preliminary experiences with Anthill in implementing a file sharing application.

Peer-to-peer (P2P) systems are characterized by decentral- ized control, large scale and extreme dynamism of their operating en- vironment. Developing applications that can cope with these character- istics requires a paradigm shift, placing adaptation, resilience and self- organization as primary concerns. In this note, we argue that complex adaptive systems (CAS), which have been used to explain certain biologi- cal, social and economical phenomena, can be the basis of a programming paradigm for P2P applications. In order to pursue this idea, we are de- veloping Anthill, a framework to support the design, implementation and evaluation of P2P applications based on ideas such as multi-agent and evolutionary programming borrowed from CAS.

The peer-to-peer (P2P) paradigm for building distributed applications has recently gained renewed attention, partly due to the enormous success of systems like Napster and Gnutella. Subsequently, a multitude of projects focusing on anonymity, security, routing and reliability aspects of P2P have been initiated. A framework that supports the design, evaluation and implementation phases of P2P application development is currently missing. The Anthill project is an attempt to fill this void. In this paper we give a brief overview of Anthill and describe Gnutant, a document sharing application that combine the scalability property of Freenet with the free search capabilities of Gnutella. In addition, we present preliminary simulation results obtained by running the Gnutant application in the Anthill simulation environment. The simulation results indicate that after an initialization period, Gnutant is effective in finding documents.

We present the design and implementation of a replication management framework for partition-aware applications based on Jgroup. Jgroup offers an extension to Java RMI based on the group communication paradigm, enabling development of dependable applications in partitionable distributed systems. The replication management framework simplifies the development of fault tolerant applications by providing exchangeable replica distribution schemes and application specific recovery strategies. The framework is extensible to several replication and recovery strategies. The only user interaction required is the creation and removal of object groups, i.e., enabling autonomous replication management.

A plug-and-play (PaP) architecture to be applied for specification and execution of telecommunication systems functionality is presented. The architecture is based on a theatre metaphor. Plays define the functionality of the system. PaP components are realised by actors playing roles defined by manuscripts. An actor's capabilities define his possibilities for playing various roles. The usability of the architecture is validated through specification, implementation and testing of a PaP support system and a tele-school application demonstrator. A PaP support system that meets the flexibility and adaptability requirements and parts of the tele-school application has been implemented and validated. The implementation is based on Java RMI.

This paper presents an architecture specified within the project “Plug-and-play for Network and Teleservice Components” supported by The Norwegian Research Council. The hardware and software parts, as well as complete network ele- ments that constitute a communication system, shall have the ability to configure themselves when installed into a network and then to provide services according to their own capabilities, the service repertoire and the operating policies of the system. Plug-and-play components and functional objects are defined. A theatre anal- ogy is chosen for the specification of the needed PaP support functionality. Plays define the functionality of the system. PaP components are realised by actors playing roles defined by manuscripts. An actor's capabilities define his possibil- ities for playing various roles.

An architecture for plug-and-play to be applied for telecommunication systems is pre- sented. The architecture is based on a theatre metaphor. Plays define the functionality of the system. PaP components are realised by actors playing roles defined by manuscripts. An ac- tor's capabilities define his possibilities for playing various roles. The usability of the architec- ture will be validated through specification, implementation and testing of a demonstrator. The PaP system implementation design and the functionality of a tele-school based demonstrator is presented.

Public key cryptography allows a user to digitally sign messages with a private signature key so that recipients in possession of the corresponding public verification key can check the correctness of the signature. The main problem with this method is to authenticate public keys in order to know that the digital signature is authentic. A Public Key Infrastructure refers to a network where the authenticity of public keys is certified by Certification Authorities in a hierarchy, and it is believed that this will be an important element in the development of electronic commerce. Certification services are already available from commercial Certifica- tion Authorities, and we are starting to see the emergence of national and international public key infrastructures. This paper describes public key infrastructures in general and discusses Norwegian as well as some international implementation efforts.