A framework for scalable (mobile agent based) distributed mining of association rules over the Internet

Abstract

The aim of this thesis is to propose a framework for scalable (Mobile Agent Based) distributed mining of association rules. The scalability is the means to maintain an optimal computation-to-communication (CTC) ratio for high mining performance. The objectives, which have been achieved successfully, include the following: a) To evaluate the impact of asynchronous agent collaboration (through asynchronous message passing) on mining performance in terms of turnaround time. b) To study the relationship between asynchronism and the CTC ratio. c) To propose a Scalable Split & Aggregate Framework (S²AF) to maintain an optimal CTC ratio. d) To propose at least two algorithms to study the feasibility of the proposed S²AF under different conditions. e) To choose a stable platform for testing S²AF so that credible test results can be collected for analysis. Mobile agents are software entities or program objects that work independently to discharge their duties similar in fashion to human agents. Meanwhile these entities can move from node to node for various reasons. Since agents can replicate or terminate themselves, by nature a mobile agent program is scalable. This research covers different relevant areas of investigations, which require both backtracking and cross-referencing operations. To make the work more effective, a methodology to "investigate & experiment & proceed with possible backtracking, cross referencing and looping (IEP)" is devised and adopted. It is implemented as a research roadmap. All the experiments were carried out over the chosen stable Java-based Aglets mobile agent platform running over part of the PolyU Intranet in our laboratory. Most of the tests in the early investigations were based on the data generated by the IBM data generated package, which is widely used by other researchers in the area of data mining of association rules. In addition, tests were carried out with real multimedia data (mainly image) in later investigations the aim being to demonstrate that the proposed S²AF is indeed applicable to real-life problems. The analysis of these test results led to the proposal of another efficient algorithm, namely, the OWLBA (Optimized Weighted Load Balancing Algorithm). The PolyU Intranet is intentionally used to gain insight into scaling the S²AF for real-life Internet based applications in the area of scalable distributed data mining, especially when mobile agents are involved.