Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/75497
Title: Matrixmap : programming abstraction and implementation of matrix computation for big data analytics
Authors: Huangfu, Y 
Liang, G 
Cao, J 
Keywords: Big data
Parallel programming
Matrix computation
Machine learning
Graph processing
Issue Date: 2017
Publisher: American Institute of Mathematical Sciences (AIMS Press)
Source: Big data and information analytics, 2017, v. 1, no. 4, p. 349-376 How to cite?
Journal: Big data and information analytics 
Abstract: The computation core of many big data applications can be expressed as general matrix computations, including linear algebra operations and irregular matrix operations. However, existing parallel programming systems such as Spark do not have programming abstraction and efficient implementation for general matrix computations. In this paper, we present MatrixMap, a unified and efficient data-parallel programming framework for general matrix computations. MatrixMap provides powerful yet simple abstraction, consisting of a distributed in-memory data structure called bulk key matrix and a programming interface defined by matrix patterns. Users can easily load data into bulk key matrices and program algorithms into parallel matrix patterns. MatrixMap outperforms current state-of-the-art systems by employing three key techniques: matrix patterns with lambda functions for irregular and linear algebra matrix operations, asynchronous computation pipeline with context-aware data shuffling strategies for specific matrix patterns and in-memory data structure reusing data in iterations. Moreover, it can automatically handle the parallelization and distribute execution of programs on a large cluster. The experiment results show that MatrixMap is 12 times faster than Spark.
URI: http://hdl.handle.net/10397/75497
ISSN: 2380-6966
EISSN: 2380-6974
DOI: 10.3934/bdia.2016015
Appears in Collections:Journal/Magazine Article

Access
View full-text via PolyU eLinks SFX Query
Show full item record

Page view(s)

54
Last Week
1
Last month
Citations as of Nov 11, 2018

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.