Distributed deep learning method and device, parameter server and main working node

Abstract

The invention belongs to the technical field of computers, and provides a distributed deep learning method and device, a parameter server and a main working node. The method comprises the following steps: receiving a gradient vector set sent by a main working node in a plurality of data operation groups, wherein the gradient vector set sent by each main working node comprises gradient vectors of all working nodes in a data operation group where the main working node is located; updating global model parameters of a preset deep learning model according to the plurality of gradient vector sets; and issuing the updated global model parameters to each main working node, so that each main working node controls all working nodes in the data operation group where the main working node is located to carry out local model training according to the updated global model parameters. According to the distributed deep learning method disclosed by the embodiment of the invention, data and task scheduling is carried out by taking the data operation group as granularity, so that the amount of data synchronized with the parameter server in each iteration is reduced, the communication overhead is reduced, and the resource utilization rate of each working node is improved. 本申请属于计算机技术领域，提供了一种分布式深度学习方法、装置、参数服务器及主工作节点。方法包括接收多个数据运算组中主工作节点发送的梯度向量集；其中，每个主工作节点发送的梯度向量集包括该主工作节点所在数据运算组中所有工作节点的梯度向量；根据多个梯度向量集对预设深度学习模型的全局模型参数进行更新；将更新后的全局模型参数下发至各主工作节点，以使每个主工作节点控制其所在数据运算组中所有工作节点根据更新后的全局模型参数进行本地的模型训练。本申请实施例的分布式深度学习方法以数据运算组为粒度进行数据以及任务的调度，减少了每次迭代中与参数服务器同步的数据量，降低通信开销且提高了各工作节点的资源利用率。