Loading...
Publication
Processamento paralelo e distribuído baseado em workflows
| Name: | Description: | Size: | Format: | |
|---|---|---|---|---|
| 4.14 MB | Adobe PDF |
Authors
Advisor(s)
Abstract(s)
As infraestruturas de computação distribuídas de larga escala suportadas em virtualização e plataformas de computação na nuvem (Cloud) permitem a execução de aplicações de processamento de dados modeladas por decomposição em múltiplas tarefas que interagem entre si, segundo modelos representados por grafos, usando um paradigma designado por Scientific Workflows. Na última década, surgiram imensas propostas de sistemas de workflow. No entanto, apesar da maior ou menor popularidade de alguns sistemas, continuam a persistir algumas questões em aberto, passíveis de serem melhoradas. O trabalho descrito neste documento é uma contribuição para encontrar respostas para algumas questões em aberto, propondo um modelo de workflow com as seguintes características: i) descentralização do controlo de execução das múltiplas tarefas de um workflow; ii) execução de um workflow com múltiplas iterações; iii) possibilidade de especificar réplicas de uma tarefa para suportar equilíbrio de carga; iv) encapsulamento de tarefas em containers, possibilitando a sua execução em múltiplos nós computacionais; v) especificação flexível de workflows independentemente da infraestrutura tecnológica de execução. A partir do modelo proposto é apresentada uma arquitetura conceptual de suporte ao modelo, culminado com o desenvolvimento de um protótipo que permite experimentação e validação do modelo, com estudo de casos de aplicações concretas modeladas segundo o paradigma de workflow. O protótipo de experimentação executa-se numa infraestrutura computacional que utiliza máquinas virtuais, formando um cluster virtual com múltiplos nós computacionais (máquinas virtuais) alojados na Google Cloud Platform. Os múltiplos nós computacionais partilham ficheiros através do sistema de ficheiros distribuídos Gluster. As múltiplas tarefas de um workflow, ativadas no contexto de componentes autónomas designadas Activities, executam-se encapsuladas em containers Docker, permitindo uma grande flexibilidade de desenvolvimento e reutilização dessas Tasks em múltiplos casos de aplicação. O trabalho termina com uma forte componente experimental de casos concretos de aplicação, nomeadamente uma aplicação que permite detetar objetos e textos em imagens, uma aplicação que implementa o modelo MapReduce para realizar o histograma de ocorrência de palavras em ficheiros de texto. A experimentação com o protótipo implementado permite concluir que o modelo é suficientemente genérico, desacoplado de detalhes tecnológicos, permitindo o desenvolvimento de workflows em múltiplas áreas das ciências e engenharia.
Abstract Distributed computing infrastructures with large scale and supported by virtualization and Cloud computing platforms enable the execution of data processing applications modeled by decomposition into multiple tasks that interact with each other, according to models represented by graphs, using the ScientificWorkflows paradigm. In the last decade, many proposals for workflow systems have emerged. However, despite the greater or lesser popularity of some systems, there remain some open issues that could be improved. The work described in this document is a contribution to finding answers to some open issues, proposing a workflow model with the following characteristics: i) decentralization of control over the execution of multiple tasks in a workflow; ii) execution of a workflow with multiple iterations; iii) possibility of specifying replicas of a task to support load balancing; iv) encapsulation of tasks in containers, enabling their execution on multiple computing nodes; v) flexible specification of workflows regardless of the execution technological infrastructure. Based on the proposed model, a conceptual architecture to support the model is presented, culminating in the development of a prototype that allows experimentation and validation of the model, with case studies of concrete applications, modeled according to the workflow paradigm. The prototype for experimentation runs on a computing infrastructure that uses virtual machines, forming a virtual cluster with multiple computing nodes (virtual machines) hosted on the Google Cloud Platform. Multiple computing nodes share files through the Gluster distributed file system. The multiple tasks of a workflow, activated in the context of autonomous components called Activities, are executed encapsulated in Docker containers, allowing great flexibility in the development and reuse of these Tasks in multiple application cases. The work ends with a strong experimental component of concrete application cases, namely an application that allows detecting objects and texts in images, an application that implements the MapReduce model to create the histogram of word occurrence intext files. Experimentation with the implemented prototype allows us to conclude that the model is sufficiently generic, decoupled from technological details, allowing the development of workflows in multiple areas of science and engineering.
Abstract Distributed computing infrastructures with large scale and supported by virtualization and Cloud computing platforms enable the execution of data processing applications modeled by decomposition into multiple tasks that interact with each other, according to models represented by graphs, using the ScientificWorkflows paradigm. In the last decade, many proposals for workflow systems have emerged. However, despite the greater or lesser popularity of some systems, there remain some open issues that could be improved. The work described in this document is a contribution to finding answers to some open issues, proposing a workflow model with the following characteristics: i) decentralization of control over the execution of multiple tasks in a workflow; ii) execution of a workflow with multiple iterations; iii) possibility of specifying replicas of a task to support load balancing; iv) encapsulation of tasks in containers, enabling their execution on multiple computing nodes; v) flexible specification of workflows regardless of the execution technological infrastructure. Based on the proposed model, a conceptual architecture to support the model is presented, culminating in the development of a prototype that allows experimentation and validation of the model, with case studies of concrete applications, modeled according to the workflow paradigm. The prototype for experimentation runs on a computing infrastructure that uses virtual machines, forming a virtual cluster with multiple computing nodes (virtual machines) hosted on the Google Cloud Platform. Multiple computing nodes share files through the Gluster distributed file system. The multiple tasks of a workflow, activated in the context of autonomous components called Activities, are executed encapsulated in Docker containers, allowing great flexibility in the development and reuse of these Tasks in multiple application cases. The work ends with a strong experimental component of concrete application cases, namely an application that allows detecting objects and texts in images, an application that implements the MapReduce model to create the histogram of word occurrence intext files. Experimentation with the implemented prototype allows us to conclude that the model is sufficiently generic, decoupled from technological details, allowing the development of workflows in multiple areas of science and engineering.
Description
Projeto Final para obtenção do Grau de Mestre em Engenharia Informática e de Computadores
Keywords
Processamento distribuído Scientific workflows Máquinas virtuais Containers Publish/Subscribe Distributed processing Virtual machines
Citation
FERNANDES, Pedro de Oliveira – Processamento paralelo e distribuído baseado em workflows. Lisboa: Instituto Superior de Engenharia de Lisboa. 2023. Dissertação de Mestrado.