Please use this identifier to cite or link to this item:
Title: Study of the Feasibility of Serverless Access Transparency for Python Multiprocessing Applications
Author: Finol, Gerard  
Arjona, Aitor  
Tutor: Garcia Lopez, Pedro  
Abstract: Access transparency means that both local and remote resources are accessed using identical operations. Transparency simpli es the complexity of programming a distributed system because the system is perceived as a whole rather than a collection of independent components. With access transparency, we can treat disaggregated compute, storage, and memory resources as if they were a single monolithic machine. This would considerably simplify the creation and execution of parallel applications in the Cloud in a scalable manner. In this work, we evaluate the feasibility of access transparency over state-of-the-art Cloud disaggregated resources. We propose an alternative implementation of Python's multiprocessing API that transparently runs distributed processes on serverless functions and that leverages disaggregated in-memory storage to maintain the shared state of processes consistent and mediate their communication. To evaluate transparency, we have used four parallel stateful applications intended to be executed locally (Uber Research's Evolution Strategies, Baselines-AI's Proximal Policy Optimization, Pandaral·lel's dataframe and ScikitLearn's Hyperparameter tuning), and, without changing the code, we have scaled them with serverless technology. We compare execution time and scalability of the same application running over disaggregated resources using our library, with the single-machine Python libraries in a large VM. Despite the higher latency and lower throughput of communication, we achieve comparable results and we observe that the applications can continue to scale beyond VM limited resources leading to a better speedup and parallelism.
Keywords: transparency
Document type: info:eu-repo/semantics/masterThesis
Issue Date: 29-Jun-2021
Publication license:  
Appears in Collections:Bachelor thesis, research projects, etc.

Files in This Item:
File Description SizeFormat 
gfinol_aarjonapeFMDP0621report.pdfReport of FMDP4,6 MBAdobe PDFThumbnail
View statistics

This item is licensed under aCreative Commons License Creative Commons