Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track)

Germán T. Eizaguirre; Daniel Barcelona-Pons; Aitor Arjona; Gil Vernik; Pedro García-López; Theodore Alexandrov

doi:10.1145/3700824.3701095

Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track)

Germán T. Eizaguirre, Daniel Barcelona-Pons, Aitor Arjona, Gil Vernik, Pedro García-López, Theodore Alexandrov

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The scalability of cloud functions makes them a convenient backend for elastic data analytics pipelines where parallelism changes drastically from one stage to the next. However, cloud functions require intermediate storage systems for communication, which limits the efficiency of stateful operations. Furthermore, cloud functions are expensive, which reduces the cost-effectiveness of pure serverless architectures. We propose a hybrid architecture for data analytics that uses cloud functions for embarrassingly parallel stages and virtual cloud instances for stateful operations under a unified serverless programming framework. Extending Lithops, a serverless programming library, we implement a parallel programming interface that proactively provisions serverless and serverful cloud resources with minimal user intervention. We validate the feasibility of a hybrid architecture, by comparing it to fully serverless and serverful versions of a production-level metabolomics pipeline. We show that mixing cloud functions with virtual instances increases the cost-effectiveness of the execution by up to 188.23% over the serverless implementation, while achieving a speedup of 3.64 compared to the serverful one.

Original language	English
Title of host publication	Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of
Subtitle of host publication	Middleware 2024
Publisher	Association for Computing Machinery, Inc
Pages	15-21
Number of pages	7
ISBN (Electronic)	9798400713194
DOIs	https://doi.org/10.1145/3700824.3701095
State	Published - 2 Dec 2024
Externally published	Yes
Event	2024 Middleware Industrial Track, Middleware Industrial Track 2024 - Hong Kong, Hong Kong Duration: 2 Dec 2024 → 6 Dec 2024

Publication series

Name	Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024

Conference

Conference	2024 Middleware Industrial Track, Middleware Industrial Track 2024
Country/Territory	Hong Kong
City	Hong Kong
Period	2/12/24 → 6/12/24

Bibliographical note

Publisher Copyright:
© 2024 Copyright held by the owner/author(s).

Keywords

Cloud computing
function-as-a-service
resource allocation
resource efficiency
serverless computing

ASJC Scopus subject areas

Information Systems
Software

Access to Document

10.1145/3700824.3701095

Cite this

Eizaguirre, G. T., Barcelona-Pons, D., Arjona, A., Vernik, G., García-López, P., & Alexandrov, T. (2024). Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track). In Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024 (pp. 15-21). (Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024). Association for Computing Machinery, Inc. https://doi.org/10.1145/3700824.3701095

Eizaguirre, Germán T. ; Barcelona-Pons, Daniel ; Arjona, Aitor et al. / Serverful Functions : Leveraging Servers in Complex Serverless Workflows (industry track). Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024. Association for Computing Machinery, Inc, 2024. pp. 15-21 (Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024).

@inproceedings{71d7681c798549929edac19a17599b59,

title = "Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track)",

abstract = "The scalability of cloud functions makes them a convenient backend for elastic data analytics pipelines where parallelism changes drastically from one stage to the next. However, cloud functions require intermediate storage systems for communication, which limits the efficiency of stateful operations. Furthermore, cloud functions are expensive, which reduces the cost-effectiveness of pure serverless architectures. We propose a hybrid architecture for data analytics that uses cloud functions for embarrassingly parallel stages and virtual cloud instances for stateful operations under a unified serverless programming framework. Extending Lithops, a serverless programming library, we implement a parallel programming interface that proactively provisions serverless and serverful cloud resources with minimal user intervention. We validate the feasibility of a hybrid architecture, by comparing it to fully serverless and serverful versions of a production-level metabolomics pipeline. We show that mixing cloud functions with virtual instances increases the cost-effectiveness of the execution by up to 188.23% over the serverless implementation, while achieving a speedup of 3.64 compared to the serverful one.",

keywords = "Cloud computing, function-as-a-service, resource allocation, resource efficiency, serverless computing",

author = "Eizaguirre, {Germ{\'a}n T.} and Daniel Barcelona-Pons and Aitor Arjona and Gil Vernik and Pedro Garc{\'i}a-L{\'o}pez and Theodore Alexandrov",

note = "Publisher Copyright: {\textcopyright} 2024 Copyright held by the owner/author(s).; 2024 Middleware Industrial Track, Middleware Industrial Track 2024 ; Conference date: 02-12-2024 Through 06-12-2024",

year = "2024",

month = dec,

day = "2",

doi = "10.1145/3700824.3701095",

language = "English",

series = "Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024",

publisher = "Association for Computing Machinery, Inc",

pages = "15--21",

booktitle = "Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of",

}

Eizaguirre, GT, Barcelona-Pons, D, Arjona, A, Vernik, G, García-López, P & Alexandrov, T 2024, Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track). in Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024. Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024, Association for Computing Machinery, Inc, pp. 15-21, 2024 Middleware Industrial Track, Middleware Industrial Track 2024, Hong Kong, Hong Kong, 2/12/24. https://doi.org/10.1145/3700824.3701095

Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track). / Eizaguirre, Germán T.; Barcelona-Pons, Daniel; Arjona, Aitor et al.
Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024. Association for Computing Machinery, Inc, 2024. p. 15-21 (Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Serverful Functions

T2 - 2024 Middleware Industrial Track, Middleware Industrial Track 2024

AU - Eizaguirre, Germán T.

AU - Barcelona-Pons, Daniel

AU - Arjona, Aitor

AU - Vernik, Gil

AU - García-López, Pedro

AU - Alexandrov, Theodore

PY - 2024/12/2

Y1 - 2024/12/2

N2 - The scalability of cloud functions makes them a convenient backend for elastic data analytics pipelines where parallelism changes drastically from one stage to the next. However, cloud functions require intermediate storage systems for communication, which limits the efficiency of stateful operations. Furthermore, cloud functions are expensive, which reduces the cost-effectiveness of pure serverless architectures. We propose a hybrid architecture for data analytics that uses cloud functions for embarrassingly parallel stages and virtual cloud instances for stateful operations under a unified serverless programming framework. Extending Lithops, a serverless programming library, we implement a parallel programming interface that proactively provisions serverless and serverful cloud resources with minimal user intervention. We validate the feasibility of a hybrid architecture, by comparing it to fully serverless and serverful versions of a production-level metabolomics pipeline. We show that mixing cloud functions with virtual instances increases the cost-effectiveness of the execution by up to 188.23% over the serverless implementation, while achieving a speedup of 3.64 compared to the serverful one.

AB - The scalability of cloud functions makes them a convenient backend for elastic data analytics pipelines where parallelism changes drastically from one stage to the next. However, cloud functions require intermediate storage systems for communication, which limits the efficiency of stateful operations. Furthermore, cloud functions are expensive, which reduces the cost-effectiveness of pure serverless architectures. We propose a hybrid architecture for data analytics that uses cloud functions for embarrassingly parallel stages and virtual cloud instances for stateful operations under a unified serverless programming framework. Extending Lithops, a serverless programming library, we implement a parallel programming interface that proactively provisions serverless and serverful cloud resources with minimal user intervention. We validate the feasibility of a hybrid architecture, by comparing it to fully serverless and serverful versions of a production-level metabolomics pipeline. We show that mixing cloud functions with virtual instances increases the cost-effectiveness of the execution by up to 188.23% over the serverless implementation, while achieving a speedup of 3.64 compared to the serverful one.

KW - Cloud computing

KW - function-as-a-service

KW - resource allocation

KW - resource efficiency

KW - serverless computing

UR - http://www.scopus.com/inward/record.url?scp=85215507706&partnerID=8YFLogxK

U2 - 10.1145/3700824.3701095

DO - 10.1145/3700824.3701095

M3 - Conference contribution

AN - SCOPUS:85215507706

T3 - Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024

SP - 15

EP - 21

BT - Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of

PB - Association for Computing Machinery, Inc

Y2 - 2 December 2024 through 6 December 2024

ER -

Eizaguirre GT, Barcelona-Pons D, Arjona A, Vernik G, García-López P, Alexandrov T. Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track). In Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024. Association for Computing Machinery, Inc. 2024. p. 15-21. (Middleware Industrial Track 2024 - Proceedings of the Middleware Industrial Track, Part of: Middleware 2024). doi: 10.1145/3700824.3701095

Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track)

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this