Big Data

There is a growing number of applications that are substituting traditional big data tools such as Hadoop and Spark for serverless counterparts. Instead of managing clusters of machines, you can create a big data pipeline, converting your input to data streams and loading chunks of data into concurrent serverless functions.

The benefit of this approach is the reduced management and ease of use. However, as you have a constant processing of data, you can expect higher costs. Also, on AWS, a Lambda function can't run for more than 5 minutes, and this limit may force changes to reduce chunks of data to smaller sizes before processing.