Use All Spark Workers’ CPUs to read from a REST API 

Spark driver and workers:   Recently, we explored how to create our data sources in Spark by using syntax like:   spark.read.format(“myrestdatasource”) to read from a REST API. Now, we’ll examine how to use all worker CPUs to read simultaneously from a REST API.    Key idea  Spark will read in parallel if we define multiple […]