Skip to main content

Popular posts from this blog

place for entertainment

Entertainment places are establishments that offer leisure activities for people looking to have fun, relax, or experience something new.  Here are some examples of entertainment places: Movie theaters: Offer a place to watch the latest films on the big screen.    Amusement parks:  Provide a variety of fun activities such as roller coasters, ferris wheels, and other thrill rides. Arcades: Feature classic and modern video games, air hockey, and other interactive activities.   Bowling alleys:  Offer a fun way to spend time with friends and family while enjoying a game of bowling. Concert venues: Host live music performances by popular artists and bands.    Theaters:  Offer stage performances of plays, musicals, and other shows.  Museums:  Provide educational and entertaining experiences through exhibitions of art, science, history, and more.    Aquariums:  Offer visitors the opportunity to observe marine ...

Deep-dive into Spark internals and architecture

     Apache Spark   is an open-source distributed general-purpose cluster-computing framework. A spark application is a JVM process that’s running a user code using the spark as a 3rd party library. As part of this blog, I will be showing the way Spark works on Yarn architecture with an example and the various underlying background processes that are involved such as: Spark Context Yarn Resource Manager, Application Master & launching of executors (containers). Setting up environment variables, job resources. Coarse Grained Executor Backend & Netty-based RPC. Spark Listeners. Execution of a job (Logical plan, Physical plan). Spark-WebUI. Spark Context Spark context is the first level of entry point and the heart of any spark application.  Spark-shell  is nothing but a Scala-based REPL with spark binaries which will create an object sc called spark context. We can launch the spark shell as shown below: spark-shell --master yarn \ --conf spark.ui.por...