It is important to calculate the storage requirement along with the file formats when designing a data lake, data platform, or big data-based solution. The performance of applications and processes…
Many things needs to be considered before finding the right hardware for Hadoop clusters. Hadoop workloads tend to vary a lot betweek different jobs. It takes experience to correctly anticipate the amounts of storage, processing power, and inter-node communication that will be required for different kinds of jobs.
One of the most important aspect of architecting a solution with Big Data is choosing a proper Data Storage options in Hadoop/Spark. Hadoop does not have a standard data storage format ,but as a standard file system ,allows for storage of data in any format ,whether it’s text,binary ,image or other.
When working with Hadoop Framework to process massive data, we will face many challanges and difficulties which includes the Input/Output(I/O) and network related bottlenecks. To overcome this problem, one needs to understand the different data compression techniques available in Hadoop Framework.
A single cancer patient on average can generate 1 Terabyte of biomedical diagnostic and clinical data. According to a paper from GLOBCAN 2018, there will be 18.1 million new cancer…
Partitions are fundamentally horizontal slices of data that allow large sets of data to be segmented into more manageable chunks. They are virtual columns acting as storage units but, not…
YARN stands for Yet Another Resource Negotiator. It is a centralized cluster resource management and job scheduling platform to deliver scalable operations across the cluster. It was introduced in Hadoop…
There are 3 types of joins, Reduce-Side joins, Map-Side joins, and memory-backed Joins that can be used to join Tables in MapReduce. Map Side Join Joining at the map side…
This post describes some of the most useful Apache Hadoop HDFS commands one would need when working in a Hadoop Cluster
Hadoop Mapreduce framework is a Big data processing framework which consists of MapReduce programming model and Hadoop Distributed File System.