Date: 10 September 2012 15:40
Subject: Re: One petabyte of data loading into HDFS with in 10 min.
To: user hadoop.apache.org
We have loaded 100GB data loaded into HDFS, time taken 1hr.with below configuration.
Each Node (1 machine master, 2 machines are slave)
1. 500 GB hard disk.
2. 4Gb RAM
3. 3 quad code CPUs.
4. Speed 1333 MHz
Now, we are planning to load 1 petabyte of data (single file) into Hadoop HDFS and Hive table within 10-20 minutes. For this we need a clarification below.
1. what are the system configuration setup required for all the 3 machine's ?.
2. Hard disk size.
3. RAM size.
4. Mother board
5. Network cable
6. How much Gbps Infiniband required.
For the same setup we need cloud computing environment too?
Please suggest and help me on this.