Permalink
Derek on hadoop
  • vote
    3
    0 starsderek | Shared With: Everyone - Oct 01 2008 | statistics, blogs, hadoop, scaling, distributed computing
    Scaling Hadoop to 4000 nodes at Yahoo! (Hadoop and Distributed Computing at Yahoo!)

    In theory, this can do an hour's worth of work in one second. That is a lot of horsepower. Glad I'm not paying the power bill.

    Quoted: • 4000 nodes
    • 2 quad core Xeons @ 2.5ghz per node
    • 4x1TB SATA disks per node
    • 8G RAM per node
    • 1 gigabit ethernet on each node
    • 40 nodes per rack
    • 4 gigabit ethernet uplinks from each rack to the core (unfortunately a misconfiguration, we usually do 8 uplinks)
    • Red Hat Enterprise Linux AS release 4 (Nahant Update 5)
    • Sun Java JDK 1.6.0_05-b13
    • So that's well over 30,000 cores with nearly 16PB of raw disk!

1 - 6 of 6 Faves

Related Content from Around Faves

scale

VIEW ALL