Permalink
Derek on scaling
  • vote
    3
    0 starsderek | Shared With: Everyone - Oct 01 2008 | statistics, blogs, hadoop, scaling, distributed computing
    Scaling Hadoop to 4000 nodes at Yahoo! (Hadoop and Distributed Computing at Yahoo!)

    In theory, this can do an hour's worth of work in one second. That is a lot of horsepower. Glad I'm not paying the power bill.

    Quoted: • 4000 nodes
    • 2 quad core Xeons @ 2.5ghz per node
    • 4x1TB SATA disks per node
    • 8G RAM per node
    • 1 gigabit ethernet on each node
    • 40 nodes per rack
    • 4 gigabit ethernet uplinks from each rack to the core (unfortunately a misconfiguration, we usually do 8 uplinks)
    • Red Hat Enterprise Linux AS release 4 (Nahant Update 5)
    • Sun Java JDK 1.6.0_05-b13
    • So that's well over 30,000 cores with nearly 16PB of raw disk!

  • vote
    26
    0 starsderek | Shared With: Everyone - Aug 21 2008 | mysql, hacks, facebook, scaling, data center
    Facebook | Engineering @ Facebook's Notes

    Interesting writeup of hacking MySQL to dirty memcached entries after replication sync.

    Quoted: REPLACE INTO profile (`first_name`) VALUES ('Monkey') WHERE `user_id`='jsobel' MEMCACHE_DIRTY 'jsobel:first_name'

  • vote
    1
    0 starsderek | Shared With: Everyone - Feb 05 2008 | caching, web, performance, scaling
    Performance Research, Part 2: Browser Cache Usage - Exposed! » Yahoo! User Interface Blog

    Good data, don't rely on cache.

    Quoted: 40-60% of Yahoo!’s users have an empty cache experience and ~20% of all page views are done with an empty cache. To my knowledge, there’s no other research that shows this kind of information. And I don’t know about you, but these results came to us as a big surprise. It says that even if your assets are optimized for maximum caching, there are a significant number of users that will always have an empty cache. This goes back to the earlier point that reducing the number of HTTP requests has the biggest impact on reducing response time. The percentage of users with an empty cache for different web pages may vary, especially for pages with a high number of active (daily) users. However, we found in our study that regardless of usage patterns, the percentage of page views with an empty cache is always ~20%.

  • vote
    7
    0 starsderek | Shared With: Everyone - Nov 12 2007 | ocaml, jocaml, erlang, performance, scaling, concurrency
    eigenclass - Conclusions about Wide Finder, C++, OCaml, JoCaml, Erlang and friends

    Quoted: The main advantage of the JoCaml version is that, unlike the C++ one (based on threads), it only takes a one-line change to make it run in multiple machines.

1 - 9 of 9 Faves

Related Content from Around Faves

scale

VIEW ALL