Death To The Datacenter!

Last month, we killed our first eBay data center. Don’t worry, it had it coming...

I arrived early at the eBay San Jose campus as the rain continued to drench northern California. I joined a group of lively eBay employees from technology operations, product development and IT on a bus headed to Sacramento. We were on a journey to put our oldest data center to rest. This journey had started over a year and a half earlier, long before I had joined eBay. At that time, an aggressive plan was put into motion. The goal was to consolidate the data center portfolio to decrease costs, increase our availability and take eBay to the next level of Operational agility. It was a lofty goal.

eBay, like many of the rising star Silicon Valley companies, had been in constant react mode to keep up with demand. They had amassed a data center portfolio that spanned three states and in twelve different data center sites. Eight years earlier, the Sacramento data center (SMF) was the first to be brought online as a disaster recovery location and it was supposed to be temporary. It quickly expanded to become much more than that. When the idea of shutting it down was raised, the feeling was it was too big a task, too complex and too costly to execute. It would be like rebuilding the engine of a jumbo jet while you were in flight.

So much for that! The operations team had created a four year strategy to consolidate the data centers from colos into owned sites. The board had approved the construction of a large, owned site in Utah and that would take two years to bring online. More about this in future blog entries. But, I digress…

I’m now on my fifth month at eBay. I joined this project as it was nearing completion, so I did not have the full context. On the bus ride up to Sacramento I heard from the team just how difficult this project had been. As I listened to them explain the huge effort, both logistically and technically that it took to line this up, I started to see the real power of the team I had joined. When the eBay collective gets behind a problem, no matter how difficult, they swarm and solve it. The collaboration between architecture, product development, domain, site speed, network, the NOC, logistics, Asset Management, corp IT and the business units was simply amazing. Hundreds of people collaborated, planned, tested and executed on this highly complex project for over 18 months and got the job done on time and with zero impact to the site availability.

As the bus continued to roll north on I-80, I saw the pride on the faces of the team as they reflected back on the project. They should be proud! This is one of the hardest problems to solve in IT. They had successfully decoupled the app and DB tiers, landed them in a temporary location, moved live databases and applications – all without causing a single outage to the site.

When the bus pulled up to the data center, the rain was still falling and everyone was eager to get our first glimpse of the emptied floor space that they had all worked so hard clear. Once the rest of the group arrived we realized it was the first time that (nearly) all of the key players in this project had literally been in the same room. After a quick meet and greet, everyone was able to put a face to the name they had worked so closely with over the last 18 months. Following a complete tour of the emptied facility, we were all able to convene for lunch. A nice spread, complete with a variety of celebratory beverages, was provided and we were able to enjoy lunch on the raised floor of what, only days before, had been a live and completely functioning data center.

Paul Santana, the longest tenured eBay operations employee (11 years) and the Sacramento region data center manager had lead the team on shutting down the physical site. As you can see from the pictures it rapidly went from order to disarray as they neared the end.

Then came the really fun part! At least for me. :-)

After some final words were spoken by Paul, Bala (the project lead), myself and even Olivier Sanche, my predecessor who kept this crazy thing going before I arrived, we got down to the business of truly putting this data center to rest. After 8 years of serving as a core to eBay’s complex portfolio of data centers, all of the 3400 assets had found a new home, except one…and it was not going to get off easy. This old x335 was sitting timidly in the last rack left on the floor, waiting for its time to go, not knowing that we had something else planned for it. So much time, effort, collaboration, planning and precise execution had been put in to bring this facility to a close, but one more action was needed to officially signify the death of this data center. And with that…WHAP! I reared back and drove an axe through the final remnant of this once complex, powerful and archaic beast of a data center. A symbolic, but appropriate end to the life of this aging server…and its home… (don’t worry the asset was recycled. :-)

With the final server properly dealt with, the only thing left to do was cut the power. The honor was given to Bala Meduri, the data center closure project lead who spent countless hours preparing the data center for this moment. He donned a pair of heavy electrical gloves and pulled the switch. The switch went down, the lights went out and the data center was dead…RIP. Some final photo ops were taken, a few celebratory toasts and then it was gone, just like that.

Now, on to the next kill...another colo in California...I feel like Dexter :-)

More shutdown pictures of the SMF DC death event are included on shutterfly: http://eBayDC.shutterfly.com

(shutdown photos courtesy of Steven Stafford, Stafford Photography - imagemaker747@gmail.com)