Before

Before
This was before I went after this rack, recabling everything.

After

After
After I went after it. There's fans and servers back there!

24 hours later...

24 hours later...
24 hours after I fixed the 3 racks pictured, I came in for my shift to see this charlie-foxtrot.

One of the better racks

One of the better racks
This one isn't near as bad as some others, but horribly hairy by datacenter standards

Internal systems

Internal systems
Our main router, plus all firewalls, gateways, and customer database servers are behind this neat bowtie of cables.

After cleanup from 1 8 hour shift

After cleanup from 1 8 hour shift
This was in the rack above, and were all extraneous cables left from decommissioned hardware that was never removed.

24 hours later part 2

24 hours later part 2
I setup and started running cables. You can see the neatly run cables at top left, and bottom right, before the admin on shift after me came in and caused the rest. This is after a full year of harping at coworkers about cable management.

Trip Hazard much?

Trip Hazard much?
The CEO came in one day and caused this lovely mess. There's a fan at the other end of the cable, right above eye level. I walk down this aisle 3-4 times a day racking $10k in server hardware, and having a trip hazard and a head strike hazard is NOT cool.

Wider angle view

Wider angle view
Aftermath of the CEO showing up one day.

WHY?!

WHY?!
WHY the hell did my coworkers remove the safety grill from this fan? I found it thrown in a dark back corner of the data center after grinding my elbow off one day.

Cooling Solutions

Cooling Solutions
This is how the company I worked for pushed air to the secondary cooler: 2 racks stacked atop each other supporting fans. Believe it or not, all this did was create a rolling hot spot that rolled around triggering high temperature warnings every hour or so.

They're everywhere!

They're everywhere!
This company spends more money on fans than they do on server hardware...and they're a BIG hosting company!

Power failure

Power failure
While running cabling, this power cable exploded in my face. A quick inspection showed that it had been run hot for so long that everything corroded. The copper conductor disintegrated in my hand.

Company issued tools

Company issued tools
Since the company refused to supply basic tools for us, we ended up using PCI blanks for about everything, including fashioning a box cutter with duct tape and a razor.

New data center

New data center
This is the new data center they built as I was on my way out. Not only does it not have raised flooring and proper cooling, but it dumps cold air out the grill at the back. The racks closest to this "cooling solution" are so cold condensation kills hard drives fast. The servers at the door are so hot that they overheat and die weekly.

A solution!

A solution!
To fix the above, the CEO took a reciprocating saw and cut through the sheetrock, without protecting the servers from the dust. The servers proceeded to choke to death for weeks, resulting in lost customer data, lost customers, and pissed off staff.

When in doubt, cut holes in the walls

When in doubt, cut holes in the walls
The CEO, as mentioned above, cut holes in the walls to vent hot air from the adjacent warehouse/storage area for new hardware into the datacenter as cooling. Another view of the mess.

A solution! part 2

A solution! part 2
This fan blocks the ONLY fire escape from the data center. There's another one like it around the corner, also blocking the fire escape. Nice work, CEO.

Poor cooling

Poor cooling
This is the temperature graph from this data center. It's on the weekly scale, so doesn't illustrate the hot spot as nicely as it should. Hot temperatures don't kill computers. Temperature swings do. This, is VERY bad.