The Overnightscape MetaFeed

We have always worked hard to provide the best hosting service you can get for the money and we’ve been pioneers in offering many unique features such as Jabber IM, WebDAV, Subversion, Phusion Passenger, and our One-Click Installer for popular web applications.  We love to play with all the newest Internet toys and provide them for you to play with also.  Our attention to all the “shiny new things” can sometimes distract us from the more fundamental aspects of our service, though, and that’s why…

The DreamHost New Year’s Resolution for 2010 is a renewed focus on performance and reliability.  Over the last few years, we have grown in leaps and bounds and our customers’ expectations of our service have grown along with it.  We have not always kept up with those expectations, and we need to put some energy into improving that.

Yesterday, I spent some time looking back at the past few years.  We’ve had our share of ups and downs and many of our loyal customers have stuck with us through it all.  We couldn’t ask for a better group of people to work for than you.  So, looking forward, we’re going to update our methods and systems to provide a more consistently high level of service.

Here’s some of the areas we’ve already improved on, and things we will be doing over the next several months…

• We are now using a more flexible storage system so we can easily keep up with your awesome demand for data storage.
• Additionally, our open-source next generation file system called Ceph is coming along nicely and will provide a great platform for our network storage needs in the future.
• A lot of work has been done over the past year on optimizing and better organizing our network infrastructure.  Everything being done is laying the groundwork for major future improvements in the works.
• By purchasing a major interest in Alchemy, our data center provider, we can rest easy knowing the relationship will be mutually beneficial for a long time.  Our system administrators are very happy to know that we will not be moving data centers again for the forseeable future.  We have plenty of spare data center space available and will be able to grow capacity at a comfortable pace.
• Our network connectivity has always been redundant but we’re going to kick that up another notch this year as we go multi-homed.  Once this is in place we’ll be able to withstand the loss an entire network point of presence without losing connectivity.
• We developed an internal system to help us manage individual server stability by improving communication and reducing wasted effort.
• We have deployed Ksplice Uptrack across our system, making mass server reboots for kernel security holes a thing of the past.  The Linux kernel has been hit with a couple of high profile security exploits in the past few months and we found ourselves scrambling to patch and reboot quickly.  Never again!
• We have a new mantra, “test, test, test”.  Some amount of testing has always been part of our procedures, but we are now making testing a fundamental component.
• Increased focus is being put on optimizing our procedures at all levels to reduce duplicated effort and errors, while also increasing accountability.  This is not something that happens overnight and it will be an ongoing process.

Having now solved our most distracting problems from the recent past, DreamHost is now positioned to tackle the future better than at any time in our history.  It won’t be easy but we have the best team in the business and a few new tricks up our sleeves on top of it.  And with that, we wish you all a very happy new year and a great 2010 (twenty-ten!).

Decade Vibrations (FN 12/31/09) 23.35 (42:25 / 39.1 MB)
The Overnightscape Underground - December 2009 - Track 35
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Frank in NJ and NYC. Snowstorm, voice messed up due to a cold, umbrellas, already 2010 in Australia, pasta, adult daycare mini-bus sliding into bus stop, hazardous conditions, lawn retreat, disabled vehicles, weird dreams (giant plastic coins, Doctor Who), get onto bus, slow going, bail plan, two hours on the bus, shoelaces, sweater, Times Square before the celebration, Nivea, head east, New Year’s resolutions, AC Revisit, low batteries, a great decade of broadcasting, 99 Cent Dreams, bank robbery transmission, and the batteries runs out.
——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Frank Edward Nora - more info at onsug.com

Decade Vibrations (FN 12/31/09) 23.35 (42:25 / 39.1 MB)
The Overnightscape Underground - December 2009 - Track 35
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Frank in NJ and NYC. Snowstorm, voice messed up due to a cold, umbrellas, already 2010 in Australia, pasta, adult daycare mini-bus sliding into bus stop, hazardous conditions, lawn retreat, disabled vehicles, weird dreams (giant plastic coins, Doctor Who), get onto bus, slow going, bail plan, two hours on the bus, shoelaces, sweater, Times Square before the celebration, Nivea, head east, New Year’s resolutions, AC Revisit, low batteries, a great decade of broadcasting, 99 Cent Dreams, bank robbery transmission, and the batteries runs out.
——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Frank Edward Nora - more info at onsug.com

AC Revisit (September 2008) 23.34 (4:21:35 / 240 MB)
The Overnightscape Underground - December 2009 - Track 35
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Last year, in September 2008, Frank and his wife Denice went on a weekend trip to Atlantic City, NJ. Frank recorded a lot of audio and released it in several different places. Now, all of the recordings from that trip are collected here in one file - with the addition of an introduction from an episode of The Overnightscape recorded a week later. Come and join Frank on the Boardwalk for another strange and wonderful journey!

The Overnightscape #741 (9/30/08) (excerpt) (10:42) - 2:30 AM, Sunday, September 28, 2008 - Went down to Atlantic City, comp rooms, shabby casino, hotel coffee maker trouble, mall, Wild Wild West Casino, money, gambling, smoking, video game slots.

The Rampler #57 (9/20/08) (25:06) - 5:54 PM, Saturday, September 20, 2008 - By the fountain in The Pier Shops at Caesars in Atlantic City, Violet Crumble, Ayala’s Herbal Water - Cloves Cardamom & Cinnamon, Oogie’s Gourmet Popcorn - Spicy Chipotle & Lime, Endangered Species Chocolate Bar - Dark Chocolate with Goldenberry & Lucuma, Dancing Fountain, Exoticar, seagull droppings, art therapy, oblivious, weird picture, piers, plans.

The Rampler #58 (9/20/08) (1:20:39) - 8:05 PM, Saturday, September 20, 2008 - Heading south on the Boardwalk, recording issues, Wild Wild West Casino, Twittering in bed, Rainforest Cafe, Casino Night Zone, artificial worlds, InDesign, pushpin, Celine Dion, free Mash, sand dunes, touching the Atlantic Ocean, traffic, fire, keys, Playcade, casino in the distance, penny slots, Skee-Ball, cross-eyed security guard, Tropicana, casino sounds, Marketplace, cat, busted bench, Primitive Radio Gods, restaurant, vegetarian food, compote, theming, Bally, The Continental, gambling, The Village People, Red Baron, That Girl, unknown fake blackjack memory, coffee maker problems, shabby hotel, sweaters, Electric Avenue, Nevada, “Dead Man’s Hand” by Gabe Leonard, insane woman, outlet shops, hard sell, bathroom issues, a lunch of pretzels, Supergrass, vouchers, your coffee and donut deal, Marketplace, Hooters, buy water, money confusion, American Express, ITC Benguiat, previous trips to Atlantic City, bachelor parties, labor memorial, Celine Dion, Rainforest Cafe.

The Rampler #59 (9/21/08) (1:06:36) - 4:02 PM, Sunday, September 21, 2008 - On the Boardwalk, Virginia City Buffet, Korean War Memorial, performers, drumming, guy talking to audience, house in buffet, On The Dunes, baked potato, dried chives, Chive Moron, henna tattoos, Merv Griffin, staring at the ocean, perceiving weird scenes, Haunted Tales, Primal Strips, Ripley’s Believe It Or Not!, amusement pier, go karts, Black Horse Pike, toll roads, kite, smoking. tapas, Tabasco Country Store, The Pink Panther, Neon Nights, Ocean One Mall, comps, weird dream of “psychodelecta music”, MDK, You Don’t Know Jack, Titanic, Adobe Wood Type, Carnival of Mystery, Lucky Ladies, Peacock Magic, a weird scene from a childhood in the 1970s, Freaks, time to get going.

The Rampler #60 (9/21/08) (33:27) - 6:45 PM, Sunday, September 21, 2008 - Music too loud, on the Boardwalk, Fred Water, OK Fred, seagulls, Ancient Rome, Westworld, Futureworld, criminal birds, reading everything, taking pictures, The Orb, phone calls, looking around, red signs, getting darker, country music, conspiracy theories, garbage picking, tongue twister, history of the Wild Wild West Casino, clocks, getting ready to head home.

Casino Ambience (9/21/08) 7.14 (45:05) - 1:43 PM, Sunday, September 21, 2008 - The ambient sounds of a casino in Atlantic City, NJ.

——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Frank Edward Nora - more info at onsug.com

As the end of the decade approaches, it’s a perfect time to look back and reflect on the past.  What has gone wrong and what has gone right?  We have a way of making waves, even if not always in the way we might like.  Here’s some reflections on some of those waves.

Power

Starting about 4 years ago, we were battling a problem of power constraints at our data center.  That led to a general inability to provision data center space the way we wanted to and we had to become very creative to manage our system.  The biggest problems from that time were two unplanned power outages of the entire building followed by an emergency planned outage 8 months later.  Power is one of the lifelines of a data center and not being able to rely on it can be a major distraction.  It was during that period that we established our off network status site (dreamhoststatus.com), which has proven to be a great asset.  Those issues are long behind us now, and looking forward we have plenty of power capacity.

Network

Pretty much in the middle of the power situation we were also hit with a networking problem between our two core routers that was causing serious website slowness.  We had grown so quickly that nobody really understood exactly how the network devices were interacting.  That combined with the distraction of the power situation made it take way too long to resolve the problem.  From that, we learned the hard way what we needed to do to improve and maintain the network.  Things have improved, but we were still recently hit with a couple network outages.  They were caused by human errors and weaknesses in the procedures we had in place.  We have already refined those procedures and future improvements to the networking infrastructure will also help to mitigate the potential for human error to result in outages.  This is still very much a work in progress but huge steps have been taken already and more are still to come.

Data Storage

The next major hurdle we faced involved our data storage infrastructure.  In the early days we had migrated from disks inside each server to network attached storage for the added redundancy and to allow us to better utilize our available storage.  At that time hard drives were only 9 gigabytes and our users were generally only using a hundred megabytes or less of space.  Boy, have things changed!  We now have users with multiple terabytes of data and even everyday websites sometimes have multiple gigabytes of files.  The huge growth of online video and the popularity of digital photography increased the demands on our storage infrastructure by a couple orders of magnitude over the years.  To accommodate that growth, our simple system of file servers had grown to a large network of over 100 individual servers that we had to constantly juggle data between.  The cost per gigabyte for our Network Appliance based system had also not come down nearly as fast as the per-user storage requirements had gone up, but we had become reliant on network storage for things such as backups, rapid recovery from server failures, and seamless sharing of data between separate hosting and email accounts.  We were addicted to network storage, and our next couple of major performance problems came from gloriously failed experiments with other storage products.  One of them was cheap and unreliable, and the other was expensive and unreliable.  We couldn’t win!  (Note that we haven’t used either of those for awhile so both of them may work better now than they did for us a couple of years ago.)

Through those years that our addiction to network storage had developed, a shift had happened that we hadn’t noticed.  First, individual hard drives had dropped like a rock in price and skyrocketed in capacity.  Second, users were consuming data at such a rate that evenly utilizing our available storage was no longer a problem.  Switching back to locally installed storage was the answer we had been looking for!  We started experimenting with the new server architecture and developing a backup strategy.  Then once the pieces were all in place, we started moving forward with it.  So the storage bottleneck was resolved and we had a clear path forward, but we were still throwing out a lot of knowledge we had learned over the preceding decade and were starting over from scratch in a lot of ways.  With every technology shift comes with it a new set of problems and this one was no exception.  It’s been about a year and a half since then and we’re already on our fourth revision of the server configuration (three different RAID cards with two different configs).  The current hardware has been working out quite well but we’re doing some testing to see if it can be optimized further.

Looking into next year, our core technology systems are under control and we’ll be able to focus more on improving the service than we have in years.  The future looks very bright…. but more on that tomorrow!

Anti-Energy (FN 12/30/09) 23.33 (38:59 / 36 MB)
The Overnightscape Underground - December 2009 - Track 33
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Frank in NYC. The anti-energy drink trend, kava kava, Drank, construction, Australian, Lost, Dream Water - Snoozeberry, the crunk lifestyle, weird dream (Memories of being on The David Letterman Show), confused, alternate timelines, Coffee Bean Lifestyle, 9th Avenue, bagels, Beer of the Month Club, Hot Sauce of the Month Club, Hell’s Kitchen, live shows, 7-11, candy, Babe Ruth Crisp, Hooray For Everything video “Side By Side”, subway, E train, chemtrails, Silk Road, Kitaro, Sprite Green, bowling alleys, Xanadu, Take Five, escalator, Ess-A-Bagel, F & B, FreeFoods, The Pump, noodle pudding, and The Rutles.
——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Frank Edward Nora - more info at onsug.com

Semiconscious (FN 12/29/09) 23.32 (1:02:57 / 57.9 MB)
The Overnightscape Underground - December 2009 - Track 32
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Frank in NYC. In a semiconscious state, bathroom line, Lost, freezing cold, fever dreams, miso, drug store, dextromethorphan, the howling wind, scrubs, the underwear bomb, Ancient Sweets, BMX, Nuclear Cowboyz, iPhone, Apple Macintosh, Doctor Who, Grand Central Station, wall, games (Wits & Wagers, Perudo, Pandemic), shopping cats, Almond Joy Pieces, laser show, The Yule Log, beer, subway, urban exploration, white and black preachers, New Years recordings, The Corridor of Losers, “Loser” by Beck, “I’m a Loser” by The Beatles, and consonants.
——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Frank Edward Nora - more info at onsug.com

Morning Commute (with Bob) #16 - “Errands” (12/29/09) 23.31 (25:25 / 23.6 MB)
The Overnightscape Underground - December 2009 - Track 31
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Join Bob LeMent (from Static Radio) rambling on his morning commute into St. Louis, Missouri…  “Errands” - Gift Cards, Gas Mileage and Hunting, help during the holidays, Morning Commute.
——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Bob LeMent - more info at onsug.com and StaticRadio.com
——————————

Thanks to everyone who tested 2.9.1 Beta 1.  We’re following that up with Release Candidate 1.  RC1 contains a few more fixes, bringing the number of fixed tickets up to 23.  If you are already running Beta 1, visit Tools->Upgrade in your blog’s admin to get RC1.  You can also  download the RC1 package and install manually.  If all goes well, 2.9.1 will be here soon.

Spaceboat O. P. Fogoncs (FN 12/28/09) 23.30 (1:33:57 / 86.3 MB)
The Overnightscape Underground - December 2009 - Track 30
“Your Late Night Broadcast” online at onsug.com
Created by Frank Nora (frank@theovernightscape.com) in New Jersey, USA
——————————
Frank in NYC and NJ. At a truck stop alone at night in the rain, 18 Wheels of Steel: American Long Haul, train going by underneath, driving like a maniac, vermicelli noodles, rice, hard falafel, honey lozenges, Grand Central Terminal, Revolutionary Road, Dream Water preview, trucking, working on “AC Revisit”, drumming, Metropolis Street Racer, Devastating Nightscape, too cold, iPhone, bogus attack, getting on the bus, VH1 station promos, Phil Collins, Ratt, sci-fi idea “Spaceboat O. P. Fogoncs”, The Doors, “Twentieth Century Fox”, violence, bus ride home, details on the storyline, new names, people living together on a spaceship, Star Trek, Firefly, Lost in Space, time travel, Howard Stern, Texas Hold’em, ice injury, polenta recipe, “Give Blood” by Pete Townsend, chord changes, “Let the Day Begin” by The Call, “Revolutionary Road” trailer, “Kick-Ass” Trailer, Hit-Girl, The Banana Splits, and name that tune.
——————————
License for this track: Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States (http://creativecommons.org/licenses/by-nc-nd/3.0/us/). Attribution: by Frank Edward Nora - more info at onsug.com