In the crap storm known as life you sometime find your self bored and you think what the heck I should by that game that everyone has been talking about. Well that game can become a crap storm of itself. You may asking how is a little game a crap storm? Well heres an example, In 2009 there was a little known game call Minecraft it was not known by anyone and released by a new indie game company. Now this was a very special game, it fell into a genre of a sandbox game. Today it is known world wide as a fun and super addictive game with a total of over 6,000,000 units sold and almost 30,000,000 registered users. Sales have brought in millions to the small company making all of it’s 16 members instant millionaires. This game is so successful because it is so addictive. This is the kind of game that you cannot put down until you are about to fall asleep. This and many other games (ex. WOW, Diablo 3, and others) can eat up a lot of your time. Although these games can be crap storm they can be also extremely fun if they are managed correctly.
Recently I inherited a legacy SQL Server 2008 infrastructure due to an acquired company. A standard setup; monolithic database with TLog backups and a log ship recipient for DR. The DR instance is on a separate SAN of different DNA (vendor and access method) and employs a several hour mechanical log replay delay to guard against data corruption. Not bad considering the size of the entity.
The entity also has an SSRS reporting instance with DASD which performs a full restore each day to present the previous day’s data to SSRS report users. The daily full restore consumes a large amount of bandwidth loading from the network backup device and due to the size of the reporting audience is not commensurate to the resources consumed.
Bright idea! I decided to kill two birds with one stone. I took the DR instance, which resides on a powerful underutilized server, and substituted SQL Mirroring with two snapshots. One every 24 hours for reporting and one with a multi hour mechanical delay to guard against near time data corruption. Easy as pie. I pointed SSRS server connection configs to the snaps and presto! Yes, I am master of my little domain.
However, I began watching the DR SAN load and noticed a large multiple compared to the source DB. Didn’t make sense. We are talking somewhere on the order of 5X verses the source DB. And almost all write IO. So I performed the usual troubleshooting steps to make sure I was seeing the correct numbers. I was. So I gave the DB and my brain a couple of days to bake in the hopes one of them would come to equilibrium.
And then it dawned on me. The dang snapshots! As changes are being applied to the mirror, snapshot chains are being written to storage. I had forgotten about the additional IO overhead in maintaining snaps. Gotta keep up with those changes which results in additional writes. In my case, 5X.
So when you are considering mirroring and snaps for your environment, don’t forget to vet your bright idea with load capacity of you storage.
If you forget, you might just end up in a CrapStorm.
I have a friend who sells commercial real estate. Nicest guy you could ever meet. Really! I hate him.
No, I don’t hate him. But I am that technical support friend so yes, I hate him. At least I do when I hear those omminous words; Hey buddy, could you help me with something? Of course the answer is and will always be yes. In this case the request is quite simple; setting up a WordPress site for his business.
So I go through the robotic motions of acquiring a hosting plan, one click install of WordPress, email, and all the standard minutia involved. Easy as pie. But it’s not. As I went through the habitual keystrokes my mind began to wander and I realized if not for being a geek I could never do this. Terms which are lecherously burned into my memory mean nothing to those not technically gifted. Gibberish in reality. And I asked, why?
Answer? The products, interfaces, and instructions were designed, implemented, and written by geeks or near geeks. Makes perfect sense to us, but might as well be an ancient Vulcan dialect to the technically challenged of the world.
Because I loathe being that tech support guy, the next time I design something for non technical users my aged mother will be the designer. There will be large blinking buttons labeled “Click This Button”, with instructions such as “Open Your Internet”, “Pay Me Please”, “What is your dog’s name?”, ad nauseam, ad infinitum. There will be ZERO technical terms, acronyms, or twenty-something speak. Clear, who cares if it page ranks, language.
Products your mother can understand. Wouldn’t that be a CrapStorm.
Recently I wrote I post titled “What is a Data Warehouse”. I received both interesting and appreciated feedback. As fate would have it, I am now responsible for building a Business Intelligence group. Fortunately, I am inheriting a few talented people and have a budget to hire one or more SQL Junkies to write reports. Some might think the sun is shining down on my face. Well it is, and it isn’t. It isn’t due to the missing link; on demand data communications. Allow me to explain.
When a company is small, your production infrastructure is either in-house or collocated at an ISP. The point being, everything is on the same switching infrastructure. Data movement tends to be straightforward; mirroring, replication, ETL, Log Ship, etc. And because of this, you can move data.
When a company is large; either through acquisitions or organic growth, capital budgets tend to be reasonable and disperse business units are connected via dedicated communication links. Data movement is straightforward, as the infrastructure is robust enough to support data bursting.
But, the middle is tough. A mid-cap company is growing; in my case largely via acquisition. We are constantly integrating new, geographically dispersed business units. Yes, constantly. As new units are being integrated, capital for temporary, dedicated communications is just not there. Plus it takes time to contract them; too many regulating cooks in the kitchen. The net result is I must find a way to implement a consolidated data and reporting infrastructure without the luxury of robust inter-company communications. I don’t know about you, but this aint no easy task. To add more to the challenge, we’ve got SQL Server, MySql, MongoDB, Redis, and a sprinkling of others not important enough to mention. The reporting infrastructure is SQL Server 2008 R2 so we have that going for us.
Until infrastructure consolidation [according to corporate computing strategy] is in place, I must figure out a way to move disparate near-time/near-line data into our infrastructure. In one particular case, physics is kicking me in the head … hard. An unnamed company has multiple terabytes of data but their Internet pipe is slower than your Cable/DSL connection at home. Yes, we could increase OpEx by contracting a bigger front door, but only if I must. That generally involves contact modifications, which can be expensive. And as you know, OpEx impacts EBITDA while CapEx does not. EBITDA is king around here.
So, much to my chagrin I must resort to USB drives to get the initial data load into our infrastructure [/shame]. Internet VPNs are notoriously suspect to packet loss and I don’t want to tie up their front door for a week. And to add insult to injury, I must implement data aggregation routines at the source to ship data updates on a regular basis. And of course that work is probably throw away.
Wouldn’t the world be a better place if a company could quickly spin up temporary, national high-speed data communications? Contract that shit, use it, and it goes away. Think about it. We need 200Mb door to door for three months. Man, that would be a life saver. Enron tried to build this and we know where they ended up. As I understand it, they attempted to model a communications exchange like the national gas pipeline network. Contract, use, and stop. Of course they screwed it up.
It is my hope this type of national infrastructure is available one day. But I don’t think we will see it for a while. There are too many communication companies protecting their kingdoms. Too many municipal governments protecting their manholes. And too many regulating authorities sticking their noses in everybody’s business. And nobody really wants to work together. Too much to lose, right? Why compete on price and features when I can protect my predictable, regulated revenue stream. I get it. Who doesn’t want to protect such things? But you can only protect so long. One day, someone will do it and that nice, predictable revenue stream will start to decline. But if we could find a way, competition would drive down data costs and corporate America would benefit in too many ways to count.
On demand data communications, wouldn’t that be a Crap Storm.
Few things in life are like a new car. That smell, the ride, the emotion of it all! Maybe it is just a guy thing, but for me it is the ultimate drug. Takes years off my reflection in the mirror. I am a man of men. It’s like winning the Internet.
When younger, and making good money for my age, I bought a new car every couple of years. It was a guilty pleasure. My prideful shame. I loved them so much I would wipe them down with baby diapers most every day. Kept them spotless, unmarked, virginal if you will. Often, I found myself in the garage just staring, smiling, in an auto-erotic daze. Damn, it was good.
But, life comes racing towards you and one day you wise up. Might was well drive the wheels off the car you have. Yes, I grew up, or old; not sure which. I will force myself to drive the ride I have. But … what about the withdrawals? I MISS THAT SMELL!!!
Over the years I found a way to drive a new car every couple of years; without actually purchasing one. How you say? What sorcery is this I wield? I will let you in on my secret. Maintenance.
I spent $892 dollars today on maintenance. New tires, oil change, filter change, new wipers, and a wipe down with that old baby diaper. Presto! New car. Rides like new again, accelerates like new again, and looks [almost] new again. My addiction has been quelled.
Next time you get that urge to [ahem] try new things, try a little maintenance on your ride. She will love you for it.
A new car for $892. Wouldn’t that be a Crap Storm.