Database administrators have enormous responsibility whether they manage one or hundreds of servers. In this article, I will tell you how she survived as the Lone DBA for 56 database servers for over a decade. While many DBAs work on teams instead of alone, I have great advice for all.
I was a Lone DBA for over 15 years and during that time I have learned a thing or two about how to survive on my own in relatively large environments. In this article, I will share with you some of the big things I learned during those years that may help you survive being a Lone DBA and still love doing it.
Calling in Help
One of the most important things to understand is when to admit you cannot do it all. When working alone on 56 servers, the workload can seem insurmountable. There are times when in one week I will do 70 plus tasks, not including project work and daily monitoring. To manage this type of workload, you must learn to work smarter not harder. That’s when you have to enlist help and hire consultants.
I am Afraid
Many people think that hiring consultants is admitting you are incapable of doing your job. Some think that if you hire consultants, it opens the door for the company to think that they may not need an ‘in-house’ DBA. It may lead them to just hire a consulting company to do the work. At the last company I worked for, employees frowned and complained every time a consultant was brought in for anything. Some even refused to share knowledge hoping to protect their jobs somehow. I think this is nonsense. You shouldn’t worry about being replaced by consultants. A consultant only has a superficial knowledge of the company. You are the one that understands the needs of the business. The consultants don’t. Don’t let it scare you.
Free Up Your Time
The biggest obstacle I must contend with as a Lone DBA is time. I have no time; every minute of my workday is used. My world is all about prioritizing what needs to get done. Sometimes there is just not enough time in the day to accomplish everything. Hiring a consultant doesn’t mean you can’t do the work; it means you are managing your workload.
I hire consultants from time to time to free up my plate and cover some of the workloads, so I can focus on higher priorities. At times, I use them to do the normal redundant or routine admin work, little things that add up to a lot of time in a week. On other occasions, I admittedly give them stuff I don’t want to do or get tired of doing (but if you know me, I never really get ‘tired’ of doing anything DBA related, I am just proving a point). I will also give them the big projects that take too much time. Time is invaluable. For example, I may need to build a new cube. That, as you may know, takes a lot of time. I know how to build and design cubes, but why should I spend hundreds of hours working on that when I can farm it out?
Do You Want to Take a Vacation Ever?
Vacation, what’s that? Most DBAs can take a vacation without having to work while away because there is someone to cover and share the responsibilities. When it’s just you, you take work on vacation along with you. One of the best benefits of hiring a consultant or a DBA service is to be able to leave that workload behind and take a real vacation. It took me years to realize this. I took my first workfree vacation just earlier this year; it was wonderful to hand the reins over for a week and not have to worry about it.
Gotchas
However, there are a few gotchas to admitting you can’t do everything and hiring a consultant. One of the main ones for me is giving up what I like to do. I love the core DBA stuff; turning that over to someone else to do is not easy for me. Relinquishing that work can be very tough. I also find that spending time hand-holding the consultant is another gotcha. Consultants do not know the ins and outs of your environment. Getting them started on a project can take time, but in the end, it’s worth it.
Embrace It
The moral of the story is that it’s hugely important to admit to yourself that you can’t do it all. It took years for me to realize that. If you are juggling a workload for many years by yourself, consider hiring help. You’ll thank me for it.
Jack of All Trades Master of None
Being a Lone DBA gives you so much exposure to so many facets of SQL Server. Since I am just a team of one, I get to work on replication, administration, security, business intelligence, disaster recovery, Reporting Services, Integration Services, Analysis Services, database design, development, performance… You name it, I get to dabble in it. However, being able to work on every facet also means I will never be a Master at any of it and that’s okay by me.
For a Type A personality like me, this is a hard thing to come to terms with. I’ve learned over time to accept not knowing everything. I relish the fact that I get to do and experience MUCH more than most. Those who are not Lone DBAs have to divide and conquer or are responsible for just a handful of areas (like security, or DR, or change management). However, in our line of work, there is always a need to GOTO experts. Through networking, I have gained several friends who have become my experts. I have an expert on things like PowerShell, database internals, storage, Availability Groups, T-SQL, etc. If I need expert knowledge on something, they are always willing to lend a hand. If you don’t have a network of GO TO experts, whether you are a Lone DBA or not, I strongly suggest you start building those relationships.
That being said, I will never be one of those GO TO experts. However, if someone asks a question about whether I have ever done something or had a particular issue, in most cases the answer is yes. How do I accomplish that? The answer is by creating a broad skill set. I self-teach by dabbling in things. I am not afraid of trial and error. I learn all the SQL tools I can and use them where appropriate. I attend as many SQL training events as I can manage. I am always trying to further diversify my knowledgebase. I attend my user group meetings (now run them), virtual training sessions and watch 24HOP (24 Hours of PASS) sessions. I get the PASS Summit sessions on USB every year to watch when I have time, and, finally, I attend SQL Saturdays. These are great ways to extend my knowledgebase.
The most important tip I can give is to learn just what you need to get most jobs done and don’t try to master it. It’s okay to be a master of none; revel in it and embrace that you get to work on so many things. It will make you very marketable; there are not many of us who are given that opportunity.
Run Book Run!
Do you have a ‘Hit-by-the-Bus’ handbook? What is that, you ask? It is a document that explains how to execute all your jobs and SSIS packages. In addition, I preface mine with all key elements someone might need, like where passwords are stored, architectures, backup times, where are backups stored, etc., then dig into the job steps. The purpose of this document is so that someone with some SQL skills could step in if needed. You never know when you will be hit by a bus or win the lottery, and someone has to take over for you.
Important Things to Note
- Step by step with pictures
- Diagrams — Pictures are worth a thousand words
- Plain English– Do this, then this, because of this, and watch out for that
- Jobs — Rerun information, what to do if fails, what not to rerun
- Make a Hard Copy
Here is an example,
LoadEDIDataandValidate
Imports a file \\EDIFTP\CUSTOMHOLDRELEASE\EDI3502.log of EDI records that were sent from Gentran to Server A and Server B. It then validates that Server A and Server B have posted those records to their systems. Alerts are sent when something does not post with 15 minutes or record is in QUEUE status on Server B for more than 60 minutes. Server A and Server B data are kept separate on purpose, do not combine those tables. As of 3/9/2017 It also sends out a TXT message if count is >50 that have not been posted.
- Schedule
Runs Daily every 15 minutes between 2:16 am and 11:21 am. This corresponds to 15 minutes after Gentran begins and ends its daily processing.
- Steps
Executes SSIS Package EDI350Import.dtsx and executes 2 stored procedures; jobValidateEDIServerAEDI350ServerB and jobValidateEDI350
- Rerun
Can be rerun any time. Right Click on Agent job and Choose Start Job at step… There is only one.
Here are some other examples of rerun information (try to be as clear as possible)
- Rerun
Can be rerun prior to 4 pm. If run p0st 4 pm, you’ll have to manually change the date (@pdate) of the data being pulled. Always verify no partial data was brought into the table before rerunning clear out any data loaded.
- Rerun
Do not rerun. Load the data manually to Server X for any missing data and use date_billed as a key field for data pull
- Rerun
This job will fail if there is a duplicate XXX number. You’ll need to resolve the duplicate before you can successfully rerun. It can be rerun prior to 4 pm. If run post 4 pm you’ll have to manually change the date (@pdate) of the data being pulled. Always verify no partial data was brought into the table before rerunning. Clear out any data loaded.
Why Share My Knowledge?
Don’t try to build job security into what you do. I know many who worry about giving up knowledge to others. Having the sole ‘how to’ knowledge gives them a sense of job security. While this might be true to a point, it also locks you in your current position. Many who hoard their knowledge never advance because they find themselves invaluable in their current position. “We can’t move them because they are the only ones who know about such and such”. Why put yourself in that position? If you can’t ever be replaced, you also can’t move up.
As a Lone DBA, I find this runbook to be vital. It allows me to direct someone to the book and I can walk them through running anything I need in my absence. It allows me to take a vacation or a day off while giving others the tools to get things done.
Why It’s Important to Have a Hard Copy
I’ve found over the years that having some tangible steps in hand to follow and make notes on helps those who have to cover for me. It’s very easy for them to grab a book off my shelf and follow step 1, 2, and 3. It also gives them a place to take notes as they go through the steps. I can then use the notes to modify the documentation for better clarity.
If you don’t have a runbook, I highly suggest you take the time to make one. Keep in mind that a run book is only a helping guide. I automate as much error handling as possible and build it into the code to minimize the use of this in this book. However, in my opinion, the book is invaluable. It can allow someone else to cover for you. When the day comes that you win the lottery, you will have left everyone with great notes on how to run things.
Leave Yourself Breadcrumbs
I think every stored procedure, function, view, etc. should contain a block of code I refer to as a preamble. If yours doesn’t, I strongly recommend you start adding it.
Why? A preamble documents the use, need and changes to the code. It also leaves breadcrumbs as to how, why, and what you did. I don’t know about you, but I may code something and not have to change it for two years. When I do, I often wonder why I did something or who changed the code last. Working as a Lone DBA, leaving breadcrumbs was critical as I constantly jumped from task to task.
Here’s the template I use the preamble for all the code I write:
The preamble tells who wrote it, what it is, what it is called by, how to run it, and lists any changes. I find one of the most helpful items on this is the Run documentation. Here I place an exact run statement. It will show how the parameters should look and gives me a quick way to test it. This is an example of a completed preamble:
There are a million and one reasons why you should be adding something like this in your code. If you’re not doing it, just take a second and start. You’ll thank me for it later.
Managing Company Expectations
The better and faster you are at your job, the more work you receive. It’s important to manage your company’s expectations. When you are the Lone DBA, you cannot be expected to do the job of an army. On the flip side, you also don’t want to be the grumpy DBA that always complains about work or is always telling people no when they have something they need for you to do.
First you really need to get your employer to invest in some database monitoring or administration tools. The tools act as your arms or co-workers and help you be proactive. They eliminate the constant need to watch each server. A tool can monitor all your servers and kick actionable alerts to you allowing you time to work on other things. Yes, you will find some tools are expensive, especially, for an enterprise environment, but it is much cheaper than hiring more DBAs. The key is working efficiently so you can get as much work done as possible.
Using your boss as a ‘shield’ is another important tip. You need to have a relationship with your boss that allows them to funnel your workload to you when possible. It puts them in the position of having to say yes or no to a new project. It also allows you to dodge what I call ‘drive bys’. That’s when a member of management walks into your office and asks for something they need done right away. In these cases, I am able to let them know that I will look into their request and get back to them. Meanwhile, I can have my boss shield me and prioritize the work.
Visualizations of your workload are important. I always keep a whiteboard in my office and put a running list of all work that has been assigned to me on it. This way if someone walks into the office, they can easily see how busy I am or see that their project is already on my list and being worked on. I find that keeping my calendar full of my work helps manage things, too. People are less likely to ask for me when they see just how busy I am.
Work-Life Balance and Keeping Your Sanity
With all this work, how do you keep from being overwhelmed and stressed out? I believe it’s all about time management, prioritizing, and keeping a positive attitude. For example, while my daughters were at a dance or outside playing, I would utilize that time to work. I would also allot enough time in the morning schedule to fix issues before waking them up. This allowed for a less chaotic morning when problems arose that required immediate attention. If a problem occurred at night that could wait just a little bit longer for attention, I would wait until the girls went to bed. That way I was not impeding on their time.
But the most important piece is having flexible in your job. Working with your boss to make sure they allow you to work from home from time to time and for you leave work early to attend your kids’ activities. Lone DBAs give so many hours after normal work hours that you need to maintain a balance and work with your boss to gain an understanding of that. Making sure you copy your boss on emails or help desk tickets that you close in the wee hours or on weekends gives them visibility to that extra time. This, in turn, helps support the need for flexibility in your schedule. All of these things made a huge difference in keeping balance and sanity.
Summary - Why I Loved It
This just touches on some of the things I learned over the years that helped me survive and love about being a Lone DBA. To me, these not only apply to those that are in that same position but to anyone who has a heavy workload or can be on call. Being a Lone DBA can be daunting for some, but I absolutely loved it. I got to be the ‘go to’ person for all things SQL Server and got to make the rules. I was the final decision maker for anything related to SQL Server. Being the only one, you constantly work on a variety of things and are never bored. Lastly, it makes you very marketable as you look towards your next steps in your career. The vast hands-on knowledge you can glean from it really enhances your resume and can open many opportunities for you. I moved on this past year after 16 years of Lone DBA work and am now a consultant for Denny Cherry and Associates Consulting. Although I miss working on my own servers, I find it’s very rewarding to step in and help others, especially other Lone DBAs.