My “Hero’s Journey” Through the Dark Underworld of WordPress Hosting

The following is an expanded and updated version of my presentation at WordCamp Salt Lake City 2017.

My girls love Moana. Especially when it first came to video and they could watch it every day… or two or three times a day if mom wasn’t feeling good or catching up on sleep from being up with baby brother the night before.

There’s this strange part of that movie where Moana follows Maui to a place under the ocean called “The Realm of Monsters.” It’s where monsters go after being killed. If you have younger kids, you know what I’m talking about. If you don’t have kids, it’s when the giant crab sings the song “Shiny.”

One common theme in myths, legends, and ancient religious writings, is where the hero visits the underworld, aka “afterlife” or “hell.” There they experience a symbolic or actual death for themselves or a loved one. Often through conquering a monster who is the Lord of the Underworld, they then re-emerge with their loved one, new knowledge and power, and/or some object to help them on their quest.

The film Moana clearly plays out this theme. She and Maui emerge triumphant from the realm of monsters, having defeated the giant crab by flipping it on its back and retrieving Maui’s magical hook.

In Return of the Jedi, Luke descends to the underworld of Jabba’s Palace. There, he’s able to bring his friend Han Solo back from a virtual death, defeat the Rancor, kill Jabba the Hut (lord of the criminal underworld), save the princess, and retrieve his lightsaber. All this happens before we see even one single, fuzzy, cute, unblinking Ewok…. Unless you believe in the “special editions,” in which case the Ewoks blink.

There are many movies, books, stories, mythologies, etc that all follow similar patterns.

Like Maui’s hook or Luke’s lightsaber, in these stories the hero often emerges from the underworld with newfound knowledge and/or a powerful object. Briefly, I’d like to take you on my own personal “Hero’s Journey” to the dark underworld of WordPress hosting. Along with it my quest of building a business on WordPress and hosting, with its own mythical monsters to slay, trials to be conquered, riddles to be solved, and ultimately new knowledge and weapons gained.

The Call to Adventure

In the early days of my business, Fiddler Online (now called wpXPRESS), I envisioned building websites for companies on WordPress, then charging a monthly fee to maintain, support, edit, update, and manage them. I got started with a few clients and wanted to grow.

I’d trusted someone to set me up with a very inexpensive VPS on unmanaged hosting. He was supposed to do the managing and ensure it all ran smoothly. I quickly learned you get what you pay for! Soon our members’ sites started going offline, usually all at once as the VPS crashed from one problem or another. It wasn’t just the server being down that caused problems. It was also corrupting data and other strange and scary things that I’d never encountered before, such as the body content of posts and pages being cut off, starting with the first special character that appeared in the content (a database issue).

These were dark and scary monsters to battle. I knew cPanel pretty well, but the APACHE stack that ran beneath it was mostly a mystery to me. So was database encoding and other advanced technical realms that all have an impact on the server, WordPress, and ultimately our clients’ businesses.

The monsters appeared, seemingly determined to kill my small and struggling business, shortly after the birth of our first child. I was frequently up all night working with “Todd” the server guy (we’ll call him) to try and get the websites back online. We’d vanquish one monster, rest a day or so, and find another had taken some sites down again.

After a few weeks of constant frustration, I said “enough” and signed up for a Hostgator reseller plan. This was back when they were an independent hosting company. On this new plan I could install as many cPanel’s as I needed and manage each separately! I thought it was wonderful and would solve all my problems. Hostgator transferred all our sites from the terrible VPS hosting we’d been on to their servers and I thought perhaps that was it!

I thought my quest was over. I’d fought the monsters and won.

Scary as it was, it had been relatively brief. I’d learned a lot, but felt like I’d been to the underworld and back!

The Illusion of Safety

Unfortunately, the wraiths had cast a spell over me, blinding me to the fact that they were still lurking in the shadows. When transferring our WordPress websites to our new account, Hostgator had copied each entire cPanel over from the VPS. This eliminated the underlying problems in the APACHE stack on that horrible VPS, but brought with it the fiends that had infiltrated cPanel and even our WordPress websites. To be clear: they weren’t infected with any kind of malware, but configuration problems, cruft, and who knows what else, caused some really bad results.

With my new spellbound, but misled confidence, I pushed our business forward. I brought on a business partner, Kurt as the sales guy. Later we acquired another website company. Through all that, we’d learned a lot, and doing okay for that stage, but still struggled financially as we bootstrapped this new business from nothing.

When we acquired this other business, they’d been running all their client’s sites on WordPress Multisite. It seemed like a great idea because of how it allowed us to manage all the websites in one place.

Despite the progress the business made, the monstrosities emerged again and this time with higher stakes since our business had grown and was now managing many more websites. These monsters emerged partly from what had been transferred over with cPanel, and the difficulties of running a large Multisite where each child site had its own theme, plus the weight of adding more and more websites and traffic to hosting that was really just shared hosting with WHM access and more control.

The Plunge Into the Underworld

In each hero’s journey, there’s often a wizard or goddess that helps guide and mentor the hero along his journey. I eventually made friends with a great guy we’ll call Sam. He ran his own hosting company with data centers and a great support staff. Sam was somewhat like Obi-Wan, Merlin, or Maui in these stories: he was my mentor and companion on my adventures into the underworld that followed. He made a great guide, because he’d been there before. Unfortunately, his own journey had never taken him to some of the deepest darkest places we would soon encounter.

So we moved to Sam’s hosting, where he kindly watched things closely and provided a server admin’s perspective and advice at a much lower price than he would have normally charged.

It wasn’t long before the demons, wraiths, monsters, and other ghouls started crawling from the darkness. It started with random downtime. Then followed strange limitations on websites. Next emails were not getting delivered. Then it was slowness. Now we were hacked… or were we? Suddenly all the contents of all pages and posts were cut off (we thought we killed this monster before)! Then email wasn’t being sent. Next our server is sending spam email. Now a hard drive is dying. It just kept going and going. Many of these demons were completely new to Sam and his very experienced team.

It didn’t take long before there were whisperings of “the Fiddler curse.” This curse referred to Fiddler Online the name of our company. The hosting support team joked that we were cursed. All kinds of issues arose that they’d never seen before, and with a frequency they’d never experienced either.

When a car crashed into the data center’s power regulation center during a freak storm, it cut off power from the normal power lines, as well as from their automatic backup power supply! It completely took the data center offline. The “Fiddler curse” was in full force.

Or put better, we were in the deepest, darkest, part of the underworld, locked in an all-out battle with the worst demons and wraiths it had to send against us.

We tried method after method to defeat the monsters. We tried rebuilding the server stacks. We tried various WordPress optimizations. We bought our own server and had it installed in their data center. We even tried a totally different Linux stack and something called Interworx, a cPanel alternative, that came with load balancing and real-time backup. But no matter how much money, time, and knowledge we threw at it, the issues continued. No matter how many monsters we slew, we were still losing the war.

The “Real World” Dragons

When I recently asked my wife Jill about that time of our lives, she said:

“From my perspective, it was hard to know when to throw in the towel and say ‘enough’ because it’s just not working. Getting the hosting sorted out totally changed the entire dynamic and perspective of doing our own business. Before that, it felt like we were building a dream on a crumbling foundation.”

And don’t get the wrong idea from this image. Jill was no damsel in distress. She’s a warrior too. I’d have never completed this quest without her there, fighting alongside me. She may have not fought the technical fiends, but there were plenty other monsters in the form of financial struggles, moves, and difficult situations that arose from my unavailability, our lack of money, and more.

This was all a lot like battling the Hydra of Lerna: we’d cut off one head, and two would grow back in its place. We’d take a breather for a week, and then here came the wraiths!

The stress of it all exacerbated a gallstone problem I had. Because of a misdiagnosis, I thought it was something there was no solution for. So even when I wasn’t up in the middle of the night battling monsters demons of the hosting underworld with Sam, I was often awake at night in massive amounts of pain as my gallbladder tried to pass gigantic gallstones. So even then, I’d be sleeping the next day when I needed to be designing websites or networking.

Throw in a healthy dose of anxiety and some intermittent depression and you get the picture. Instead of growing, our business stagnated. The quest through this dark underworld seemed to have no end in sight. It ruined vacations, stole away family time, punished me physically, hurt our client’s business, and was pushing myself, Jill, and our finances, to our very limits.

Death and Rebirth

It all seemed to come to a head when the pain of my gallstones became so intense I thought I was going to die. After 2 visits to the Emergency Room and 3 days in the hospital, I gave birth to this baby. They actually saw a larger one than this in the ultrasound beforehand, but my body had apparently broken it up before they removed my gallbladder. I returned home with real-world wounds that would turn into scars, and lighter by one gallbladder and several massive gallstones.

I left the hospital with a new lease on life. I felt like I’d been resurrected, fighting my way out of the underworld and back to the land of the living. It helped me open up to a completely different approach to hosting and allowed me to see that tiny speck of light that ended up being the doorway out of this underworld made up of the dark side of WordPress and hosting.

I was able to use my newfound perspective to find some awesome new weapons, and fight my way to that exit. Luckily for you, you can learn from my pain and battles with the underworld.

The Road Back: Simplify

Illumination was mine! Of the knowledge I gained, one key principle stood out among the rest: make things as simple as possible, while avoiding single points of failure on mission-critical systems. Multisite was great for managing all the websites at once, but if one had a problem, they all went down! The same with having a single server to run all our sites: if the server went down, we had the urgency of 30 or 40 clients (back then) all being negatively impacted at once.

We started by killing our Multisites and traditional hosting setups. Instead of a single Multisite with 1 database where a problem could take down all the sites in the Multisite, we moved to individual WordPress installs for each site. As we pulled each site out of the Multisite and migrated it to our new cloud hosting, we also checked the databases and files thoroughly to ensure they were clean, light, and that we eliminated any cruft that had built up in the database. We also stopped running email and DNS through our web servers. This effectively killed the demons that had moved with us in previous hosting migrations.

Here’s my recommended do’s and don’ts for any smaller businesses hosting and maintaining WordPress websites:

Don’t:

1. use shared hosting or hosting that uses cPanel
2. use Multisite (SPoF)
3. put all your sites on 1 server (SPoF)
4. use your web server as an email server
5. send WordPress transactional emails from your webserver
6. use your web server for DNS
7. use your hosting company for domain registration

Do:

1. use cloud hosting with WordPress-optimized stack and custom control panel
2. use a bulk-site-management tool
3. spread your sites across multiple servers
4. use an email suite (Hover, Zoho, G Suite, Office 365)
5. use a transactional email service (SendGrid, Postmark, MailGun)
6. use your registrar for DNS
7. use a different registrar for registering domains, than your hosting

Following these tips eliminates many single points of failure, simplifies things greatly, and gives you the tools and ability to go right to the source of the problem. Since all your important WordPress functionality isn’t in one place with one point of failure, you can go where the problem is.

For example, if a client isn’t receiving WooCommerce new order emails from their website, we can quickly go to SendGrid to see why that is and what needs to be done to fix it in an easy to use interface. Try that on a traditional APACHE/cPanel setup that sends your client’s company emails, WordPress emails, etc. all from one place.

Cloud Hosting

We eliminated cPanel and the normal Linux hosting stacks by moving to CloudWays, which has a nice WordPress setup that they run on top of a number of cloud hosting services such as Google, Amazon, Digital Ocean, and Vultr. They have their own in-house customized stack and management dashboard. CloudWays removed all the normal bloat and potential for problems that comes with it, and really took away most of the pain, hassle, and responsibility of the hosting part of WordPress and for quite cheap.

On CloudWays, instead of putting all 40+ sites on one server, we split them up, with about 15 – 18 sites on one small Digital Ocean (and later Vultr) server. This meant that if one site had issues, it wouldn’t take down all our other sites. And even if the issue was bad enough to affect the whole server, or the server had its own issues, only a small portion of our clients would be affected at once.

Email Accounts and Transactional Email

Additionally, we stopped running email through our servers. Part of simplifying is outsourcing to people/services who can just do it better than you (or that old hosting you’re clinging to because it’s cheap). I love that good WordPress hosts like CloudWays, Flywheel, and Kinsta have no options for you to do this or include built-in services like SendGrid. I slew a lot of email monsters. Using something like SendGrid or Postmark for WordPress and G Suite or Office 365 for email accounts, eliminates tons of headaches.

We set up WordPress’s emails to go through SendGrid and all our members’ email accounts we migrated to Hover, Google Apps, or Office 365.

Bulk-WordPress Management

In place of Multisite, we found MainWP and chose it over other options like InfiniteWP. It provided us the bulk-control of Multisite, but without the single point of failure issue. The upside is that it’s fairly inexpensive and runs on a WordPress install, so you control it on your hosting.

That’s kinda its downside as well. If something goes wrong, it’s on my team and I to run the problem down and fix it. Or we have to go through the cumbersome process of reporting it to the MainWP support team, then providing them access to both the dashboard site and an affected child site. Eventually we decided to move to ManageWP because it’s a hosted platform. That means when something goes wrong, much of the time, it’s on them to fix, and they have access to fix their own platform, plus the logs, etc from our sites.

MainWP:

• Inexpensive
• Runs on top of your WP install
• You maintain control
• Familiar interface
• Free to use the basics
• Lifetime extensions purchase option
• GPL licensed
• Great support
• Good community

ManageWP:

• More expensive
• Runs on their servers, so problems are largely theirs to deal with
• Less overall responsibility and time drain
• Free to use the basics
• Great support

Master of Two Worlds of WordPress

Fortunately today there are many awesome hosting options and bulk-management tools that simply weren’t available to me years ago when I started on this journey. After this last, final push, my team and I stood back and waited and rested, expecting more monsters. And occasionally one crawled out of it’s hole. But by and large the underworld was defeated and left far behind. Moving to cloud hosting and simplifying were finally the spell that broke the Fiddler curse and freed us from the underworld. Our business’s core service was stable and safe and running like it should be. We could start growing again! It was such a relief!

To reiterate some of the illumination gained on my journey: simplify your WordPress websites and hosting through offloading everything you can to experts who do it better, often for cheaper (if you properly calculate the value of your own time). Focus on your super-power whether it be design or development, or just creating solutions on WP with existing plugins and tools. If your super-power isn’t WordPress at all, you can outsource maintenance, content updates, backups and security, plugin and core updates, and much more to a company like wpXPRESS. I’ve built this company based on the knowledge and tools I learned on my quest.