Not a great way to wake up (west coast) or spend your first coffee break (east coast) to see your page.ly powered website unable to connect to the database.
The bad news: The database nodes were refusing connections for about 25 minutes this morning and page.ly sites were unreachable.
The good news: Our team was aware of the problem and remedied it almost immediately. However it took 20 minutes or so for the cache tables to rebuild before service was fully restored. It was a simple configuration limit ceiling that was reached, and that ceiling has been raised.
This evening around midnight CST we’ll be adding a big chunk of additional RAM to db1.
The great news: Page.ly 3.0 is nearing, and the new database setup is pretty amazing. The failover response time if a node becomes locked will be near instant, and the nodes themselves are much more powerful. It’s essentially a brand new setup top to bottom.
We are very sorry for this issue this morning. It sucks, we know it, and we have been tirelessly working to mitigate these issues on page.ly 2.0 as well as preparing 3.0 for your pleasure.
Thank you to all our customers for your trust and patience.
I would have appreciated a tweet to let us know what was up so that I could plan my day. Not only was the site down, my ability to work in my sites stopped as well. Didn’t hear a peep from you guys til about 40 minutes into the troubles.
Neil,
I get it and I am sorry. The team was working remotely and our first priority was restoring service. We’ll be arming our new support tech with twitter access shortly after he comes fully up to speed on our system.
I need to apologize too. I thought my tweets were going out but my hootsuite api connection was unavailable. I feel horrible and I’m sorry. This is not how we do business and I hope that our past performance proves this. We really care about providing a great service and we care about the success of our clients. We believe in solutions and not excuses, so please know that we are taking this matter seriously.
Good work on fixing quickly.
These things happen even with the best will/tech in the world!