This is the fifth and final installment of a mulitpart part series where we aim to share with you some of the technical aspects of what powers the Managed WordPress Hosting system we developed here at page.ly, how we started, the recent server improvements and a bit on the things to come. [Part 1] [Part 2] [Part 3] [Part 4]

Out with the old

So we have deployed a new server stack and continue to tune it for optimal performance and scalability. What’s left to do you ask?

Our business dashboard, has performed it’s duties well all these years but it is time to be retired. This handy little system I coded in Code Igniter (as is 99% of page.ly) and has pretty much allowed us to perform nearly any task needed without ever hitting the command line. We can pull up accounts, orders, domains and DNS. Manage affiliates, resellers, and track our growth.

Here is a sample page from the current dashboard we are retiring. Just a note, the CC number is only the last 4 digits and pulled from Authorize.net with an API call, we never sit on this data.

It ain’t pretty but it does the job. We’ll show you what we are replacing it with a few pixels down.

From a system management and app deployment angle the current setup works, but is slow, and error prone in it’s old age. It worked exceptionally well when managing 500 domains, and adding 2-3 a day.  It is really straining when managing thousands of domains and adding 10-20 a day.

A new backend

Remember up thru page.ly v2 we were utilizing the Plesk SOAP API to manage vhost deployments on specific web nodes, and then with page.ly v3 we bashed together some scripts to replace Plesk. Those are temporary fixes as we have some very exciting things in the works.

The whole deployment and management system is due for a rewrite, so that is exactly what we are doing. This rewrite includes:

• Reseller management (deployment and management of accounts on behalf of resellers)
• Account management (account ownership, logins, external services)
• Domain management (DNS, aliases, swapping)
• Billing system (order intake, invoicing, payment handling, receipts, notifications)
• DNS management (already at Dynect)
• Domain registration
• App provisioning system (installing new apps, file system, database)
• Reporting/logging (nagois, munin, puppet)
• Affiliate management
• Infrastructure management (auto scaling, auto cloning, config management)
• Product management (plans, packages, upgrades, custom items)
• Support Integration (tying zendesk into our dashboard)
• and everything else…

As you can see there is much more that goes into a functional and successful WordPress Hosting business then just installing the WordPress app.

We are happy to say we are about 60% through the rewrite of all page.ly systems as I write this post, and we are aiming for a Feb20th launch date. The entire system is built upon a Code Igniter powered REST API.

One API to rule them all.

Okay so it is actually more than one. We took the philosophy of making everything in page.ly a RESTful service. So we have a API specifically for the installer and server/system processes, and another for account/user related functions, and another specifically for job queuing, etc. This approach allows us to:

• Decouple code
• Maintain narrowly defined unit tests
• Craft lightweight REST client apps and GUI’s
• Document it and make it accessible to partners.

Partners will gain deeper and more meaningful integration with page.ly

We are pretty excited about this. We do a few things exceptionally well, deploy and manage WordPress. We dont do themes, or plugins, or marketing, or SEO. People that do those things may also want to include hosting as an offering but really don’t want to mess with the technical stuff.

By exposing many of our core provisioning and account management API endpoints to authorized partners they can easily add page.ly managed hosting to their checkout process, on their site. Additionally more developer savvy partners may ping the API to generate reports and lists of their current customers, manage aspects of those customers, and manage their integration points into page.ly such as defining which themes, plugins, or content is pre-installed on their new customer sites. All these things can also be done via our new Brains GUI we’ll show you below if the partner is less then code savvy.

This API is also the underpinnings of a entirely new page.ly service we are working on for Q2.

The new Brains

So that old clunky dashboard is being retired and the backend behind it as well. We wanted a new GUI that was light, responsive, and worked well with our new API service methodology. Enter Bootstrap by Twitter. Our Code Igniter client apps serve up the data and Bootstrap makes it easy to interact with.

What’s ahead for us

Through Q1 of 2012 we’ll be focusing on continued refinements and improvements to the system architecture we deployed with page.ly v3 and deploying the new API and Brains of page.ly v4. Come Q2 and through the summer we will be expanding our offerings with new  products and packages aimed at various markets yet staying close to our core competency of managed Software-as-a-Service.

By the time this goes to press I will likely be a new first-time father. What a trip this journey has been. From failed concept in 2006, to a second go in 2009, to insane growth and success in 2010, to server madness in 2011, to a bright future for our bootstrapped company and parenthood in 2012.

Why are we bootstapping page.ly?

In 2010 we had some dialog with a much larger and well know hosting company that wanted to acquire page.ly. In 2011 the same thing again with another company. Also in 2011 we flirted with the idea of raising funds. In all cases we came back to the same conclusion: We like what we do, we like doing it our way, and we like doing it on our terms. We have a vision of the type of company page.ly will grow-up to be. That vision does not include maximizing profit with gimmicks at the expense of customers to appease a board or make our earn-out. That vision does not include a ‘win at all costs’ mindset that pushes integrity and professionalism aside.

Our vision is to be the best at what we do, provide a real service, and treat our customers and employees as we would wish to be treated, with fairness and honesty. These values are compatible with a healthy profit, not exclusionary to it. Could we achieve this with outside capital, sure. Are we going to waste time trying to convince a VC to play along, probably not. Someday perhaps. I am not anti VC, but I am anti getting into bed with someone that may not share our set of values.

Conclusion

I hope you enjoyed and got some value out of this series of posts. We tried not to get too technical or bore you with all the business details of our story. We look forward to what’s next and are immensely proud of what we have done this far.  And finally we hope you will consider joining us on our journey and give Page.ly a try for your WordPress hosting needs. Cheers.

This is the fourth installment of a mulitpart part series where we aim to share with you some of the technical aspects of what powers the Managed WordPress Hosting system we developed here at page.ly, how we started, the recent server improvements and a bit on the things to come. [Part 1] [Part 2] [Part 3]

Time to go big.

When we started this journey many years ago, we made system choices fitting at the time. We grew, and grew some more, and kept on growing and the old system was straining. In the summer of 2011 we began work on page.ly 3.0. Our most advanced Managed WordPress Hosting system to date. A system designed for reliability, security, scalability and most of all performance. It’s all the hawtness of page.ly, just faster and more reliable.

The build process was long and unfortunately not without it’s problems. The goals of page.ly3 were:

• It has to scale easily
• It has to be fast
• It has to be secure
• It has to be reliable

That’s it. SImple order right?

Strategy

Our plan was to develop page.ly3 next to pagely2. Build a mansion next to a condo, and then come moving day, everyone just packs up and moves nextdoor. However it turned out that we moved and improved things in pieces.

We started by consulting some of the best and brightest the web community has to offer. I spent some hours consulting with my good friend Joe Stump (Ex lead architect at digg) among others with the ‘scaling’ knowledge. We developed a simple plan. Use small nodes, to do specific tasks, and scale out in layers.

We decided we needed dedicated machines for mysql, web, varnish, memcache, and the like. We needed load balancers out front and we wanted robust and fast storage behind it. We also had to get rid of Plesk. While it was fine for a smaller page.ly, it simply did not scale and frankly is a festering turd, like all control panels in our opinion.

Phase 1

The first thing we did was add more database servers. 2 new slaves coming off the master. We then added WordPress’s HyperDB system wide to segment those reads and writes. We went from a single 8core 17gb machine to 3, 4core 10gb machines. The slaves happily process 700-800 queries a second at 0.20 load.

Phase 2

We then setup the new storage. Instead of running storage local to each machine and working out a rsync strategy we went with NFS. At first we went with a solid 700gb disk but then decided it was better to serve 10x70gb disks up to the webnodes by NFS4. Firehost also upgraded all our storage to Fibre. The disk speeds are crazy fast, and are still in the 90mb/s range serving over NFS. We do all hard file operations directly on the storage nodes which speeds things up as well.

Phase 3

Zeus load balancers were setup out front in a failover configuration, and dual Varnish nodes directly behind it. This gives us a lot of flexibility allowing us to segment and direct traffic around or through specific varnish nodes directly to some or all of the web nodes. It’s very fault tolerant as if both varnish nodes fall over, everything goes gracefully to the web nodes. Varnish is of course a web caching system, serving pages and static assets in 20-50ms a 4x improvement over pagely2.

Phase 4

We setup some other nodes to handle a few management tasks like puppet, memcache, cron jobs, backups, logging, etc.

We also had to gut all that code that relied on Plesk, since it is dead to us. It is not terribly pretty right now but we bashed together a temporary set of scripts to install and manage our system. This is just a band-aid for now, and we’ll give you a preview of the new system  in the final chapter of this series.

Finally we direct certain types of traffic to specific nodes tuned for the task. We dont blanket all web nodes with all traffic. As an example wp-cron.php commands hammer a pool of servers all day, allowing others to work without those nasty processes sapping cpu.

Here is a snapshot of what the page.ly 3.0 server topology consists of. Below we will talk about each part of it.

1. All traffic of course comes from the world famous interwebs, a series of tubes that connect you to LOLcats wikipedia.
2. That traffic hits the Firehost security layer. This is like fort knox for data. Thousands of nefarious attacks are blocked here daily.
3. Good traffic then travels over our 10gb internal wan and hits our twin Zeus load balancers. These bad boys are the latest and greatest in awesome. So much so Rackspace uses them like a mo-fo but calls them Cloud Load Balancers, which is total BS cause Zeus lived above the clouds like a Boss.
   • Internal DB read requests are load balanced to the DB Slave pool.
   • Web requests are seamlessly routed around Varnish if needed to land directly at the web pool for normal processing.
4. Requests are then routed to our Varnish layer where cached content (images, js, css, and full page output) is served directly back to the user at crazy fast speeds.
5. Backend requests from Varnish are handled by our pool of Litespeed web servers. These are well equipped but dummy machines that do nothing but process PHP and serve up web pages. The nice thing here is that we can scale out, adding as many as we need for as long as we need. Why not nginx? We are testing some options and benchmarking a variety of webservers for future use.
6. The Database queries come from the web servers through HyperDB, a WordPress database class designed and used by the folks at WordPress.com. Reads & Writes go to separate database pools tuned for their particular task. Again here we can scale out as many slaves as we need with replicating masters.
7. All Files are served from our latest generation SAN array which we can scale damn near infinitely in size. This large volume is mounted as many smaller NFS shares to provide optimum performance while maintaining a global shared file system. When we feel like it, an upgrade to SSD disks is a click away.
8. The entire system is controlled by Puppet with manifests to maintain server configs and handle server cloning.
9. Our Utility node does all manner of tasks like taking out the garbage, running Nagios, and keeping my vodka glass filled.
10. The Page.ly node, is frankly where the magic happens regarding our Managed WordPress system. Server architecture is cool and all, but without badass software it’s just 1′s and 0′s.
11. Finally, every node is HA and the entire system is backed up every evening with snapshots. We can restore any node, any data, at any time.

Currently we vary between 16-24 nodes running depending on load.

El Pollo Diablo

El Pollo diablo is essentially the same architecture stack. Typically we route the traffic thru the load balancers, to a dedicated varnish node and web server pool specific to that client. Sometimes the client runs their own mysql pool as well, sometimes they take advantage of our mysql cluster. The flexibility allows customers true ‘dedicated’ resources but also use of our redundant and failover safe systems as needed. Customers have been very pleased with the hybrid approach allowing them flexibility and a ‘managed’ WordPress solution.

We did not share every piece of the puzzle as there are a few blackops systems in place to add that little extra awesome that makes Page.ly special. Sure it cost us more to roll out than most people’s annual salaries but we spent it out of love, you guys deserve it.

This is not rocket science as robust and redundant server stacks are the norm for large scale websites. However we are happy that now we have implemented a system that can keep up with our insane growth AND serve not 1 but thousands of websites at speed and scale.

Phase 5: The transition

Moving clients to the new systen was not easy. It spanned over 3 months from October 2011 to just after Christmas. We moved 1 of the old plesk servers at a time, rsyncing all that data over, popping ip’s off those nodes and onto the load balancers. We did the last client move after Christmas and finally shut down those old nodes. After every block, we would tune and test, uncover bad mojo and cure it.

October-December 2011 were probably the hardest days our company has faced, and hopefully ever will. We had hard time getting the balance just right. Redoing our storage setup, and Varnish configs at least 4x while serving live traffic. It’s hard to plan for what you dont know. An example of which is a nasty bug in PHP itself.. that essentially lstats a file 4x-8x EVERY single time it requests it. On local storage you probably would not notice the hit, over NFS with the millions of files we access millions of times a day.. it was literally shredding the SAN, and sending IO on the web nodes through the roof. This is just 1 of the handful of things we found at scale, as in they only cropped up under load.

We had to work through each on of these, again under live conditions. As once we moved a block of customers, there was no going back.

We saw some client attrition as we worked through the issues. We win them with a solid offering and stellar support, but even the most understanding customer is only going to hang around so long if you cant keep the system up for more than 10 hours at time. We wish them well, and we know they will be back. They always come back as the other solutions out there just dont stack up (their words, not ours).

Even with ALL that.. investing over $100k in labor and new hardware, losing some nice customers, and slogging through a painful 3 month transition, we still grew. Every month since we started has been our best month to date and it show now signs of letting up.

While we were at it, We hired 4 employees in 2011, more than doubled our customer base and Page.ly got a new site design as well.

Getting race ready

I enjoyed racing cars back when it seemed I had free time. You always put in a couple warm up laps to heat the tires before rolling on the throttle. The promise of page.ly3 is nearly realized but we still have some tuning to do.

We are investigating alternate storage systems and are looking at everything from gluster to netapp, to GFS. As Joe said, “Disk is always the bottleneck”. We are constantly tuning and tweaking the varnish nodes to maximize hitrate and delving deep in to WordPress itself to tune it for our system. At this point we have essentially forked WordPress. We maintain a set of patches we apply to new versions and test before rolling out to the customers.

We are also looking at alternate caching systems around the database, and object cache to make full use of memcache, APC is also running on a very narrow fileset that could be expanded. So we still have a bit of work to do, but we are making progress in days now instead of months.

So the servers are nice and fast now. What about page.ly itself? You know the app. We’ll give you a preview in the final part of this series.

###

Look for part 5 of this series where we give you a preview of the new application and API we are developing.

This is the third installment of a mulitpart part series where we aim to share with you some of the technical aspects of what powers the Managed WordPress Hosting system we developed here at page.ly, how we started, the recent server improvements and a bit on the things to come. [Part 1] [Part 2]

Page.ly finds a new home and begins to scale.

Mid 2010 and Page.ly was really starting to take off. It was time to think about scaling and we also needed a new hosting partner that would help us manage the hardware. We were also very aware of security concerns and sought out a partner who could help.

Firehost.com met all the criteria, I knew the CEO, they were small (at the time) and nimble like us, and security was their core offering. We made the choice and over 1 evening migrated our single server over Firehost.

Firehost has proven to be such an asset to us that we cannot recommend them enough. Apparently others agree as 1 of the first things our early competitors did when setting up was beeline to Firehost as well.

Page.ly also saw a couple design refreshes during this time.

Better security.

The move to Firehost  added multiple levels of security to our system. It was a huge addition to our offering and a value add to our customers as well. We pivoted our marketing a bit to stress this new benefit and have not strayed from it since. We all saw how the big econo-hosts faired in the summer of 2010 as they all got railed by malware. There is no profit-margin on $3.99 hosting to include security I guess.

New Security features of v2 included:

• Managed Redundant Firewall Protection
• Managed Redundant Web Application FireWalls
• Managed Redundant DoS/DDoS Mitigation
• Multi-level Intrusion Prevention and Detection
• Real time virus scanning
• Real time malware scanning

More Servers

Now here I have to say in hindsight we made a mistake. Rather then push forward to the stack we will describe in the next part of this series, I elected to just scale out with additional Plesk servers.  It worked okay, but it also postponed a ‘proper’ solution. The customers were coming in hot, and we were still a company of just 2 founders. This decision to just replicate our single server solution cost us time and money, as it made it harder to migrate later.

So over the course of the next year we went from 1 server to 4 servers. All large monolithic beasts. 8cores/8-16GB of RAM, each one a single point of failure. We had to monkey with DNS, assigning specific clients to specific IP’s/nodes etc. We still used Plesk to manage things and that made load balancing near impossible.

We at least made the wise choice at this time to offload MySQL to a single dedicated machine. Here again though a 4core 18gb ram beast, another single point of failure. #fail

Hindsight is always 20/20.

Under the hood of Page.ly v2

• All nodes HA (High Availability)
• 4 web servers running Plesk & Litespeed, doing DNS as well
• Single Mysql Node
• Used the Plesk SOAP API for vhost configs
• Started experimenting with WordPress caching plugins
• Opcode Caching

So we were moving in the right direction. We had better security, we had enough power to serve thousands of domains and things were moving right along. Speed wise this setup was not bad. Initial hit (not full page load) times were about 200-400ms. If you were a page.ly customer back then you felt some of the pain though of our single server per user setup as we would have to take down an entire node when we needed to run maintenance on it.

In 2010 we also:

• Created our Reseller system that allowed folks to white label page.ly
• Launched our Affiliate system
• Deployed our Vertical Platform that allowed select partners deeper integration with custom reseller packages
• Kicked email off the system and moved it over to an outside provider.
• Had a PowerUp system for a while packaging bundles of themes and plugins from out providers. We moved away from even though it is was fairly successful.
• Flew pass 1000 customers.

In late 2010, we knew we had to scale again.. well scale better. So we embarked on what would be an all consuming project known as page.ly3.

Community

WordPress has a community, and one filled with egos. We ran into a few of these ego’s along the way. Initially a big part of our strategy was engaging the WordPress community. Unfortunately we were soured a bit on the whole thing. For the better we turned inward to improve our product and pursued alternative channels and marketing strategies to recruit new customers which have been very successful.

The market heats up

In 2010 we saw the arrival of a handful of what most would refer to as competitors, it was inevitable. We did the hard part proving the space had legs and there was revenue to be made. And no good idea goes un-copied for long. We were also starting to make a name for ourselves in the WordPress community and that helped quite a bit to get the word out.

A couple interesting points regarding these new arrivals. At SXSW in 2010 I went to the WordPress BBQ at a co-working space in Austin where I handed out a few shirts and were talking to folks about what Page.ly was. One fellow I distinctly remember talking to must have really been listening, a few months later he was a co-founder of a competing company. Around the time this company was launching their other co-founder solicited us about using our technology to power their new offering rather than ‘re-inventing’ the wheel. I was amicable to the idea (fits squarely with our collaboration over competition philosophy) and agreed to a phone conversation which never took place. They decided to roll their own and off they went.

One of the other new players was not happy merely re-factoring our idea, but went so far as to be heavily ‘inspired’ by our marketing to the point I had to have a private conversation with them about the overt similarities between our website copy.

At the end of the day, we did not then and we still do not see any of these chaps as competitors at all. We work from a mindset that with 30+ million WordPress powered sites out there the space is deep enough to support 5-10 page.ly’s and we rather pull clients over from the econo-hosts like bluehost, mt, and godaddy and provide them with a better product.

###

Look for Part 4 in this series where go into detail on what powers page.ly today.

Unfortunately this year we (Sally and I) won’t be in attendance. It’s our home town and last year we really enjoyed helping make WordCamp PHX 2011 probably the best WC ever. Our two newest employees will likely make a showing, so if you run into them, say hello.

Instead of WordCamp, Sally and I will be home with our brand new baby son who is due to arrive any day now.

Enjoy your WordCamp and we’ll see you next year.

This is the second of a mulitpart part series where we aim to share with you some of the technical aspects of what powers the Managed WordPress Hosting system we developed here at page.ly, how we started, the recent server improvements and a bit on the things to come. [Part 1]

Building a better mousetrap.

Page.ly was launched in 2009 after a complete rewrite of the WordPress hosting system prototype we talked about in Part1 of this series. That idea worked, but I think it was ultimately ahead of it’s time and suffered from lack of effective marketing and audience identification.

In late 2009 after many months of re-tooling we went back to the market with an improved managed WordPress hosting solution.

The new system was based on the core logic of the prototype, that being an automated way to deploy WordPress powered websites. But this system had some major improvements which we developed here first, and some of which have since have become the basis for all competitive offerings that have come after.

Page.ly 1 introduced:

• Automatic WordPress Core updates
• Automatic WordPress Plugin updates
• Automatic Nightly Backups
• An integrated “Hosting Panel” via a custom developed WordPress plugin that communicates with our core system, wrapping the vital hosting administration tasks like DNS, Email, and Billing management without the bloat of a full Cpanel, or Plesk interface.
• Our one-of-a-kind Whitescreen Eliminator

Initially we emphasized the Choose your Theme aspect and allowed customers to pre-select from a library of chosen WordPress themes. We later downplayed this in Page.ly2 as it turned out to be a conversion killer.  People would stress over which theme to choose and abandon the cart.

Under the Hood of Page.ly 1

Page.ly1 was run on the same infrastructure as the prototype we described earlier, a single LAMP box utilzing the Plesk SOAP API. Except we were running PHP 5 by then. Really not that exciting.

Off to the races

Everything worked like a charm, customers were signing up (a modest 100 or so by end of 2009) and people for the most part were ‘getting’ what we were trying to do. Make your WordPress experience better by automating the technical tasks.

The feedback we got was interesting, some people loved it, some people hated it. There are those types that always think “oh I could do X, why would I pay for it” and find no greater pleasure then trolling. Those that got it, got it. We were offering a better WordPress experience via managed automation.

By early 2010 we saw the need to begin scaling as customer count was approaching 500 and our hosting provider at the time was not very helpful. Nothing against them (theplanet) but they were a very hands off provider, basically making sure the server was plugged in and thats about it.

We went in search of a new hosting provider, that would also address our concerns for Security and one that offered more a of ‘managed’ offering that could assist with some of the tuning and setup required.

Thats when we found Firehost.

###

Look for Part 3 in this series where we dive into Page.ly v2, scaling, and security.

This is the first of a mulitpart part series where we aim to share with you some of the technical aspects of what powers the Managed WordPress Hosting system we developed here at page.ly, how we started, the recent server improvements and a bit on the things to come.

Backstory

Way back in the internet dark ages; like pre-twitter Sally and I owned and operated a web consultancy shop in Phoenix Arizona. We served all manner of clients with your typical fare of web related services like site design (remember dreamweaver?), SEO, light programming and the like. Around late 2005 we started using WordPress to deploy client sites on. Even in the early days WordPress v1.2 showed promise of what it has since become.

Sally saw the time we were putting into custom sites and always wondered why it took so long. What she said in her words were : “I thought you guys just clicked a few buttons and added some images and the website appeared, why can’t it be that easy”. Well she was a prophet.

As our business grew and we started going upmarket, charging 10′s of thousands of dollars vs hundreds of dollars we were having to say no to our early clients that were coming back for more work. One day Sally overheard me on the phone telling the 3rd or 4th client that day “Sorry we just cannot do this work at that rate any more” and she said “Josh you said no to a few thousand dollars today, we are leaving money on the table”.

Identifying the opportunity.

So there it was, an opportunity staring us in the face. How can we still serve these lower-paying customers profitably? We had to automate, give them the power to build the website themselves. Page.ly v0.0 was born, or as was known then Flare9.com.

Some rough wireframes from the early prototype

We did our market research and saw no one else in the space. As in, no one else had anything looking like an automated software-as-a-service approach to managing WordPress.org sites.  There were consultants clicking buttons for people, but flare9 was the first of it’s kind WordPress Software-as-a-Service.

Under the hood of v0.0.

The first crack at our managed WordPress hosting solution was code written by our employee at the time Joshua Eichorn (now Director of Engineering at Stumbleupon and one of the most talented programmers I have ever worked with). He bootstrapped a slick little system that would intake and bill orders, send them over a socket connection to our ‘installer’ library which then ran around moving files, creating databases and restarting apache.

At the time we were using the Plesk hosting panel on all our web servers and we tapped into the Plesk SOAP API to handle much of the vhost and db config for us. The server was a simple dual core xeon hosted at EV1 (now defunct). A run down of the basics.

• 1 Server doing everything (PHP, MySql, Email, etc.)
• Simple code library written in the then brand new Code Igniter framework
• Used the Plesk SOAP API for vhost configs
• There were no ‘caching’ plugins
• There was no http acceleration like Varnish or Nginx
• No firewalls
• We did not use memcache, or APC
• Just php4, mysql, and apache
• No automatic WordPress or Plugin upgrades
• Real basic file and database backups.

It works, now how do we find customers?

The provisioning system worked pretty well and over the course of 2-3 months in late 2006 we had 40 or 50 paying customers. The “hosting” part was easy.

Ultimately the marketing was hard as this was pre-social media and the heyday of Google PPC. And of course no one knew what WordPress was. We simply had a real difficult time getting anyone to notice outside of our local business community. After a couple months we turned off active signups, and moved on to other things.

Try, Try again.

So in 2009 with the economy in a skid and me being really tired of doing client work we decided to revisit this idea, we even had 70-80% of those old customers still faithfully paying their monthly fees. We started rewriting the code base and chose a better name and got to work on what would become Page.ly.

###

Look for Part 2 in this series where we dive into Page.ly v1 and v2.

1. Customer attempts to signup for our service.
2. The CC transaction triggers a fraud warning, and transaction is denied.
3. Customer contacts us, states the legitimate nature of their ‘self’.
4. We respond that we follow the fraud warning and their is not much we can do.
5. They ask for a free account anyways
6. We decline, and apologize.
7. They rant.

Our system at page.ly is 100% automated, and when working correctly everyone is happy. We make choices on how to run our business, as I am sure people out their make choices on how to run theirs. We recently lowered our fraud score threshold as we saw an uptick on bogus credit card attempts. If you wait long enough you’ll see all sorts of bad folks game your payment form trying out stolen credit card numbers. Some perfectly well meaning patrons may be caught up in the drag net. It happens, and it sucks, but we make the choice to play it conservative on credit card transactions.

We never see a users credit card, it is passed directly to the processor over a secure connection when supplied, tokenized and that’s it. The best we can ever get is the last 4 of the CC number, that even then we have to make an API request for. Even further, we do not have the means in our internal system to take/run a credit card manually, it does not fit with our automation model. If a transaction fails the multiple fraud screens, then 99.999% of the time we did our job limiting credit card fraud with a .001% of denying a valid customer.

So what’s the alternative? Most company’s just take the money and let the bank and poor chap who’s card got yanked sort it out, and maybe, but not likely the company see’s a chargeback. That old “as long as I got mine” attitude that has made the world economy so awesome of late /sarcasm.

Well here at page.ly we operate with a “greater good” philosophy. Allow me to explain, Lot’s of companies out there play it loose with customer trust and credit cards, what do they care? As long as they get paid right. The issuing bank eats the fraudulent charge and passes it back on to the consumers at large as a cost of doing business. Everyone pays higher fees and interest because of this. If we loose 1 of every 500 customers because we actively try to prevent fraud, so be it. The greater good prevented 499 others the aggravation of contesting a charge with their bank, and perhaps worse yet cleaning up after identity theft.

So dude can rant, and call us silly. I am confident he is a real person, with a real card, and our filters may have been overly aggressive.  And hey maybe we could have pulled up our external merchant account that does not have the same fraud prevention measures, ran the card manually and reversed into the signup process. But then again, maybe we can just keep doing what works for us and our thousands of other customers seem to appreciate.

Please welcome Bernardo el Panadero who is a straight up systems administrator and has already been making changes to our server stack to improve performance.  We expect great things to come as Bernardo whips up some custom recipes for awesome.

In addition please welcome El Mecánico del Robot aka Mech whom is leading the charge on the rebuild (pagely4) of our management and provisioning systems as well as working on our new product.

And we say goodbye to El Chupacabra Luchador. He had great talent, but unfortunately was unable to ship code in the time frame we require. It happens, we all move on.

What’s Next?

Page.ly3 is rolled out and we are fine tuning it a bit to get the best performance out of it. We defined page.ly3 as an infrastructure upgrade, moving from 4 monolithic servers to 16 secure cloud servers in a redundant and fault tolerant setup. (Very long post coming soon).

Page.ly4 is an upgrade to our management app, or all the logic that controls how sites are provisioned and upgraded, as well an overhaul to our internal dashboards and such. Bernardo and Mech are working on page.ly4 now and expect a full roll out expected sometime before Q2 2012.

But wait, theres more.

We have an entirely new page.ly product (codename: Mothra) under development, one that we started tinkering with last fall. We feel it will serve as an avenue to empower commercial plugin and theme authors. We’ll have more details soon.

Look for our post soon on the nitty gritty of page.ly3, the roll out of page.ly4, and details about our new product.

Host: sftp.pagely.com
User: <your existing user>
Pass: <your existing pass>
Port: 22 

Purchase receipts for sftp details are now sent in 2 parts for a little added security. Other receipts containing sensitive information will follow suit shortly.

Also: We have a nice blog post we have been working on queued up to post soon that goes into great detail discussing the new page.ly3 architecture, the improvements it brings, the tuning left to do on it, and the not so awesome move over to it that has been the cause of much frustration all around. We look forward to sharing and thank you for your patience.

All but 6 sites are running on WP3.3 and the page.ly3 system.

A belated Merry Christmas and a Happy new years to all of you!