Source
r/aws•
ByteTheBit

Four months of 100% website uptime! Thanks AWS!

general aws

We used to host our website using a dedicated server with each web service in separate Docker containers. Although it worked, it wasn't elegant and we often had downtime. We're now using Elastic Beanstalk and Aurora through AWS.

When we initially moved over, our I/O charges with Aurora were horrendous. It forced us to tweak our code to decrease the database queries. As a result, lower charges and our web app runs quicker.

Even after multiple tweaks, AWS is still roughly twice the price of hosting the website on a dedicated server but I've now gained something invaluable which is better sleep.

Thought I'd share my experience in-case anybody else is thinking of switching from dedicated to cloud.

Posted Apr 3, 2020, 3:20 PM UTC

15453
FantasticBreakfast9•

Nice try AWS marketing dept

96
ByteTheBitOP•

Haha you got me! ;-)

11
lunarNex•

Yeah, my home web server has 4 months uptime too, do I get a cookie?

5
silverman_66•

Only if you accept the terms and policies by continuing to browse the site

6
Nikhil_M•

As you were already on Docker containers, did you consider something like fargate? You could save some cost by using fargate on spot.

24
ByteTheBitOP•

Have to admit, we never considered using Fargate. We're currently using multi-container docker with Beanstalk so might make sense switching to Fargate at some point. In terms of cost, I'm not really sure how much we'd save as we find that Aurora is the largest cost. We've managed to get our web app to run pretty efficiently which allows us to use small EC2 instances which makes things pretty cheap.

8
truechange•

Since you mentioned Aurora being your largest cost, do you mind sharing how much is it? How big is your database and how many users/traffic?

2
ByteTheBitOP•

No probs, so quick break down:

We have a few databases totalling about 10GB.

Roughly $50 / month in I/O

We have one db.r5.large and one db.t3.medium which acts as a reader / failover. This costs $1866 for a year all upfront (Roughly $155 / month).

Data transfer costs minimal as the Beanstalk webserver is in the same region.

Generally speaking, database side of things totals to about $200 USD / month, maybe tad more.

The Beanstalk side of things, we have three t3.small instances all on 1 year reserved. Including data transfer, ELB, prob totalling about $50 / month.

I'd say the whole thing costs us about $300 / month and that's with plenty of spare capacity to handle business growth.

The primary reason for having a db.r5.large was to be able to get the majority of our queries to hit cache in the RAM rather than the disk and as a result being faster and saving I/O cost. In theory we could save a lot by switching the db.r5.large to the db.t3.medium tier but I'm unsure how it'll effect the performance.

In the schemes of things, we've managed to get the costs down really low. Think the next step will be to try and tune things further to switch the db.r5.large to a db.t3.medium.

The web app itself usually has about 500 users using it at one time. Last time I looked, seem to remember roughly 30 or 40 HTTP requests per second. Not sure on exact DB usage on the top of my head tho!

13
truechange•

Thanks, I appreciate it.

2
jobe_br•

You could probably save a lot with DynamoDB. Wouldn’t be a trivial refactoring, though.

2
lotofstudy•

Never evaluated serverless aurora? Your db do stuff all day long?

2
piginapokie•

So higher costs but did you have a reader/failover in your previous iteration?

1
Rebles•

I’m also curious. I assumed for small operations like websites, you could run aurora for <$30/mo

-1
Jai_Cee•

You can use rds for that price but aurora has a minimum of medium instance so is more expensive than that. Aurora serverless could be < 30 though

2
seamustheseagull•

Cost-wise, Fargate is a good solution for terminating containers, not so much for ones with long uptime.

If you have batch processes with a long runtime, then it's the cheapest option between Lambda and Glue but for permanent containers like microservices or web servers, then your own EC2 clusters in ECS are much cheaper.

At least it was the last time I looked!

5
justin-8•

Depending when you looked they did reduce the prices a fair bit last year: https://aws.amazon.com/blogs/compute/aws-fargate-price-reduction-up-to-50/

Also savings plans can apply to fargate as well, which is another 20-50% off the pricing again.

1
packeteer•

Fargate gets expensive for any continuous load

3
pm_me_n_wecantalk•
  • How did you guys do database schema changes in aurora
  • Was there any downtime or production issue when schema changes were applied to aurora?
  • Did you guys have any production issues ever since you guys moved to AWS? If yes, what kind of issues were those?
  • Any pattern that you guys started to follow to optimize the queries? Or anything that you would like to share?
6
ByteTheBitOP•
  • When we transferred over from MySQL to Aurora, it was pretty much a simple mysqldump from the Docker container on the dedicated server, transferred it to a EC2 instance and then imported it directly back into Aurora. Fortunately no schema changes were required.
  • Initially we got Beanstalk up and running with our web app. For the database, we effectively went into read-only and prevented use of the control panel while we migrated the database over to Aurora. Once imported into Aurora, we edited the DNS and lastly shutdown the containers on the old server. We did a good couple of weeks of testing before we did the final migrate. It all went surprisingly smoothly!
  • I'd say the only problem we had is when we didn't configured a rate limited in nginx. There was kind of a (vulnerability?) in the web app where if someone was to upload a file a ton of time, it'd cause apache to lock up. We had a client which had a problem with their browser which effectively DoSed the file upload system we have. Since adding a simple rate limiter, those kinds of issues didn't happen again.
  • We used the Aurora's performance insights dashboard to find the poorly performing queries. We then went through our code to find where the query came from and then optimised the code. The majority of the time, it was just old legacy code which wasn't really built for the number of users using it. For example, we have a few graphs on the control panel which we tweaked which had a huge reduction in iops. We did this with possibly 10 or so queries. As a result, we saw pretty much a 10 fold decrease in I/O. Also meant that we could use a smaller Aurora instance. We have the advantage that we've built the control panel from the ground up and have the flexibility of changing anything. If we were to have a system built by a third party, can imagine it could of been far more costly moving to the cloud. When we were using a dedicated server, we definitely had performance in mind when programming the control panel but it certainly wasn't our top priority. Since moving to AWS, it's definitely become more important. As a result, it benefits the user experience and our wallets! Happy days!
8
pm_me_n_wecantalk•

Thanks. One more question.

For new features, how do you guys do schema changes ? I mean does Aurora has any out of box solution which provides safety from database being locked while some tables are being altered? Or did you guys use other approaches which were not AWS specific?

1
joelrwilliams1•

Aurora MySQL natively handles DDL (adding columns, etc) just like MySQL. It doesn't lock the whole database or even the table. The only negative is that it takes a while if you're modifying a table with lots of rows. You can also enable a Lab option called 'Fast DDL' which adds nullable columns at the end of table instantly.

More info here: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.FastDDL.html

2
ByteTheBitOP•

Sorry I'm not really familiar with anything Aurora specific to aid in the process of schema changes.

If we were to perform any major changes to the database which could cause some locking, we'd disable the part of the control panel which would be effected. Fortunately, we don't perform many large schema changes which could lock tables. For most features we add to the website, it usually means adding a new table to the DB.

1
WaltDare•

Another option to look at for file upload are Presigned URLs for S3. In short, you hand off load your file uploading to S3. Once the file has been uploaded you can then process it.

1
Peebo_Peebs•

I can help with this l, I admin a 6TB Aurora server

Schema changes. I use Percona tools online schema change.

Downtime. None

Issues. We’ve had a few random failover events due to AWS having issues with the storage cluster. Failover usually completes within 1 minute.

Optimize. Split your reads and writes properly. Use MySQL slow log to capture long running queries and optimize those. Also use bulk Inserts as much as possible. Consider table partitioning.

1
mazza77•

Good to hear thx for sharing ! Assuming in the past you were running on a single multi role servers (sopf) and now using the right architecture you have a fully scalable and HA Solution . In regards costs there are many things that you might be able to do to reduce costs and I would recommend after 6 months review the platform performance pattern and move your RDS instances to Reserved Instances.

:)

5
ByteTheBitOP•

Very true! We've recently had a 3 fold increase in users and things scaled really well. We're already on reserved instances with both EC2 and Aurora. When the renewal comes up, we'll probably review the code to try and make things more efficient. Amazing how much can be saved with a few code changes.

3
BadDoggie•

If I can suggest 1 thing - don’t wait for the renewal.. you’ll realise there’s more work than you thought and just decide to renew without changes. Get started now and have the work planned / tested so you can just switch. Maybe you work out that you can make use of current RIs with a new architecture. A lot of customers get hung up on getting every last second out of the RIs when they could have saved even more by dumping them, taking the hit, and moving to a much more efficient architecture. Of course, it depends on many things.

2
ByteTheBitOP•

Good advice!

1
stemid85•

How do you measure uptime? By HTTP return code or by connectivity? And how many times have you updated the website in that years time?

Impresses me more with frictionless updates than a static website being up for a year in AWS.

3
ByteTheBitOP•

We measure uptime by checking HTTP codes. We generally push updates out every couple of days.

The web app is an in-house built game server control panel which currently caters for roughly 10K users. The control panel's API also receives API calls from the 10K game servers, with status updates and such.

7
warren2650•

OP not having to get a call on his cell from the client is probably the measure ...... and a good one at that.

6
yeathatsmebro•

Congrats on choosing better sleep over cost. I work with Elastic Beanstalk and Aurora to handle millions of impressions daily for an ad network and with some projects related to betting with high amounts of sport events and live scores.

The cost is pretty high, although we use t2.medium for db and m4/m5 spot instances for workloads. The autoscaling is great, the node failure healing is gorgeous and not to say that eb cli makes a great job at deploying new versions directly from the pipeline.

If your app is stateless, you can deploy it with spot instances and no on demand capacity. I work with spot instances right from the moment they got into eb and i had no downtime. I just filled in enough instance types to make sure it won't run out.

3
ByteTheBitOP•

That’s interesting, sounds like you’ve got a good setup. Admittedly when starting out learning about AWS, I dismissed spot instances but it sounds like they would work well with beanstalk and auto scaling. I’ll take another look into it, cheers.

1
joelrwilliams1•

I'm always harping on devs that DB performance is directly related to I/O...the less, the better.

1
VIDGuide•

Don’t jinx it!

1
ByteTheBitOP•

I fear it’s too late! The damage is done ha

2
debarongh•

AWS is a reliable resource for every tech team

1
Jai_Cee•

Aurora is great but it is definitely expensive. For most production workloads the features are probably worth it though.

1
ByteTheBitOP•

Definitely agree that Aurora doesn’t always make sense. Initially we were considering using Aurora alongside a self hosted MySQL server on a single EC2 instance where we’d have our unimportant high I/O MySQL databases. We’d then have the Aurora for the mission critical databases. In the end we managed to optimised the poorly performing database queries enough where we could put everything on Aurora. It’s definitely the way to go as aside from the obvious benefits of Aurora, it would of been a pain having to manage that extra MySQL server.

2
FullSprinkles4•

You might be spending more on a monthly basis, but you also have to consider the money you'd have to spent on new hardware, licenses (if applicable), hosting cost, etc.

If you add that up, it won't be much expensive at all. 😊

Well done!

1
ByteTheBitOP•

Thanks! Also, something which isn’t usually talked about and that is the amount of time and energy that is required to maintain infrastructure and software. Going with a provider such as AWS, it’s kinda like hiring a database administrator. In a way, we save a ton of money in that way.

2
FullSprinkles4•

Absolutely!

1
ge0n1•

Those are rookie numbers even for on-prem.

0
ByteTheBitOP•

Worth noting that before optimisations, cost would of been roughly 10 times the cost of dedicated.

2
ge0n1•

Yeah, and the cost will come back when you hit scale too :/

1