r/aws•
TiDaN

DynamoDB should make it easier to export backups. Data Pipeline/EMR is very inefficient and cumbersome.

database

Why can't DynamoDB backups be made available for download? This is one of our biggest fiction point with DynamoDB:

At the moment, it is very inefficient, complicated and cumbersome to export all data from a DynamoDB table with Data Pipeline/EMR.

To add insult to injury, the current Data Pipeline template does not even support On-Demand Capacity Tables.. You're either forced to:

As it is, it feels DynamoDB tries its best to keep our data hostage, which is starting to negatively impact its attractiveness for our future projects.

Does anyone have other ways to achieve this? We've used dynamodump.py in the past, but again, it is very inefficient to scan the whole database.

Posted Oct 25, 2019, 4:52 PM UTC

2214
_thewayitis•

ugh... dealing with this right now.. Data Pipeline is the worst AWS Service I've come across so far.

10
Algorithmopolus•

Even worse than Cognito?

7
_thewayitis•

I was at least able to get Cognito to work after much frustration. Data Pipeline not so much.

1
data_addict•

You can grab the DynaamoDB stream with kinesis [firehose] and direct that into s3. It's the full history of your DynamoDB table too as opposed to a snapshot on a date. I could probably build this pipeline in a day (so I'm guessing one of your devs could too). You could pretty easily catagorize the logs too either while streaming with lambdas or after the fact with emr sourcing the s3 for nightly batch jobs.

Would this solution work?

7
TiDaNOP•

We're using streams for other things, but it's not applicable to our use case which needs a snapshot of the DB.

1
data_addict•

In an effort to understand better and I don't meant to sound rude - but why not? Are you creating the snapshots encase of a table failing, auditing, data mining, replicating, etc...? The stream has everything so it would have the table snapshotted up to any arbitrary point in time simply by taking the latest version of all records based on the event time it occurs. This solves a data mining, auditing, and other things when you're looking to work on the data (especially if you want a wide range of options to study the data). This wouldn't produce a table that's immediately available though if that's what you're looking for - you'd need to create emr jobs to build the table.

I'm guessing you're more interested in keeping the table aspect of dynamo in tact and not manufacturing it from logs. One idea I had that maybe gives you what you're looking for would be to write snapshots of DynamoDB to Redshift (as separate Redshift tables). Then if you need to send back to Dynamo you unload that entire table.

Here are 2 things I found. Unloading to redshift should be a cakewalk. Creating a job to load into Dynamo will take some manual work but it should be very straightforward.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/RedshiftforDynamoDB.html

https://stackoverflow.com/questions/42158572/migrating-data-from-amazon-redshift-into-dynamodb

3
TiDaNOP•

How can we begin to get some traction on this issue?

Paging u/jeffbarr

I've just tweeted about this but I barely use twitter so I've no followers. Retweet? https://twitter.com/70DanLab/status/1187774703232004097

3
chaospatterns•

AWS needs to work on their entire big data story. Glue, Data Pipeline, Athena, etc. Are all haphazard and don't really work that well for a serious data problem.

2
johnly81•

Side question, is there an easy way to copy/move tables between regions?

2
richardfan1126•

Are you talking about region? DynamoDB don't have AZ

2
johnly81•

Yep, sorry, I will fix my comment.

2
TiDaNOP•

I believe your only official options are Data Pipeline and DMS.

2
mwarkentin•

If you're creating a table from scratch you can use DDB Global Tables to support multi-region replication.

2
myroon5•

Keep in mind the regional availability of Global Tables: https://aws.amazon.com/dynamodb/pricing/on-demand/

1