Once off loads, batches, real time streaming, if you have lots of data from lots of places and you want to be able to make use of it, that's when you would turn to one of these products. They are designed (along with Snowflake with whom BigQuery and Redshift are often compared) to ingest, store and analyse vast quantities of data for the purposes of reporting, business intelligence, dashboards, machine learning and more. Both products are - at their core - data warehousing solutions. What I present here are some of the key things I learned, in the hopes they may help someone else one day!Īs with all things I don't understand fully and before I got to solving the problem, I spent some time reading about AWS Redshift and Google BigQuery, both to reinforce the things I already knew, and discover some things I didn't. Strolling through the 'Dinosaurs of the Antarctic' exhibit at the Melbourne Museum.īut me? Well, I seem to spend my time with far too many instances of VSCode open on one laptop.Īnd so it was, that my weekend plans of being a vegetable on the couch and playing PlayStation 5 gave way to tinkering with AWS Redshift, Google BigQuery and a bunch of things in-between. Some people spent their Saturday's playing in the park. It can be a blessing or a curse, but I justify it by knowing that no matter what I'll always be learning something new. Of course - me being me (and before my brain could send a stop signal to my mouth because let's face it, at the time I knew very little about AWS Redshift and even less about Google BigQuery) I said sure thing! I'll have a think about it over the weekend, maybe I can offer some advice.Įxcept I could have promised, because I have a terrible habit that once I've been asked to work something out, or build something, or investigate something, I become absolutely consumed by whatever 'it' is, until I've solved it. Is that something, you know, something you know anything about?." "I need to get all the data out of my AWS Redshift cluster and into my shiny new Google BigQuery project. Late last week I had a friend come to me with a problem he wanted to solve:
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |