Data Thing Part 4: "The more I learn, the more I realize how much I don't know"
“The more I learn, the more I realize how much I don't know.” ― Albert Einstein
Albert Einstein, famed for the theory of relativity, devoted much of his later years trying to understand Microsoft documentation. Despite his genius, the complexities of Microsoft licensing eluded him. Tragically, he passed away in 1955, never having fully deciphered it.
As you may remember from the last blog, some nobble unthinkingly started our Fabric trial a few months ago so most of the free time has been wasted 😳. So I didn't do much last week as it kept threatening to delete everything unless we gave it money.
I did ask our Microsoft buddies to extend it, but it turns out that when you get down to the last few days you get a button pop up to add another 60 days yourself. Wopeeeeee!
Since I got the trial extended I've been trying to get my head round how it all works and crucially how much it might cost. I still can't confidently answer either of those things but here's what I've been working on so far...
A lovely Lakehouse
A Lakehouse sounds much nicer than a warehouse so lets have one of those. I like the idea that you can shove all sorts of files in it. This link explains the different data stores. The lakehouse lets you store any old toot. The warehouse has to be proper structured tables. The datamart sounds like a noddy version of the warehouse.
I decided to do this course:
Implement a Lakehouse with Microsoft Fabric which I recommend. It doesn't take too long and the interactive labs are good. The activities help to understand the different ways of getting data into the system.
There are pipelines, which are automated steps for copying and transforming data. A pipeline can copy from another source, then run code from a notebook to transform it. Or you can use a Dataflow (Gen2), which does the transformation on the fly, without the need for a copy first.
Also honorable mentions for:
Get started with Microsoft Fabric and
Microsoft Power BI Data Analyst
How should we structure it?
I have no idea lol. I'm not sure there is a Proper Way. This link describes a couple of different scenarios: Lakehouse end-to-end scenario: overview and architecture.
For now, I am imagining we have a lakehouse setup with all of our toot that gets ✨transformed✨ into nice tables. Then we create semantic models for the end users to connect to and create their reports. Further down the line, I'd like to have some POWER USERS 💪 in business areas that can work on the semantic models.
How much does it cost?
There are many options for licencing and some big prices. I hoped that as part of the trial I'd be able use this to estimate our SKU requirements: Install the Microsoft Fabric capacity metrics app.
But it keeps erroring 🤡 so I've got a ticket open to try and fix it.
Until then Reddit to the rescue, where the general consensus is start with F2 and hope for the best...
Starting to think about a business case
As this is gonna cost money I'll have to write a report for board. So far all I have are a bunch of headings:
- What is it
- Why should we use it instead of
- Doing nothing
- Using on prem SQL or Postgres databases for everything
- Any other options I think of by then
- How much will it cost
- What benefits will we get
- How it fits with the council vision etc
- What will we need to do and how long will it take
Other posts with tag "data thing":