@HoopAlchemy
Underrated video. A data engineer actually willing to give up the secrets ❤
@make725daily1
This video sets an exceptional benchmark! -- "Value the journey, for it shapes your path towards unprecedented accomplishments."
@MarcusJFloyd
Thanks, awesome video. I scaled by deploying Airflow on kubernetes and using the kubernetes executor so that jobs continue to run during deploys. We do have the problem of the dags being in the same repo as our image so running the kubernetes executor was a happy medium. I plan to move the dags to another repo and use a git-sync sidecar container to pull in dag updates at a scheduled interval
@Ahmed_Alnaqa
Thanks for this valuable info. I need ur feedback about Mage, the Airflow alternative, and ur feedback regarding scaling
@lcwanigasinghe8834
Thanks @Seattle data guy
@rembautimes8808
This is very good content, shares perspective based on specialised work experiences
@jerbear97
literally been trying to deploy Airflow in the past 3 days
@paul_devos
I so much trauma from trying to Deploy Airflow 3 separate times at 3 different orgs prior to the "Managed Airflow" era (AWS, Astronomer) that I can't even watch this video. Ultimately, I prefer to work in organizations that are generally smaller, more intimate and greater ownership of their own orchestration locally save for when they have data sets that might be agreed upon to be mission critical at the organizational level and ergo that data set moves to the "hub" where a data mesh like governance system may also take on those data sets in a "hub and spoke" like vibe.
@neuronqro
...how about looking at more "modern" alternatives to Airflow? Dagster, Prefect etc. What do you think about their deployment?
@rnzqt
Hey Ben, Watched loads of your videos, and in one of your older ones (or a comment afterwards) you mentioned udacity being s good resource for aspiring professionals, but wishing they had a data engineer nanodegree. Now that they do have (a couple of platform specific) de nanodegrees, is it something you have looked at? Potentially my company is willing to fund a course for me, hoping to move into the database from help desk support. Wondering if that would be a good course to get into. Thanks Olie
@SHARIFDERHEM
the first one looks important but tbh I didn't really get it
@ledinhanhtan
Because it is bad
@SeattleDataGuy