Taylor A Murphy on LinkedIn: Talked with a data leader at a PE firm recently and just had to share my… (2024)

I talked with a prospect this week that shared they would be spending close to $75,000 on Fivetran a month if they just let it run and do its thing. They've been able to spend around 1/6th of that by... writing against the Fivetran API to resync the tables periodically via Airflow (which Fivetran lets you do for free).It feels wrong that as a customer you have to fight your tools in order to keep spend in order. Especially when we're talking about not *that* much data (< 10 TB). They were pleased with how Meltano was able to handle it in a self-hosted instance and we're working with them to move it to Arch where they'll pay less than you-know-who while getting top-notch professional support 😀

61

10 Comments

I read through Situational Awareness over the weekend https://lnkd.in/gC7WQggR It's an interesting look into a possible future of our world if you extrapolate a few trendlines. It's easy to be overwhelmed by all of the AI speculation, and I sometimes do. But I tried to take a realistic, pragmatic approach to this information given what I know and where I sit today. First, there is no economy, there is no world worth living in, without people. Say what you want about AI Sales tools, but people buy from people and humans will always want to interact with other humans. Second, even if we do have superintelligent AI and fully capable robots, these aren't going to be evenly distributed for a long time. And I doubt we'll get to the point any time soon where a Superintelligent AI robot will be as capable in all dimensions as the amazing humans I know. It's just going to take way longer than you think to get there.This leaves plenty of time, space, and opportunity for many people to build incredible businesses that provide real value to people. In fact, perhaps the opportunities will *increase* thanks to Jevon's Paradox https://lnkd.in/gFHFTiCJ So how do you proceed? For one, have an abundance mindset. Don't give into AI doomerism. Nobody really knows how this will all play out and there are a lot of good things that can come from this. Second, lean into creativity and learning. For all it's amazing powers, current AI just isn't curious at all. It's not like a human that can take lived experience and go down rabbit holes and follow its passions to create something unique and new. Learn how to use these AI tools to serve your innate abilities.Lastly, just work hard solving real problems for other humans. Give a sh*t about other people. AI will not save us and there's plenty of people that need help. All the compute in the world will not solve all the problems we have. So yeah. I don't know if that's the best strategy but it's the one I'm using at the moment. 𝗕𝗲 𝗽𝗼𝘀𝗶𝘁𝗶𝘃𝗲, 𝗯𝗲 𝗵𝘂𝗺𝗮𝗻, 𝘄𝗼𝗿𝗸 𝗵𝗮𝗿𝗱. What's your strategy?

8

3 Comments

One of the many reasons I love open source is the use cases it unlocks. I just learned about the Tuva project (https://lnkd.in/eaTeskgx) which aims to make it easy for more organizations to work with healthcare data. The core of this project is a set of dbt models (https://lnkd.in/erbBfp5z). Back when I worked in healthcare I would have loved something like this! We had Redshift but this was a pre-dbt world and we had to build things like this by hand (uphill! both ways!)The person who introduced me to this is a big fan of Meltano and was able to get Meltano + dbt+ Tuva up and running on a local postgres instance incredibly quick. It genuinely makes me smile to know we're a part of the journey in unlocking use cases like this all over the world. We use Meltano under the hood with Arch and will continue investing in our free and open source tools. It's a core part of our mission and who we are as a company 💪

33

5 Comments

Just had a great conversation with a data engineer who just joined a healthcare startup. He shared some of their stack and almost sheepishly admitted that their data warehouse was "just Postgres". I cut him off right there and told him that there is nothing wrong with using Postgres as a data warehouse. Plain vanilla postgres with no extensions likely works for the majority of an organization's data needs. The person I was talking to almost felt a sense of relief because he thought he was doing something wrong by using it!At GitLab, our very first data warehouse was Postgres (GCP Cloud SQL). It wasn't until we brought Snowplow into the product that we moved to Snowflake. Had there been the option to use extensions like Hydra (https://lnkd.in/edx5DFJh) we may not have needed to switch...Arch comes out of the box with a Postgres database (with the Hydra columnar extension - all powered by Tembo) because we believe it's a *fantastic* database that works for the majority of organizations. We have support for other cloud data warehouses if a company needs to BYODB, but for those that don't, Postgres is the best choice for them. Data engineers: it's okay to use PostgreSQL! It's a great database! Don't be embarrassed 😀

211

46 Comments

As we've been building Arch, I've become more excited about working with a specific type of company: those that could benefit from the outputs of a great data team, are unlikely to ever have a data team, and may not even know what a data team should be doing. There are a ton of these companies out in the world. My bet is there are more companies that don't have a data team than those that do. Think of all the random businesses in your home town. What percentage of those do you think are doing big revenue numbers (>$10M/yr) and don't have a data team? Probably most of them.One strategy when faced with this is to try and convince these companies they need a data team. Show them the benefits of what a data team can do and then sell them on tools built for data teams. That's not our strategy. Instead we're saying to these companies that they deserve great analytics but that it shouldn't require spending $250k+/yr - all to hire a single expert and provide them with a hodgepodge of tools.What these companies need is very simple: * well-defined metrics * a maintainable and scalable system to support calculating their metrics * the understanding of what has happened and is currently happening with their metricsWith that foundation they can then turn to what *may* happen with their metrics. This shouldn't cost over a quarter million a year! It really, really shouldn't.Nerding out about all things data and data teams is fun. But if we can get out of our own bubble and biases, then there's a huge opportunity to drive a ton of value for organizations that need it. Special thanks to all the people who have unintentionally influenced my thinking on all this 😃

67

3 Comments

I'm excited we can start offering this again! The Meltano community is amazing and we're excited to help companies build awesome data integration capabilities internally. If you need guaranteed support, technical training, or implementation support - please reach out! https://lnkd.in/gjyPXb9v

14