Introduction
The modern data landscape is witnessing a surge in the adoption of open table formats like Apache Iceberg. Driven by the desire for self-managed, warehouse-agnostic solutions to house large amounts of data, Iceberg has emerged as a powerful technology in this space. It offers many features you would typically only find in tools like Snowflake, while remaining open source and deployable with a simple server to host your catalog and just about any S3-compatible bucket to host your data.
This case study documents an experiment we conceived to achieve two goals: unify our partnerships with DigitalOcean and Snowflake (though our model would work with any cloud provider), and create a budget-friendly data lake model for smaller organizations by mixing and matching compute methods both in-house and with Snowflake. We have broken this article up into 3 parts that cover the vision for this architecture, the storage implementation, and the compute implementation.
Summary
We investigated the feasibility of building a budget-friendly data lake infrastructure that combines being centered around Apache Iceberg, deployed on DigitalOcean, and integrated with Snowflake's powerful analytical/BI capabilities. While established data warehouses like Snowflake offer incredible ease and speed to build complex data strategies, their cost structures can be challenging for smaller organizations with large data workloads. Conversely, building and maintaining custom data infrastructure from scratch presents a significant resource burden in terms of development time and expertise.
The primary goal is to significantly reduce the operational costs associated with traditional data lake architectures, opening the door for resource-constrained organizations such as startups, small businesses (particularly non-technical ones), and non-profits to implement sophisticated data strategies. We're excited about the potential to empower these organizations with capabilities previously out of reach, and to provide a pathway for them to seamlessly transition to a more Snowflake-centric architecture as they grow. This study also aims to pave the way for a more democratized approach to data management and AI by making open source data tooling accessible to a broader audience.
Experiment Goals and Hypotheses
Our central hypothesis is that an Iceberg-centric data lake architecture, deployed on DigitalOcean and integrated with Snowflake, can significantly reduce the cost of data operations compared to a Snowflake-only approach or a fully custom-built solution, all without sacrificing the benefits of either method. To test this, we established the following goals/tests:
- Feasibility of Iceberg on DigitalOcean: Successfully deploy an Apache Iceberg catalog on DigitalOcean, utilizing its S3-compatible Spaces object storage. This is expanded on in Part 2.
- Data Ingestion: Establish data ingestion pipelines using Airbyte, writing data to the Iceberg catalog residing in DigitalOcean. In the future, we're interested in other technologies such as Mage AI for this portion of the strategy.
- Snowflake Integration: Seamlessly connect Snowflake to the Iceberg catalog and perform queries against the data stored in DigitalOcean.
- In-House Compute: Explore the use of DigitalOcean's serverless Functions (equivalent of Lambdas in AWS) and in-house compute resources for data transformation/querying in order to reduce warehouse time in Snowflake. This is expanded on in Part 3.
- AI Integration: Integrate DigitalOcean's GenAI Platform with the Iceberg catalog, utilizing Function Routing to facilitate AI model interaction with the data.
To help visualize this, we created the following diagram to show how we pictured this experiment working:
Motivation and Rationale
We were driven by the following goals in conceiving this experiment:
- Democratizing Data Capabilities: Smaller organizations often delay implementing warehousing/data lake solution due to concerns about the high cost of existing solutions. Using Iceberg, we believe we can enable them to make data-driven decisions on par with larger enterprises. We believe this is particularly impactful for non-profits and startups looking to implement grander strategies from the ground up.
- Growth and Transition: This model is designed to grow with organizations, particularly startups. As a startup scales, they can gradually transition more workloads to Snowflake, leveraging its advanced features as their needs and budget evolve. Snowflake can query Iceberg directly, with no need for ETL to move data, so as time goes on we can simply build out more sophisticated solutions in Snowflake downstream from the Iceberg Tables.
- Bridging the Worlds of Software and Data: Our focus on data-intensive application development drives this project. We aim to create reusable infrastructure, managed with tools like Terraform, Ansible, and Helm, that allow us to boot this infrastructure quickly and easily, and most importantly reuse what we've built here for future projects. This unlocks flexible data operations and many possibilities for smaller apps and tools to be built off the same source of truth. In time, as our model matures, we hope to eventually open-source it as a StarterKit.
Methodology Overview
To perform this experiment, we adopted the following high-level methodology:
- Infrastructure Provisioning: Utilize Terraform and Ansible to automate the deployment of all necessary infrastructure components on DigitalOcean, including compute instances, object storage, application installation (Airbyte, the Iceberg catalog, etc), and networking configurations. In the end, we'd like to be able to deploy the entire experiment with a single terraform apply command.
- Iceberg Catalog Selection and Deployment: Evaluate and select a suitable Iceberg catalog implementation (e.g. Polaris, Nessie, Lakekeeper) compatible with DigitalOcean's infrastructure. Deploy the chosen catalog and configure it to use DigitalOcean Spaces.
- Data Pipeline Construction: Set up Airbyte to ingest data from various sources and write it to the Iceberg tables in the DigitalOcean-based catalog.
- Snowflake Integration: Configure Snowflake to connect to the Iceberg catalog using Snowflake's Iceberg Tables feature, enabling querying of data residing in DigitalOcean.
- Compute Strategy Implementation: Develop Functions to query data from Iceberg for the GenAI platform and to prove it's possible to offload compute to serverless Functions.
- AI Platform Integration: Connect DigitalOcean's GenAI platform to the Iceberg catalog via Functions, enabling AI models to interact with and derive insights from the data lake directly.
Expected Outcomes and Future Directions
We anticipate that this experiment will demonstrate the viability of a budget-friendly, Iceberg-centric data lake architecture on DigitalOcean. We expect to encounter challenges, particularly in areas like integrating Snowflake with a usually unsupported platform (vs AWS for example) and GenAI Platform integration. However, what journey is worth its rewards without challenges along the way?
The subsequent parts of this case study will delve into the technical details of the implementation, and the results of each step:
- Part 2: The Storage - A deep dive into Apache Iceberg and the process of setting it up on DigitalOcean, including the challenges we faced and the solutions implemented [link].
- Part 3: The Compute - An exploration of our hybrid compute strategy, detailing the integration of Snowflake, in-house compute, and the GenAI Platform, along with a conclusion to the experiment [link].
By documenting our journey and sharing our findings, we aim to contribute to the growing body of knowledge surrounding Iceberg and budget-friendly data management in general. We also hope to show potential clients of Polar Labs the capabilities and thought process we take to develop our solutions, and what we could do for you. We envision this work ultimately leading to a refined, open-source starter kit that simplifies the deployment of this architecture, further lowering the barrier to entry for organizations seeking to implement robust data strategies.
