Care.com operates as a dynamic marketplace offering a range of personalized care services, including nannies, elder care, and tutoring. The company is distinguished by its thorough screening process for both customers and service providers, prioritizing safety and reliability for all parties involved. As a mid-sized, fast-growing entity, Care.com was continuously broadening its scope. This expansion included introducing Care.com as a corporate employee benefit and partnering with established service provider networks. Additionally, the company was enhancing its platform with new AI-driven capabilities to further improve its services.

‍

Problem Statement

Care.com faced the necessity of overhauling their backend data platform to cater to the demands for real-time data access, accelerated reporting, improved monitoring, stricter governance, and enhanced resiliency. Their existing system, a patchwork of isolated silos, was a direct consequence of multiple acquisitions over the previous five years. This fragmented setup resulted in the absence of centralized governance, a unified data dictionary, and robust data quality controls. Additionally, there was a lack of standardization and normalization in data handling, as well as inconsistent patterns for sourcing data from production systems and partners. This necessitated a comprehensive transformation to streamline and upgrade their data infrastructure.

‍

Solution

I designed a flexible architecture that was not only capable of evolving with the business but also seamlessly integrated with existing systems. This design allowed for easy incorporation of new systems through reusable code and established patterns. Additionally, I developed a comprehensive set of governance rules and standards. These were crafted not only to gauge success effectively but also to aid compliance teams in ensuring that Care.com adhered to all relevant regulatory requirements.

‍

We selected Snowflake as our data storage solution, primarily due to its comprehensive range of data ingestion capabilities, a versatile and robust security model, and the ability to alter data views without modifying the original base tables. My architectural design leveraged Snowflake as a Data Lakehouse, integrating various connectors from Snowflake itself, Apache Airflow, Kafka by Confluent, and custom-built adapters.

While domain expertise regarding data sets remained with the original development teams, it was imperative that the transformation and normalization of this data not occur upstream. Therefore, we opted for regular streaming or batch exporting of raw data into the lake. Data cleansing was conducted within the data lake by data scientists and analysts, utilizing data dictionaries provided by each source team to create tables suitable for reporting. Tools like Tableau and Excel were employed for this purpose.

Adopting a 'Snowflake-first' approach as a data lake enabled us to store both structured and unstructured raw data for diverse applications, including Machine Learning. This setup allowed for the direct extraction of raw data from the lake, which could then be refined into training data sets. This strategy ensured efficiency and flexibility in managing and utilizing our data resources.

‍

Challenges

During a period of management transition at Care.com, we encountered a variety of challenges that were more related to interdepartmental dynamics than technology. While several engineering teams were enthusiastically engaged and open to changes, others exhibited a degree of resistance. This situation required spending the time to understand the diverse perspectives and concerns of each team, ultimately striving towards consensus and inclusion.

My architectural solution was designed with flexibility and adaptability at its core, allowing it to accommodate the specific needs and scenarios of each team. Through patient collaboration and a deep understanding of each team's unique context, we successfully onboarded every department and product engineering team, ensuring a cohesive and efficient transition across the company.

‍

Tools

• Miro
• Snowflake
• Tableau
• Kafka
• Python
• SQL
• JSON
• AWS (various)
• Apache Airflow