Pachyderm

Pachyderm provides an open-source platform focused on data versioning and pipelines for Machine Learning Operations (MLOps). Founded in 2014 by Joe Doliner and Joey Zwicker, the San Francisco-based company was created to help data science teams automate and scale their machine learning lifecycle while ensuring reproducibility. The platform is built on Docker and Kubernetes, allowing it to run any language or library and deploy projects across various cloud platforms.

The company's core offering addresses the challenge of managing complex ML workflows where data, code, and models are constantly changing. It provides a data foundation with features like data-driven automation, petabyte-level scalability, and end-to-end reproducibility through immutable data lineage. This allows users to track every version of data and code, automatically trigger pipelines upon data changes, and debug processes efficiently. The business model included a free open-source Community Edition and a commercial Enterprise Edition with advanced features like enhanced security, access controls, and dedicated support. Clients range from biotech and healthcare to financial firms and intelligence agencies.

Before its acquisition, Pachyderm had raised approximately $28 million in funding from investors including M12 (Microsoft's venture fund), Y Combinator, and Benchmark. In January 2023, Hewlett Packard Enterprise (HPE) acquired Pachyderm to integrate its reproducible AI capabilities into HPE's AI-at-scale offerings. This acquisition followed an earlier investment from HPE's venture arm in 2022. Pachyderm now operates within HPE's HPC & AI Global Business Unit, where it is being integrated with other HPE AI solutions, such as Determined AI, to create a more comprehensive AI and machine learning development platform.

Keywords: data versioning, MLOps, data pipelines, machine learning, data lineage, reproducible AI, containerized pipelines, Kubernetes, data science, automation, unstructured data, petabyte scale, model training, open source, data provenance, AI infrastructure, data-driven automation, CI/CD for data, enterprise AI, big data analytics