{"id":494916,"date":"2024-06-05T16:16:26","date_gmt":"2024-06-05T16:16:26","guid":{"rendered":"https:\/\/dev-td-next.pantheonsite.io\/?page_id=494916"},"modified":"2025-04-09T16:05:04","modified_gmt":"2025-04-09T23:05:04","slug":"opensource","status":"publish","type":"page","link":"https:\/\/www.treasuredata.com\/opensource\/","title":{"rendered":"Open-Source CDP | Projects Invented by Our Engineers"},"content":{"rendered":"

Open-Source Contributions<\/h1>
Open source is in our DNA. Check out a list of open-source projects invented by Treasure Data engineers and projects we contribute to.<\/span>
<\/div><\/div>
\n\n<\/div><\/div><\/div><\/div>
{“colorVariant”:”DARK”,”layout”:”50\/50″,”vSplit”:”50\/50″,”stackPreference”:”mobile”,”stackOrder”:”headings”,”removeTopPadding”:false,”roundTopCorners”:false,”roundBottomCorners”:false,”eyebrow”:””,”showEyebrowOnMobile”:true,”showEyebrowOnTablet”:true,”showEyebrowOnDesktop”:true,”heading”:”Open-source pioneers”,”showHeadingOnMobile”:true,”showHeadingOnTablet”:true,”showHeadingOnDesktop”:true,”subheading”:” We know first-hand that open source makes software more accessible, developers more connected, and the world a little smaller. We believe in the open-source community, invest in moving projects forward, and welcome your contributions to our projects. \u00a0 <br><br>Over the last 13 years, we\u2019ve invented many key software components for modern data stacks. Treasure Data engineers pioneered and subsequently open-sourced these innovations, while including them in our customer data platform (CDP). \u00a0 <br><br>Highlights include an open data protocol adopted by Apple and thousands of other companies, creating one of the world\u2019s largest Hadoop user communities, and bringing Linux to the Fortune 100.”,”showSubheadingOnMobile”:true,”showSubheadingOnTablet”:true,”showSubheadingOnDesktop”:true,”ctaText”:””,”ctaUrl”:””,”showCTAOnMobile”:true,”showCTAOnTablet”:true,”showCTAOnDesktop”:true,”secondCTAText”:”Get a demo”,”secondCTAUrl”:””,”showSecondCTAOnMobile”:true,”showSecondCTAOnTablet”:true,”showSecondCTAOnDesktop”:true,”showInnerBlockOnDesktop”:true,”showInnerBlockOnTablet”:true,”showInnerBlockOnMobile”:true,”selectedOption”:”Standard”,”backgroundHorizontalAlign”:”center”,”backgroundVerticalAlign”:”center”,”backgroundFitMode”:”contain”,”backgroundMedia”:{“title”:{“type”:”string”,”default”:””}},”defaultContent”:”card”,”extraPadding”:”0″,”imageLoc”:”left”,”rowIndex”:0,”primaryButtonTargetBlank”:false,”secondaryButtonTargetBlank”:false}<\/div>

Open-source pioneers<\/h3> We know first-hand that open source makes software more accessible, developers more connected, and the world a little smaller. We believe in the open-source community, invest in moving projects forward, and welcome your contributions to our projects. \u00a0

Over the last 13 years, we\u2019ve invented many key software components for modern data stacks. Treasure Data engineers pioneered and subsequently open-sourced these innovations, while including them in our customer data platform (CDP). \u00a0

Highlights include an open data protocol adopted by Apple and thousands of other companies, creating one of the world\u2019s largest Hadoop user communities, and bringing Linux to the Fortune 100.<\/span>
<\/div><\/div><\/div><\/div><\/div>\n\n
\"\"<\/figure><\/div><\/div><\/div><\/div><\/div><\/div>

Open-source projects: Invented by Treasure Data engineers and hosted in our CDP<\/h2>
\"\"<\/figure><\/div>

Fluentd<\/h3>

With more than 600 plugins and native support for Docker and Kubernetes, Fluentd is one of the most popular tools for log management.<\/p>Learn more<\/a><\/div>\n\n

\"\"<\/figure><\/div>

Embulk<\/h3>

An open-source data loader to move massive data across storage systems, software services, file formats, and data centers.<\/p>Learn more<\/a><\/div>\n\n

\"\"<\/figure><\/div>

Apache Hivemall<\/h3>

Democratizes machine learning by bringing cutting-edge ML algorithms to the fingertips of SQL analysts.<\/p>Learn more<\/a><\/div>\n\n

\"\"<\/figure><\/div>

MessagePack<\/h3>

An efficient binary serialization format. It lets you exchange data among multiple languages like JSON, but it\u2019s faster and smaller.<\/p>Learn more<\/a><\/div>\n\n

\"\"<\/figure><\/div>

Digdag<\/h3>

A simple tool that helps you to build, run, schedule, and monitor complex pipelines of tasks. It handles dependency resolution so that tasks run in series or in parallel.<\/p>Learn more<\/a><\/div>\n\n

\"\"<\/figure><\/div>

Fluent Bit<\/h3>

An open-source data collector that collects data from different sources, unifies it and sends it to multiple destinations.<\/p>Learn more<\/a><\/div>\n\n

\"\"\/<\/figure><\/div>

wvlet<\/h3>

Wvlet is a cross-SQL\u00a0flow-style query language<\/a>\u00a0for functional data modeling and interactive data analysis.<\/em><\/p>Learn More<\/a><\/div>\n\n

<\/div>

pyenv<\/h3>

Lets you easily switch between multiple versions of Python.<\/p>Learn more<\/a><\/div>\n\n

<\/div>

snappy-java<\/h3>

A Java port of the snappy, a fast C++ compresser\/decompresser developed by Google.<\/p>Learn more<\/a><\/div>\n\n

<\/div>

httpclient<\/h3>

A Ruby HTTP client library.<\/p>Learn more<\/a><\/div><\/div><\/div><\/div><\/div>

Projects We Contribute To<\/h2>
\"Docker\"<\/figure><\/div>

Docker<\/h3>

Contributed the integration of Logging Driver for FluentD<\/p><\/div>\n\n

\"Kubernetes\"<\/figure><\/div>

Kubernetes<\/h3>

Contributed the integration of logging capabilities with Fluentd<\/p><\/div>\n\n

\"Ruby\"<\/figure><\/div>

Ruby<\/h3>

Several of our engineers are committers<\/p><\/div>\n\n

\"Ruby<\/figure><\/div>

Ruby on Rails<\/h3>

Various contributions, including MySQL support<\/p><\/div>\n\n

\"Apache<\/figure><\/div>

Apache Spark<\/h3>

Various contributions, including MLLib and Spark SQL<\/p><\/div>\n\n

\"Trino\"<\/figure><\/div>

Trino<\/h3>

Many bug fixes and organization of conferences and events for Presto and Trino.<\/p><\/div>\n\n

\"Keras\"<\/figure><\/div>

Keras<\/h3>

Various contributions, including pretrained frontier Deep Learning model, core tensor operations, weight conversion and benchmarking scripts.<\/p><\/div>\n\n

\"Apache<\/figure><\/div>

Apache Hive<\/h3>

Various contributions, including Apache Iceberg integration, logical and physical optimizers (CTE optimization or SharedWorkOptimizer), decision-making regarding language specifications.<\/p><\/div>\n\n

<\/div>

RecTools<\/h3>

Implemented new models such as ImplicitBPRWrapper, support for using hardware acceleration (GPUs), partial fitting on datasets for LightFM, minor bug fixes and Python version support.<\/p><\/div><\/div><\/div><\/div><\/div>

\n
\n
\n
\n
\n

Discover the capabilities and technologies powering Treasure Data CDP<\/h2>\n <\/div>\n
\n Explore the platform<\/a>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"

Open source makes software more accessible, developers more connected, and the world a little smaller. Learn more about Treasure Data CDP’s open-source contributions.<\/p>\n","protected":false},"author":17,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"footnotes":""},"class_list":["post-494916","page","type-page","status-publish","hentry"],"acf":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/pages\/494916","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/comments?post=494916"}],"version-history":[{"count":42,"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/pages\/494916\/revisions"}],"predecessor-version":[{"id":521474,"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/pages\/494916\/revisions\/521474"}],"wp:attachment":[{"href":"https:\/\/www.treasuredata.com\/wp-json\/wp\/v2\/media?parent=494916"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}