Understanding Databricks Spark Write Behavior: The Mystery of _committed and _start Files

Discover why extra files like `_committed` and `_start` appear during Spark write operations in Databricks, and learn how to effectively manage them.
---
This video is based on the question https://stackoverflow.com/q/77749881/ asked by the user 'agaonsindhe' ( https://stackoverflow.com/u/1753273/ ) and on the answer https://stackoverflow.com/a/77749937/ provided by the user 'Mohamed Azarudeen Z' ( https://stackoverflow.com/u/22257235/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, comments, revision history etc. For example, the original title of the Question was: Databricks Spark Write Behavior - Files like _committed and _start

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding Databricks Spark Write Behavior: The Mystery of _committed and _start Files

When using Databricks Spark to write data to storage solutions like S3, many users encounter unexpected files—specifically, _committed and _start—concurrently with their data files. This behavior can cause confusion, especially for those who may not be utilizing Delta Lake in their implementations.

In this post, we'll explore why these files are generated, their purpose, and how you can work around their presence in your Databricks environment.

The Problem: What Are the _committed and _start Files?

During standard Spark write operations in Databricks, several auxiliary files may appear. These include:

_committed: Indicates that the write operation has been completed successfully.

_start: Typically signifies the beginning of a write operation.

If you're observing these files in your S3 bucket, don't be alarmed. They are part of the internal workings of Databricks and the Databricks File System (DBFS) when writing to external storage.

Why Are These Files Generated?

Transactional Nature of Write Operations

The creation of _committed and _start files relates directly to the transactional behavior of Databricks when it writes data. This mechanism ensures that your write operations are reliable and recoverable, essentially providing a safeguard against incomplete writes.

Databricks Runtime Features

The generation of these files can be influenced by:

The specific version of the Databricks Runtime you are using. Different versions may have slight variations in file management behavior.

The architecture of DBFS, particularly when interacting with external storage such as Amazon S3.

Can You Omit These Files?

Unfortunately, there is no straightforward way to prevent the creation of _committed and _start files. However, there are a couple of workarounds that you can consider:

Post-Processing Steps

After the data has been written, you could implement post-processing scripts to filter out these auxiliary files. Below is an example of how you can utilize the Databricks REST API to list and filter files:

[[See Video to Reveal this Text or Code Snippet]]

Upgrade Your Runtime

Consider upgrading your Databricks Runtime version. Newer releases may introduce optimizations and improvements regarding file handling. Always check the release notes to understand changes that could affect your operations.

Conclusion

The appearance of _committed and _start files during write operations in Databricks can initially seem perplexing, but understanding their purpose and behavior in relation to Spark's transactional processing can alleviate concerns. While there is no direct method to omit these files, implementing post-processing scripts and upgrading your runtime may provide relief and improved management.

By having this knowledge, you can navigate Databricks Spark write operations more confidently, ensuring a smoother data management experience.