Why Small File Problems Quietly Drain Data Lake Performance and Budgets
Learn how small files affect data lake performance, increase cloud storage costs, and create query delays, along with practical techniques for file compaction and optimization.
Data lakes require efficient file management to maintain query performance and control storage costs. The Data Engineering Course in Chennai at FITA Academy covers data lake architecture, file optimization, data processing, and pipeline management, helping learners understand how to handle large datasets, reduce performance issues, and build scalable data solutions.
What the Small File Problem Really Is
A data lake sits on object storage such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage. Query engines like Spark, Trino, Presto, and Athena read data by listing files and opening each one. Analytical formats such as Parquet and ORC perform best when files are large enough to amortize the cost of opening them, reading metadata, and scanning column chunks. A good target is usually between 128 MB and 1 GB per file, depending on the engine.
The small file problem appears when a table is made of thousands or millions of files that are only a few kilobytes or megabytes each. The total data may be modest, but the engine has to touch every one of those files. Each file carries fixed overhead, and that overhead starts to dominate the actual work of reading data.
How Small Files Get Created
Small files are rarely a deliberate choice. They tend to appear as a side effect of otherwise sensible pipeline designs.
-
Streaming ingestion. Micro-batch and streaming jobs write data every few seconds or minutes. Each interval produces a new set of tiny files, and a day of ingestion can yield tens of thousands of them.
-
Over-partitioning. Partitioning by high cardinality columns, or by too fine a time grain such as hour plus customer, splits data into many partitions that each hold only a little data.
-
Excessive parallelism. A Spark job with many shuffle partitions writes one file per task per output partition. A job with 2,000 tasks writing to 365 daily partitions can produce hundreds of thousands of files in a single run.
-
Frequent incremental loads. Change data capture and upserts append small delta files continuously, and nothing merges them afterward.
The Performance Cost
Small files hurt performance in several compounding ways.
First, metadata operations become expensive. Listing a prefix with a million objects takes far longer than listing one with a thousand, and object stores limit how fast listings can run. Query planning can take longer than query execution.
Second, task scheduling overhead grows. Distributed engines typically assign files or file splits to tasks. When files are tiny, the engine launches a huge number of short tasks, and the time spent starting them outweighs the time spent reading data.
Third, columnar formats lose their advantages. Parquet relies on row groups, column statistics, and compression working over reasonable amounts of data. A tiny file has little to compress, weak statistics, and poor skipping potential, so the engine reads more bytes than it should.
Fourth, memory pressure rises. Drivers and coordinators must hold metadata for every file in memory. At large file counts, this causes out-of-memory errors and unstable jobs.
The Budget Cost
The financial side is just as real. Object storage charges per request as well as per gigabyte. Reading a million small files means a million GET requests, and listing them adds LIST requests on top. A table that costs very little to store can become costly to query because of request charges alone.
Compute costs follow. Slower queries keep clusters running longer, and serverless engines that bill by data scanned or time spent charge more for inefficient layouts. Teams often respond by scaling up clusters, which treats the symptom and raises spend further. Storage costs can also grow quietly, because many small files compress worse and carry more metadata overhead than the same data in fewer large files.
Fixing the Problem
The most effective strategy combines prevention at write time with maintenance after the fact.
Compaction. Periodically rewrite many small files into fewer large ones. Table formats make this easier. Delta Lake offers an optimize operation, Apache Iceberg provides rewrite data files procedures, and Apache Hudi supports clustering and compaction. These run as scheduled maintenance jobs, ideally on partitions that are no longer receiving new data.
Right-sized writes. Tune the number of output partitions before writing, using repartition or coalesce in Spark, so each task writes a healthy file. Adaptive query execution and auto-optimize features in managed platforms can handle some of this automatically.
Sensible partitioning. Choose partition columns with low to moderate cardinality, and avoid partitioning on a grain finer than the data volume justifies. A partition holding only a few megabytes usually signals that the scheme is too granular. Techniques like clustering, Z-ordering, or hidden partitioning in Iceberg can deliver data skipping without exploding the file count.
Batching streaming writes. Increase trigger intervals where latency allows, or land streaming data in a staging area and compact it into larger files on a schedule. Many teams use a two-tier approach, where fresh data stays in small files for low latency while older data is compacted for efficient analytics.
Cleanup of old files. Compaction leaves behind superseded files until they are expired. Regularly run snapshot expiration and vacuum operations so storage does not balloon, while keeping enough history for time travel and recovery.
Monitoring What You Cannot See
Because the problem builds slowly, it needs deliberate monitoring. Track file counts and average file size per table and partition, and alert when the average drops below a threshold. Watch query planning time as a share of total runtime, since a rising ratio is an early warning. Table format metadata makes these checks straightforward, as Iceberg and Delta both expose file statistics through system tables.
Small files can affect query performance, increase storage costs, and create operational challenges in data environments. DevOps Training in Chennai introduces learners to efficient data management, monitoring, automation, and infrastructure optimization practices that help maintain reliable systems, improve performance, and manage resources effectively.
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Angry
0
Sad
0
Wow
0