LOSF: Lots of Small Files movement

PI Name Zhengchun Liu
PI Institution Argonne National Laboratory
Collaborating ANL Division Data Science & Learning (DSL)
Project Description

Presently, many applications need to process lots of small files, which are organized using file system directories and frequently transferred between compute centres for different computing purposes. Moving data in a wide area network (WAN) environment has been studied extensively, especially for large files. However, many present data movement solutions are not efficient for handling lots of small files (LOSF). We examine the storage impact on the performance of LOSF movement. Furthermore, we improve the efficiency of moving small files by extending GridFTP to optimize the consumption of system resources.

Testbed

IT and Thing