Hadoop is not very efficient for storing a lot of smaller files.

If you need to access a lot of files nevertheless you can use HAR to get rid of the small file problem. Here are the steps that I did to get access to the files from PHP.

1. Copy the files from local filesystem to HDFS

Let hadoop create ONE single HAR file with name hadoop-api.har from the whole directory
/tmp/har/ (HDFS)

This command will start a MapReduce job that creates the HAR file without deleting the original small files in HDFS.

5. HAR file content (HDFS filesystem)

The HAR file is not really just one file but a directory with a couple of files. Let’s have a look to it with raw hdfs commands

6. Structure of the HAR file index (how to get access the single files)

hdfs dfs -cat /har/hadoop-api.har/_index
...snip...

--------
...snip...


Each row of the index file contains several space-separated columns:

• The url encoded path in the HAR file
• The type of the entry, i.e. file or dir
• The HDFS file which contains the content
• The offset in the HDFS content file
• The length of the file