Using ExecutorService with Java8 - Using List Partition

Executor - Star Wars

In the last post, where we used Executor service to move the files from one directory to another, I made some enhancement so that we could move all the files without having to face the limitation at all. The enhancement is List Partition.

So ArrayList in Java is provided with a function called subList() which partition the list based on the fromIndex and toIndex.

The task was to move the files from one directory to another using the multi-threads in Java8.

As part of the last post, where we were using a ConcurrentHashMap to keep the number of files picked so that others threads can be aware of the file already being consumed by other threads. But this logic has it limitation and you cannot escape the NoSuchFileException or NullPointerException explained in the last post.

So as part of new enhancement, I changed the ProcessFiles thread to work on a limited number of files. And when I say limited number of files, each thread will work on a segment of the list of files collected from the source directory. That way there will be no such limitations as faced before with the approach mentioned in the last post. 

This is where the List partition comes into picture. 

As part of Main class, now we will partition the file list into multiple list (in our case, 3 sub-list will be created from the main list of files). The ProcessFiles is now changed to accept a list of file as input parameter and we are using submit() of the ExcecutorService which can return a Future object.

List<List<File>> partitionedLists = partitionListOfFiles(fileList, 3);
for(List<File> files : partitionedLists) {
    Runnable worker = new ProcessFilesUsingListPartition(files);
    executorService.submit(worker);
}

The partition of list is done using the subList method of the List interface:

private static List<List<File>> partitionListOfFiles(List<File> files, final int breakInto) {
    List<List<File>> fileParts = new ArrayList<>();
    final int totalFiles = files.size();
    for (int i = 0; i < files.size(); i += breakInto) {
        fileParts.add(
                new ArrayList<>(files.subList(i, Math.min(totalFiles, i + breakInto)))
        );
    }
    return fileParts;
}

Here, lets say you have 10 files and you are partitioning the list into 2 parts, then breakInto parameter  will be 5. 

This will actually speed up the task of moving the files from one directory to another. You can vary the size of sub-list of files, based on the number of files you have. So that the program will use same amount of threads as the number of partitioned list of files.

You can find the entire code base here. Github-Link. Look for MainClassUsingListPartition.java and ProcessFilesUsingListPartition.java files.

Let me know your suggestions in the comment section or if you find some other way to implement.