how to Split data in 3 folds (train,validation,test) using ImageDataGenerator when data is in different directories of each class

问题

How do I split my data into 3 folds using ImageDataGenerator of Keras? ImageDataGenerator only gives validation_split argument so if I use it, I wont be having my test set for later purpose.

My data is in the form of

>input_data_dir
 >class_1_dir
  > image_1.png
  > image_2.png
 > class_2_dir
 > class_3_dir

回答1:

As you rightly mentioned, splitting the Data into 3 Folds is not possible in one line of code using Keras ImageDataGenerator.

Work around would be to store the Images corresponding to Test Data in a separate Folder and apply ImageDataGenerator, as shown below:

# Path to Training Directory
train_dir = 'Dogs_Vs_Cats_Small/train'

# Path to Test Directory
test_dir = 'Dogs_Vs_Cats_Small/test'

Train_Gen = ImageDataGenerator(1./255)
Test_Gen = ImageDataGenerator(1./255)


Train_Generator = Train_Gen.flow_from_directory(train_dir, target_size = (150,150), batch_size = 20, class_mode = 'binary')

Test_Generator = Test_Gen.flow_from_directory(test_dir, target_size = (150, 150), class_mode = 'binary', batch_size = 20)

Sample Code to extract some images from the Original Directory and place them in two separate folders, train and test, which may help you, is shown below:

import os, shutil

# Path to the directory where the original dataset was uncompressed
original_dataset_dir = 'Dogs_Vs_Cats'

# Directory where you’ll store your smaller dataset
base_dir = 'Dogs_Vs_Cats_Small2'

os.mkdir(base_dir)

# Directory for the training splits
train_dir = os.path.join(base_dir, 'train')
os.mkdir(train_dir)

# Directory for the test splits
test_dir = os.path.join(base_dir, 'test')
os.mkdir(test_dir)

# Directory with training cat pictures
train_cats_dir = os.path.join(train_dir, 'cats')
os.mkdir(train_cats_dir)

# Directory with training dog pictures
train_dogs_dir = os.path.join(train_dir, 'dogs')
os.mkdir(train_dogs_dir)

# Directory with Test Cat Pictures
test_cats_dir = os.path.join(test_dir, 'cats')
os.mkdir(test_cats_dir)

# Directory with Test Dog Pictures
test_dogs_dir = os.path.join(test_dir, 'dogs')
os.mkdir(test_dogs_dir)

# Copies the first 1,000 cat images to train_cats_dir. 
fnames = ['cat.{}.jpg'.format(i) for i in range(1000)]
for fname in fnames:
    src = os.path.join(original_dataset_dir, 'train', fname)
    dst = os.path.join(train_cats_dir, fname)
    shutil.copyfile(src, dst)

# Copies the next 500 cat images to test_cats_dir
fnames = ['cat.{}.jpg'.format(i) for i in range(1500, 2000)]
for fname in fnames:
    src = os.path.join(original_dataset_dir, 'train', fname)
    dst = os.path.join(test_cats_dir, fname)
    shutil.copyfile(src, dst)

# Copies the first 1,000 dog images to train_dogs_dir
fnames = ['dog.{}.jpg'.format(i) for i in range(1000)]
for fname in fnames:
    src = os.path.join(original_dataset_dir, 'train', fname)
    dst = os.path.join(train_dogs_dir, fname)
    shutil.copyfile(src, dst)

# Copies the next 500 dog images to test_dogs_dir
fnames = ['dog.{}.jpg'.format(i) for i in range(1500, 2000)]
for fname in fnames:
    src = os.path.join(original_dataset_dir, 'train', fname)
    dst = os.path.join(test_dogs_dir, fname)
    shutil.copyfile(src, dst)

# Sanity Check to ensure that Training, Validation and Test Folders have the expected number of images

print('Number of Cat Images in Training Directory is {}'.format(len(os.listdir(train_cats_dir))))
print('Number of Dog Images in Training Directory is {}'.format(len(os.listdir(train_dogs_dir))))
print('Number of Cat Images in Testing Directory is {}'.format(len(os.listdir(test_cats_dir))))
print('Number of Dog Images in Testing Directory is {}'.format(len(os.listdir(test_dogs_dir))))

Hope this helps.

来源：https://stackoverflow.com/questions/61535668/how-to-split-data-in-3-folds-train-validation-test-using-imagedatagenerator-wh

标签

tensorflow

keras

tensorflow2.0

tf.keras