Multi threading in Dataset api

后端 未结 1 1082
心在旅途
心在旅途 2021-01-21 01:10

TL;DR: how to ensure that data is loaded in multi threaded manner when using Dataset api in tensorflow 0.1.4?

Previously I did something like this with my images in dis

1条回答
  •  迷失自我
    2021-01-21 01:26

    So it appears that the way to achieve this is as follows:

    dataset = dataset.map(parse_upscaler_corrector_batch, num_parallel_calls=12).prefetch(32).batch(self.ex_config.batch_size)
    

    If one changes num_parallel_calls=12 one can see that both network/hdd load and cpu load either spike or decrease.

    0 讨论(0)
提交回复
热议问题