dask.distributed

I have 4 machines, M1, M2, M3, and M4. The scheduler, client, worker runs on M1. I've put a csv file in M1. Rest of the machines are workers. When I run the program with read_csv file in dask. It gives me Error, file not found When one of your workers tries to load the CSV, it will not be able to find it, because it is not present on that local disc. This should not be a surprise. You can get around this in a number of ways: copy the file to every worker; this is obviously wasteful in terms of disc space, but the easiest to achieve place the file on a networked filesystem (NFS mount, gluster,

I want to find an alternative of pandas.dataframe.sort_value function in dask. I came through set_index , but it would sort on a single column. How can I sort multiple columns of Dask data frame? So far Dask does not seem to support sorting by multiple columns. However, making a new column that concatenates the values of the sorted columns may be a usable work-around. d['new_column'] = d.apply(lambda r: str([r.col1,r.col2]), axis=1) d = d.set_index('new_column') d = d.map_partitions(lambda x: x.sort_index()) Edit: The above works if you want to sort by two strings. I recommend creating integer