kubeflow-pipelines

Issue when trying to pass data between Kubeflow components using files

若如初见. 提交于 2021-02-11 18:25:43
问题 I made two components using python functions and I am trying to pass data between them using files, but I am unable to do so. I want to calculate the sum and then send the answer to the other component using a file. Below is the partial code (The code works without the file passing). Please assist. # Define your components code as standalone python functions:====================== def add(a: float, b: float, f: comp.OutputTextFile(float)) -> NamedTuple( 'AddOutput', [ ('out', comp

Issue when trying to pass data between Kubeflow components using files

怎甘沉沦 提交于 2021-02-11 18:24:46
问题 I made two components using python functions and I am trying to pass data between them using files, but I am unable to do so. I want to calculate the sum and then send the answer to the other component using a file. Below is the partial code (The code works without the file passing). Please assist. # Define your components code as standalone python functions:====================== def add(a: float, b: float, f: comp.OutputTextFile(float)) -> NamedTuple( 'AddOutput', [ ('out', comp

dsl.ContainerOp with python

江枫思渺然 提交于 2021-01-05 10:38:55
问题 What are the options to download .py files into the execution environment? In this example: class Preprocess(dsl.ContainerOp): def __init__(self, name, bucket, cutoff_year): super(Preprocess, self).__init__( name=name, # image needs to be a compile-time string image='gcr.io/<project>/<image-name>/cpu:v1', command=['python3', 'run_preprocess.py'], arguments=[ '--bucket', bucket, '--cutoff_year', cutoff_year, '--kfp' ], file_outputs={'blob-path': '/blob_path.txt'} ) run_preprocess.py file is

How to get the id of the run from within a component?

人走茶凉 提交于 2020-12-31 06:51:30
问题 I'm doing some experimentation with Kubeflow Pipelines and I'm interested in retrieving the run id to save along with some metadata about the pipeline execution. Is there any way I can do so from a component like a ContainerOp ? 回答1: I tried to do this using the Python's DSL but seems that isn't possible right now. The only option that I found is to use the method that they used in this sample code. You basically declare a string containing {{workflow.uid}} . It will be replaced with the

Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled

非 Y 不嫁゛ 提交于 2020-03-19 05:59:52
问题 While running kubeflow pipeline having code that uses tensorflow 2.0. below error is displayed at end of each epoch W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled Also, after some epochs, it does not show log and shows this error This step is in Failed state with this message: The node was low on resource: memory. Container main was using 100213872Ki, which exceeds its request of 0.

kubeflow ParallelFor using the previous containerop output

徘徊边缘 提交于 2020-01-25 10:10:40
问题 I can create a static for loop using with dsl.ParallelFor([1,2,3]) as item: .... How can I use a container_op.output as an input to ParallelFor ? Assume the first container outputs an integer n , and then I want to run ParallelFor n times. Attempts like this does not work: container_op = ContainerOp(...) with dsl.ParallelFor(container_op.output) as item: .... I'm trying to stimulate a parallel python range(n) function. 回答1: The change to suport withItem (static looping) and withParams