kubeflow-pipelines

问题 I made two components using python functions and I am trying to pass data between them using files, but I am unable to do so. I want to calculate the sum and then send the answer to the other component using a file. Below is the partial code (The code works without the file passing). Please assist. # Define your components code as standalone python functions:====================== def add(a: float, b: float, f: comp.OutputTextFile(float)) -> NamedTuple( 'AddOutput', [ ('out', comp

问题 I made two components using python functions and I am trying to pass data between them using files, but I am unable to do so. I want to calculate the sum and then send the answer to the other component using a file. Below is the partial code (The code works without the file passing). Please assist. # Define your components code as standalone python functions:====================== def add(a: float, b: float, f: comp.OutputTextFile(float)) -> NamedTuple( 'AddOutput', [ ('out', comp

问题 What are the options to download .py files into the execution environment? In this example: class Preprocess(dsl.ContainerOp): def __init__(self, name, bucket, cutoff_year): super(Preprocess, self).__init__( name=name, # image needs to be a compile-time string image='gcr.io/<project>/<image-name>/cpu:v1', command=['python3', 'run_preprocess.py'], arguments=[ '--bucket', bucket, '--cutoff_year', cutoff_year, '--kfp' ], file_outputs={'blob-path': '/blob_path.txt'} ) run_preprocess.py file is

问题 I'm doing some experimentation with Kubeflow Pipelines and I'm interested in retrieving the run id to save along with some metadata about the pipeline execution. Is there any way I can do so from a component like a ContainerOp ? 回答1: I tried to do this using the Python's DSL but seems that isn't possible right now. The only option that I found is to use the method that they used in this sample code. You basically declare a string containing {{workflow.uid}} . It will be replaced with the

问题 While running kubeflow pipeline having code that uses tensorflow 2.0. below error is displayed at end of each epoch W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled Also, after some epochs, it does not show log and shows this error This step is in Failed state with this message: The node was low on resource: memory. Container main was using 100213872Ki, which exceeds its request of 0.

问题 I can create a static for loop using with dsl.ParallelFor([1,2,3]) as item: .... How can I use a container_op.output as an input to ParallelFor ? Assume the first container outputs an integer n , and then I want to run ParallelFor n times. Attempts like this does not work: container_op = ContainerOp(...) with dsl.ParallelFor(container_op.output) as item: .... I'm trying to stimulate a parallel python range(n) function. 回答1: The change to suport withItem (static looping) and withParams