问题
I'm trying to deploy a pipeline in GCP Data Fusion. I was initially working on the free account, but upgraded in order to increase quotas as recommended in the following question seen here.
However, I am still unclear based on the accepted answer as to what specific quota to increase in GCE to enable the pipeline to run. Could someone either provide more clarity in the above linked question or respond here to elaborate on what in the IAM Quotas needs to be increased to resolve the issue seen here:
com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Insufficient 'DISKS_TOTAL_GB' quota. Requested 3000.0, available 2048.0.
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:49) ~[na:na]
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) ~[na:na]
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) ~[na:na]
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) ~[na:na]
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) ~[na:na]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) ~[com.google.guava.guava-13.0.1.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) ~[na:na]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) ~[com.google.guava.guava-13.0.1.jar:na]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) ~[com.google.guava.guava-13.0.1.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) ~[com.google.guava.guava-13.0.1.jar:na]
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515) ~[na:na]
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490) ~[na:na]
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[na:na]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[na:na]
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[na:na]
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) ~[na:na]
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[na:na]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[na:na]
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[na:na]
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) ~[na:na]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510) ~[na:na]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) ~[na:na]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630) ~[na:na]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518) ~[na:na]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692) ~[na:na]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681) ~[na:na]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[na:na]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_232]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_232]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_232]
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Insufficient 'DISKS_TOTAL_GB' quota. Requested 3000.0, available 2048.0.
at io.grpc.Status.asRuntimeException(Status.java:533) ~[na:na]
回答1:
The specific quota related to DISKS_TOTAL_GB is the Persistent disk standard (GB) as you can see in the Disk quotas documentation.
You can edit this quota by region in the Cloud Console of your project by going to the IAM & admin page => Quotas and select only the metric Persistent Disk Standard (GB).
回答2:
- Deploy Pipeline.
- Press "Configure" and choose "Compute config".
- Press "Customize".
- Configure "Master Nodes" and "Worker Nodes" setting "Master Disk Size (GB)" and "Worker Disk size (GB)" to use less space.
- List item
Compute Config
来源：https://stackoverflow.com/questions/59738570/gcp-data-fusion-statusruntimeexception-invalid-argument-insufficient-disks-to