google-compute-engine

GCP-Storage: do files appear before upload is complete

一世执手 提交于 2020-06-16 03:48:07
问题 I want to transfer files into a VM whenever there is a new file added to storage, the problem is that i want the transfer to be done only when the upload is complete So my question is : Do files appear even when the upload is still going on ? which means if I build a program that looks for new files every second, would it transfer the files from gcs to VM even if the upload is incomplete or the transfer would start whenever the upload is complete and not while it is uploading ? 回答1: Google

google compute engine tool gcloud is exceptionally slow

寵の児 提交于 2020-06-10 08:05:29
问题 I tried downloading and using the gcloud bash tool to manage my accounts, however everything I do with the tool is exceptionally slow. It will take MINUTES to reply to a command that is typed. Is there perhaps a firewall I need to open up on my router or something else to get this to work fast like it's supposed to? For example, the "Installing..." lines in this video https://youtu.be/4y4-xn4Vi04?t=1m21s -- you'll notice they are all complete in the tutorial within a few seconds. This takes

google compute engine tool gcloud is exceptionally slow

北城以北 提交于 2020-06-10 08:05:19
问题 I tried downloading and using the gcloud bash tool to manage my accounts, however everything I do with the tool is exceptionally slow. It will take MINUTES to reply to a command that is typed. Is there perhaps a firewall I need to open up on my router or something else to get this to work fast like it's supposed to? For example, the "Installing..." lines in this video https://youtu.be/4y4-xn4Vi04?t=1m21s -- you'll notice they are all complete in the tutorial within a few seconds. This takes

Memory reduction Tensorflow TPU v2/v3 bfloat16

早过忘川 提交于 2020-05-30 03:14:33
问题 My model is too big to get a batch >64 with the normal v2 TPU devices. On the troubleshooting site it is mentioned that upcoming tensorflow versions will have bfloat16 support. Are the newly supported tf versions 1.9-1.12 capable to use bfloat16 now and if yes, is there a limited set of optimizers I can use? I did not find any further documentation on this but saw the usage of bfloat16 in the tensor2tensor model, so I guess there must be a way. Furthermore I read that TPU v3 supports bigger

Memory reduction Tensorflow TPU v2/v3 bfloat16

*爱你&永不变心* 提交于 2020-05-30 03:12:45
问题 My model is too big to get a batch >64 with the normal v2 TPU devices. On the troubleshooting site it is mentioned that upcoming tensorflow versions will have bfloat16 support. Are the newly supported tf versions 1.9-1.12 capable to use bfloat16 now and if yes, is there a limited set of optimizers I can use? I did not find any further documentation on this but saw the usage of bfloat16 in the tensor2tensor model, so I guess there must be a way. Furthermore I read that TPU v3 supports bigger

Reaching quota for ip address in use too fast for App engine

若如初见. 提交于 2020-05-29 05:52:11
问题 I have a webserver running in App Engine and the client is a mobile app. I am seeing that a lot of requests on the mobile are starting to fail once we scaled up to a lot of users. I am not seeing any failures in our logs however. I noticed in our quotas that our ip address in use for Compute Engine API is at its max of 8 (even though we're not running any services on Compute Engine). I am not sure if this is the root cause but it wasn't like this before, I was wondering if there is any advice

How to install python packages in a Google Dataproc cluster

廉价感情. 提交于 2020-05-28 23:20:14
问题 Is it possible to install python packages in a Google Dataproc cluster after the cluster is created and running? I tried to use " pip install xxxxxxx " in the master command line but it does not seem to work. Google's Dataproc documentation does not mention this situation. 回答1: This is generally not possible after cluster is created. I recommend using an initialization action to do this. As you've noticed, pip is also not available by default. So you'll want to run easy_install pip followed

How to prevent public access to Compute Engine External IP Address?

冷暖自知 提交于 2020-05-28 07:36:04
问题 I need to prevent users from accessing the site through the External IP Address on Compute Engine, they should can access site only via domain name (www.some-domain.com), not the IP Address itself. Let says the instance's IP Address of the Compute Engine is A.A.A.A. I am using Load Balancer to redirect it to secure connection, with these settings: Frontend HTTP : B.B.B.B:80 HTTPS : B.B.B.B:443 Backend Redirected to the active instance And I pointing the DNS (A) to B.B.B.B. Now, I am able to

What is the maximum bandwidth gcp instance can use with single network interface?

不羁岁月 提交于 2020-05-28 03:33:28
问题 I am uploading data to Google cloud storage bucket using n processes parallelly, egress data rate is ~16Gbps. For only downloading data from gcp storage bucket using n processes parallelly, ingress traffic is ~26Gbps. But when I execute upload and download at the same time with same no. of processes, ingress rate reduces to ~7Gbps and egress to ~11/12Gbps. Using different buckets for upload and download. Instance and buckets are in the same region. Using Windows Server 2012 r2 OS and

What is the maximum bandwidth gcp instance can use with single network interface?

元气小坏坏 提交于 2020-05-28 03:33:19
问题 I am uploading data to Google cloud storage bucket using n processes parallelly, egress data rate is ~16Gbps. For only downloading data from gcp storage bucket using n processes parallelly, ingress traffic is ~26Gbps. But when I execute upload and download at the same time with same no. of processes, ingress rate reduces to ~7Gbps and egress to ~11/12Gbps. Using different buckets for upload and download. Instance and buckets are in the same region. Using Windows Server 2012 r2 OS and