cluster-computing

问题 I am running python scripts on a computing cluster (slurm) with two stages and they are sequential. I wrote two python scripts, one for Stage 1 and another for Stage 2. Every morning I check if all Stage 1 jobs are completed visually. Only then, I start Stage 2. Is there a more elegant/automated way by combining all stages and job management in a single python script? How can I tell if the job has completed? The workflow is similar to the following: while not job_list.all_complete(): for job

问题 TL;DR: Is there any way to get SGE to round-robin between servers when scheduling jobs, instead of allocating all jobs to the same server whenever it can? Details: I have a large compute process that consists of many smaller jobs. I'm using SGE to distribute the work across multiple servers in a cluster. The process requires a varying number of tasks at different points in time (technically, it is a DAG of jobs). Sometimes the number of parallel jobs is very large (~1 per CPU in the cluster),

问题 Does anyone know if there is a platform independent way to get memory usage of a worker? I would expect it would work like this: console.log('App process memoryUsage: ',process.memoryUsage()); cluster.on('online',function(worker){ // doesn't work! console.log('Workers memory usage: ',worker.process.memoryUsage()); }); But the workers process hasn't the method memoryUsage() . Is there a valid reason this isn't implemented ? The only idea to realize this is to work with unix top -pid 1234

问题 Does anyone know if there is a platform independent way to get memory usage of a worker? I would expect it would work like this: console.log('App process memoryUsage: ',process.memoryUsage()); cluster.on('online',function(worker){ // doesn't work! console.log('Workers memory usage: ',worker.process.memoryUsage()); }); But the workers process hasn't the method memoryUsage() . Is there a valid reason this isn't implemented ? The only idea to realize this is to work with unix top -pid 1234

问题 This is an extension of a previous question I asked: Using name of BASH script as input argument My goal is to write a BASH script which takes the arguments from the file's name and uses them as inputs for a Julia code I'm writing, and then submit the BASH script to a remote cluster. Using @AndriyMakukha's solution, I was able to write the following script through Torque: #!/bin/bash #PBS -l mem=10gb,nodes=1:ppn=2,walltime=1:00:00 #PBS -N ES_100_20_100 #PBS -j oe #PBS -o ./log/julia.${PBS

问题 This is a follow-up to my previous question: Julia alias in .bashrc not working correctly My main issue is that I cannot add Julia to my path on a Linux Cluster. If I go into the bin subfolder of the Julia folder, I can run Julia itself fine. However, I cannot edit the .bash_profile to get it into my path. As opposed to my previous question, I believe my issue has less with building an alias (or something Julia related) and more concerning my .bash_profile itself not updated. The Julia file

问题 I am finding it incredibly difficult to follow rays guidelines to running a docker image on a ray cluster in order to execute a python script. I am finding a lack of simple working examples. So I have the simplest docker file: FROM rayproject/ray WORKDIR /usr/src/app COPY . . CMD ["step_1.py"] ENTRYPOINT ["python3"] I use this to create can image and push this to docker hub. ("myimage" is just an example) docker build -t myimage . docker push myimage "step_1.py" just prints hello every second

问题 I want to run my code on a Cluster: my code: import java.util.Properties import edu.stanford.nlp.ling.CoreAnnotations._ import edu.stanford.nlp.pipeline._ import org.apache.spark.{SparkConf, SparkContext} import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer object Pre2 { def plainTextToLemmas(text: String, pipeline: StanfordCoreNLP): Seq[String] = { val doc = new Annotation(text) pipeline.annotate(doc) val lemmas = new ArrayBuffer[String]() val sentences = doc

问题 I want to run my code on a Cluster: my code: import java.util.Properties import edu.stanford.nlp.ling.CoreAnnotations._ import edu.stanford.nlp.pipeline._ import org.apache.spark.{SparkConf, SparkContext} import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer object Pre2 { def plainTextToLemmas(text: String, pipeline: StanfordCoreNLP): Seq[String] = { val doc = new Annotation(text) pipeline.annotate(doc) val lemmas = new ArrayBuffer[String]() val sentences = doc

问题 I want to run my code on a Cluster: my code: import java.util.Properties import edu.stanford.nlp.ling.CoreAnnotations._ import edu.stanford.nlp.pipeline._ import org.apache.spark.{SparkConf, SparkContext} import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer object Pre2 { def plainTextToLemmas(text: String, pipeline: StanfordCoreNLP): Seq[String] = { val doc = new Annotation(text) pipeline.annotate(doc) val lemmas = new ArrayBuffer[String]() val sentences = doc