queue instance "" dropped because it is full or is full due to overload discard all queues

Other similar error

queue instance " " dropped because it is temporarily not available 

queue instance " " dropped because it is disabled

Error display :

$qstat -j
queue instance "peteris.q@sge00" dropped because it is full

Explain: sge due to overload or discard all the queue is full

 

Solve the answer:

You seem to have reached a practical limit active job queue at any given time can handle the number. I'm not sure where SGE-defined maximum, but it seems likely:

max_jobs

Sun Grid Engine while allowing active (not completed) by the parameter control of the number of jobs. Defined limit value greater than 0. The default value of 0 means "unlimited." If the job is submitted beyond the max_jobs limit, the commit command will exit with an exit status 25, and displays the appropriate error message. Max_jobs changes will take effect immediately. This value is the only global configuration parameters. Execution host local configuration can not cover it.

From: HTTP  :  //gridscheduler.sourceforge.net/htmlman/htmlman5/sge_conf.html?pathrev=V62u5_TAG

If this is correct, then the value is infinite. However, SGE may be unable to properly manage approximately one million active work, so you may encounter this problem. I recommend that you use a job array , because this is the purpose of such operations, namely the management and operation of many nearly identical tasks .

SGE There are many online resources available for operation array, for example:

http://wiki.gridengine.info/wiki/index.php/Simple-Job-Array-Howto

http://talby.rcs.manchester.ac.uk/~ri/_linux_and_hpc_lib/sge_array.html

https://wiki.duke.edu/display/SCSC/SGE+Array+Jobs

If you edit a question depending on the specific requirements of each task, we will be happy to provide further assistance. For example, about one million tasks whether each requiring one or more parameters as input?

 

18 years May 29 at 12:03  answer

Vince

 

•  
  Thank you for your answer. But I do have maxjobs set to 0  -   kim Jason  18 on April 4 at 8:58 
•  
  So maxjobsto 0solve this problem? Or, you have set maxjobsfor the 0 problem persists?  -  Vince  18 years   at 12:53 on June 4
•  
  I suspect the problem is that since the author resource node, SGE can not control the number of jobs. For tasks involving thousands of workloads, it is recommended to use an array of tasks.  -  Vince  18 on   June 19 at 17:07

   

   

  Source: https://stackoverflow.com/questions/50551042/sge-all-queues-dropped-because-of-overload-or-full

   

   

  Sun Grid Engine array job

  1。

  why?

  Suppose you want to run a large number of jobs is largely the same: You may want to use a different parameter or parameters of multiple runs of the same program; or processing a thousand different input files. You can write a Perl script to generate all the necessary qsub file and write a BASH script to submit all these documents. However, this is not your time easy to use, Submit it would have on the cluster (login) node dire impact.

  Use SGE array job will be better!

  2。

  what?

  SGE array job can be described as a built-for-loop operation. This is a simple example:

    #! / Bin / bash 
  
    # $ -cwd 
    # $ -S / bin / bash 
  
    # $ -t from 1 to 1000 
        # ... This is a tell SGE array jobs, "tasks" are numbered starting from 1 
        # to 10000 ... 
  
    ./ myprog <data. $ SGE_TASK_ID> results. $ SGE_TASK_ID
  

  By calculation, this is equivalent to 1000 submitted separate queue, wherein SGE_TASK_ID values 1,2,3. . 1000, where the input and output files indexed by ID. however:

  • Only issue a qsub command (and only one qdel command to delete all jobs);
  • qstat output display only one entry ;
  • SGE load submitting node (ie login node cluster) on considerably less than 1000 submitted to load a separate job!

   

  A slight change - in a separate directory (folder) to run each job:

    ＃！/ bin / bash
  
    ＃$ -cwd
    ＃$ -S / bin / bash
  
    ＃$ -t 1-1000
  
    mkdir myjob- $ SGE_TASK_ID
    cd myjob- $ SGE_TASK_ID
    ../myprog-one> one.output
    ../myprog-two <one.output> two.output
  
  
  

  3。

  More

  For more information on SGE job arrays, please visit:

  • wiki.gridengine.info
  • wikis.sun.com

   

  4。	For more general circulation

  SGE_TASK_ID  do not have to start at 1; Increment not one. E.g:

    ＃$ -t 100-995：5
  

  So SGE_TASK_ID the value of 100,105,110,115  ... 995.

   

  Incidentally, for example, the upper limit is not equal to an integral multiple of the case plus the increment,

    ＃$ -t 1-42：6
  

  SGE automatically changes the upper limit, that is,

    Tips> qsub array.qsub 
    your job array 2642.1-42: 6 ( "array.qsub") has been submitted 
  
    tips> qstat 
    job ID priority status to submit a user name / start to queue slot Task-ID-ja 
    ----- --------------------------------------------- ----- ------------------------------------ 
     2642 0.00000 array.qsub simonh QW 04/24/2009 12: 29:29 11-37: 6
  

   

  5，	Related environmental variables

  You can use the other three environment variables are automatically created, such as the following simple qsub shown script:

    #! / Bin / the bash 
  
    # -cwd $ 
    # $ -S / bin / the bash 
  
    # $ -t 1-37:. 6 
  
    echo "increments ID: $ SGE_TASK_STEPSIZE" 
  
    if [[$ SGE_TASK_ID == $ SGE_TASK_FIRST] ]; then 
        echo " The first " 
    elif [[$ SGE_TASK_ID == $ SGE_TASK_LAST]]; then 
        echo" last " 
    other 
        echo" not " 
    science fiction
  

   

  6。	Enter the file list

  Can be sneaky - Suppose we have an input file list instead of the input file explicitly indexed by the suffix:

    ＃！/ bin / bash
  
    ＃$ -cwd
    ＃$ -S / bin / bash
  
    ＃$ -t 1-42
  
    $ INFILE =`awk“ NR == $ SGE_TASK_ID” my_file_list.text`
        ＃
        ＃...或使用的sed：    
        ＃sed -n“ $ {SGE_TASK_ID} p” my_file_list.text
        ＃
  
    ./myprog <$ INFILE

  
  
  Source: 
  http://talby.rcs.manchester.ac.uk/~ri/_linux_and_hpc_lib/sge_array.html