InputFunctionException: unexpected EOF while parsing

问题

Major EDIT: Having fixed a couple of issues thanks to comments and written a minimal reproducible example to help my helpers, I've narrowed down the issue to a difference between execution locally and using DRMAA.

Here is a minimal reproducible pipeline that does not require any external file download and can be executed out of the box or clone following git repository:

git clone git@github.com:kevinrue/snakemake-issue-all.git

When I run the pipeline using DRMAA I get the following error:

Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3
InputFunctionException in line 22 of /ifs/research-groups/sims/kevin/snakemake-issue-all/workflow/Snakefile:
SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=A

However, if I run the pipeline locally (--cores 1), it works:

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3

[Sat Jun 13 08:49:46 2020]
rule cat:
    input: data/A1, data/A2
    output: results/A/cat
    jobid: 1
    wildcards: sample=A

[Sat Jun 13 08:49:46 2020]
Finished job 1.
1 of 3 steps (33%) done

[Sat Jun 13 08:49:46 2020]
rule cat:
    input: data/B1, data/B2
    output: results/B/cat
    jobid: 2
    wildcards: sample=B

[Sat Jun 13 08:49:46 2020]
Finished job 2.
2 of 3 steps (67%) done

[Sat Jun 13 08:49:46 2020]
localrule all:
    input: results/A/cat, results/B/cat
    jobid: 0

[Sat Jun 13 08:49:46 2020]
Finished job 0.
3 of 3 steps (100%) done
Complete log: /ifs/research-groups/sims/kevin/snakemake-issue-all/.snakemake/log/2020-06-13T084945.632545.snakemake.log

My DRMAA profile is the following:

jobs: 100
default-resources: 'mem_free=4G'
drmaa: "-V -notify -p -10 -l mem_free={resources.mem_free} -pe dedicated {threads} -v MKL_NUM_THREADS={threads} -v OPENBLAS_NUM_THREADS={threads} -v OMP_NUM_THREADS={threads} -R y -q all.q"
drmaa-log-dir: /ifs/scratch/kevin
use-conda: true
conda-prefix: /ifs/home/kevin/devel/snakemake/envs
printshellcmds: true
reason: true

Briefly, the Snakefile looks like this

# The main entry point of your workflow.
# After configuring, running snakemake -n in a clone of this repository should successfully execute a dry-run of the workflow.


report: "report/workflow.rst"

# Allow users to fix the underlying OS via singularity.
singularity: "docker://continuumio/miniconda3"

include: "rules/common.smk"
include: "rules/other.smk"

rule all:
    input:
        # The first rule should define the default target files
        # Subsequent target rules can be specified below. They should start with all_*.
        expand("results/{sample}/cat", sample=samples['sample'])


rule cat:
    input:
        file1="data/{sample}1",
        file2="data/{sample}2"
    output:
        "results/{sample}/cat"
    shell:
        "cat {input.file1} {input.file2} > {output}"

Running snakemake -np gives me what I expect:

$ snakemake -np
          sample  condition
sample_id                  
A              A  untreated
B              B    treated
Building DAG of jobs...
Job counts:
    count   jobs
    1   all
    2   cat
    3

[Sat Jun 13 08:51:19 2020]
rule cat:
    input: data/B1, data/B2
    output: results/B/cat
    jobid: 2
    wildcards: sample=B

cat data/B1 data/B2 > results/B/cat

[Sat Jun 13 08:51:19 2020]
rule cat:
    input: data/A1, data/A2
    output: results/A/cat
    jobid: 1
    wildcards: sample=A

cat data/A1 data/A2 > results/A/cat

[Sat Jun 13 08:51:19 2020]
localrule all:
    input: results/A/cat, results/B/cat
    jobid: 0

Job counts:
    count   jobs
    1   all
    2   cat
    3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

I'm not sure how to debug it further. I'm happy to provide more information as needed.

Note: I use snakemake version 5.19.2

Thanks in advance!

EDIT Using the --verbose option, Snakemake seems to trip on the 'default-resources: 'mem_free=4G' and/or drmaa: "-l mem_free={resources.mem_free} that are defined in my 'drmaa' profile (see above).

$ snakemake --profile drmaa --verbose
Building DAG of jobs...
Using shell: /bin/bash
Provided cluster nodes: 100
Singularity containers: ignored
Job counts:
    count   jobs
    1   all
    2   cat
    3
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 100}
Ready jobs (2):
    cat
    cat
Full Traceback (most recent call last):
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 941, in apply
    res, _ = self.apply_input_function(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 684, in apply_input_function
    raise e
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 678, in apply_input_function
    value = func(Wildcards(fromdict=wildcards), **_aux_params)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/resources.py", line 10, in callable
    value = eval(
  File "<string>", line 1
    4G
     ^
SyntaxError: unexpected EOF while parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/__init__.py", line 626, in snakemake
    success = workflow.execute(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/workflow.py", line 951, in execute
    success = scheduler.schedule()
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 394, in schedule
    run = self.job_selector(needrun)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 540, in job_selector
    a = list(map(self.job_weight, jobs))  # resource usage of jobs
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/scheduler.py", line 613, in job_weight
    res = job.resources
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/jobs.py", line 267, in resources
    self._resources = self.rule.expand_resources(
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 977, in expand_resources
    resources[name] = apply(name, res, threads=threads)
  File "/ifs/devel/kevin/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/rules.py", line 960, in apply
    raise InputFunctionException(e, rule=self, wildcards=wildcards)
snakemake.exceptions.InputFunctionException: SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=B

InputFunctionException in line 20 of /ifs/research-groups/sims/kevin/snakemake-issue-all/workflow/Snakefile:
SyntaxError: unexpected EOF while parsing (<string>, line 1)
Wildcards:
sample=B
unlocking
removing lock
removing lock
removed all locks

回答1:

Thanks to @JohannesKöster I realised that my profile settings were wrong.

--default-resources [NAME=INT [NAME=INT ...]] indicates indicates that only integer values are supported, while I was providing string (i.e., mem_free=4G), naively hoping those would be supported as well.

I've updated the following settings in my profile, and successfully ran both snakemake --cores 1 and snakemake --profile drmaa.

default-resources: 'mem_free=4'
drmaa: "-V -notify -p -10 -l mem_free={resources.mem_free}G -pe dedicated {threads} -v MKL_NUM_THREADS={threads} -v OPENBLAS_NUM_THREADS={threads} -v OMP_NUM_THREADS={threads} -R y -q all.q"

Note the integer value 4 set as default resources, and how I moved the G to the drmaa: ... -l mem_free=...G setting.

Thanks a lot for the help everyone!

来源：https://stackoverflow.com/questions/62328724/inputfunctionexception-unexpected-eof-while-parsing