Import own python modules in nextflow script block?

Question

I created a python script called utilities.py in bin/ directory:

#!/usr/bin/env python3

import numpy as np
import pandas as pd
from datetime import datetime
import io 


def print_info(in_df, fname_base):
    buffer = io.StringIO()
    df = in_df.copy()
    df.info(buf=buffer)
    s = buffer.getvalue()
    with open(fname_base+"_info.txt", "w", encoding="utf-8") as f:  
        f.write(s)

def print_desc(in_df, fname_base):
    df = in_df.copy()
    desc = df.describe()
    desc.to_csv(fname_base+"_desc.tsv", sep = '\t')
    
def print_data(in_df, fname_base):
    df = in_df.copy()
    print_info(df, fname_base)
    print_desc(df, fname_base)
    df.to_csv(fname_base+".tsv", sep = '\t')

and made it executable with chmod +x . I would like to use these functions in a several script blocks in various processes in my workflow. Currently when I try importing a function from my utilities module:

#!/bin/bash nextflow 

process transform_data {

    input:
    path(data)

    output:
    path("out.tsv"), emit: out_data

    script:
    """
    #!/usr/bin/env python3
    import pandas as pd
    import io
    from utilities import print_info
    """
}

I get the following error:

  Traceback (most recent call last):
    File ".command.sh", line 4, in <module>
      from utilities import print_info
  ModuleNotFoundError: No module named 'utilities'

Is it possible to import own modules in this way?

Answer 1

Which version of Nextflow are you using? I tested with v22.04.5 and the following works:

My setup is little bit different, instead of specifying #!/usr/bin/env python3 , I directly invoked a python script ( test.py ) which has from utilities import print_info inside it, and it works fine.

script:
    """
      test.py
    """

Note that the following won't work: from.utilities import print_info . Therefore, you can import custom Python module with Nextflow.