[英]Snakemake : SFTP on local machine
I am connected with ssh on a remote server from my local machine.我从本地机器与远程服务器上的 ssh 连接。
I run my Snakemake on the remote server.我在远程服务器上运行我的 Snakemake。
I would like to use as input of a rule, a file that is on my local machine.我想将本地计算机上的文件用作规则的输入。
Of course, since I run my Snakemake on the server, the server become the local machine and the local machine the remote one (for Snakemake).当然,因为我在服务器上运行我的 Snakemake,服务器成为本地机器,本地机器成为远程机器(对于 Snakemake)。
from snakemake.remote.SFTP import RemoteProvider
# I am not sure about the private key, is it the one I have on the server ?
# I have the same result with or without private_key anyway
# SFTP = RemoteProvider(port=22, username="myusername", private_key="/path/to/.ssh/id_rsa")
SFTP = RemoteProvider(port=22, username="myusername")
configfile : "config.json"
localrules: copyBclLocalToCluster
rule all:
input:
"copycluster.txt"
rule copyBclLocalToCluster:
input:
SFTP.remote("adress:path/to/filelocal.txt")
output:
"copycluster.txt"
shell:
"scp {input} {output}"
-----------------------------------------
Building DAG of jobs...
MissingInputException in line 26 of /path/to/Snakefile:
Missing input files for rule copyBclLocalToCluster:
adress:path/to/filelocal.txt
https://snakemake.readthedocs.io/en/stable/snakefiles/remote_files.html
https://snakemake.readthedocs.io/en/stable/snakefiles/remote_files.html
The remote file addresses used must be specified with the host (domain or IP address) and the absolute path to the file on the remote server.使用的远程文件地址必须与主机(域或 IP 地址)和远程服务器上文件的绝对路径一起指定。 A port may be specified if the SSH daemon on the server is listening on a port other than 22, in either the RemoteProvider or in each instance of remote():
如果服务器上的 SSH 守护进程在 RemoteProvider 或每个 remote() 实例中侦听 22 以外的端口,则可以指定端口:
The doc says that the port shouldn't be port 22, but why ?文档说端口不应该是端口 22,但为什么呢? I really would like to use it since I don't know how to configure another port and I'm not even sure to have the rights to do it.
我真的很想使用它,因为我不知道如何配置另一个端口,而且我什至不确定是否有权这样做。
Is it really a port issue ?真的是端口问题吗? Or I just don't understand how to use SFTP with Snakemake.
或者我只是不明白如何将 SFTP 与 Snakemake 一起使用。
What is the best way to use a file on my local machine as input of my snakemake ?在我的本地机器上使用文件作为我的 snakemake 输入的最佳方法是什么?
It is not the port the problem, I don't even need to specify it because it is port 22.这不是端口的问题,我什至不需要指定它,因为它是端口 22。
I tried to specify the good ssh private key :我试图指定好的 ssh 私钥:
SFTP = RemoteProvider(port=22, username="myusername", private_key="/path/to/.ssh/id_rsa")
-----------------------------
Building DAG of jobs...
MissingInputException in line 26 of /path/to/Snakefile:
Missing input files for rule copyBclLocalToCluster:
adress:path/to/filelocal.txt
If I try sftp myusername@adress:path/to/filelocal.txt .
如果我尝试
sftp myusername@adress:path/to/filelocal.txt .
on my console on the server it works fine.在我服务器上的控制台上它工作正常。
Why it doesn't work inside snakemake ?为什么它在 snakemake 中不起作用?
When I try to use my password instead of ssh-key in remoteProvider
I have the same error.当我尝试在
remoteProvider
使用我的密码而不是 ssh-key 时,我remoteProvider
了同样的错误。
SFTP = RemoteProvider(port=22, username="myusername", password="mypassword")
--------------------------------
Building DAG of jobs...
MissingInputException in line 26 of /path/to/Snakefile:
Missing input files for rule copyBclLocalToCluster:
adress:path/to/filelocal.txt
I am sure the adress, username, password, ssh-key are correct and file exist, I can do it outside snakemake it works fine.我确定地址、用户名、密码、ssh-key 是正确的并且文件存在,我可以在snakemake 之外进行它工作正常。
Since RemoteProvider
uses pysftp
, I tried to copy the same file with pysftp
in a python
script.由于
RemoteProvider
用途pysftp
,我试图相同的文件复制与pysftp
在python
脚本。
import pysftp
with pysftp.Connection(adress,
username="myusername",
private_key_pass="/path/to/.ssh/id_rsa") as sftp:
sftp.get(path/to/filelocal.txt, /path/on/cluster/fileCOPY.txt)
It works fine, so the problem come from my Snakefile for sure.它工作正常,所以问题肯定来自我的 Snakefile。
RemoteProvider
also need ftputil
, I tried ftputil in a python script. RemoteProvider
也需要ftputil
,我在 python 脚本中尝试了 ftputil 。
import ftputil
with ftputil.FTPHost("adress", "myusername", "mypassword") as ftp_host:
print(getcwd())
ftp_host.download(remote_path, local_path)
----------------------------------------------
Traceback (most recent call last):
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/site-packages/ftputil/host.py", line 129, in _make_session
return factory(*args, **kwargs)
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/ftplib.py", line 117, in __init__
self.connect(host)
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/ftplib.py", line 152, in connect
source_address=self.source_address)
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "sftptest.py", line 16, in <module>
with ftputil.FTPHost("adress", "myusername", "mypassword") as ftp_host:
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/site-packages/ftputil/host.py", line 69, in __init__
self._session = self._make_session()
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/site-packages/ftputil/host.py", line 129, in _make_session
return factory(*args, **kwargs)
File "/work/username/miniconda3/envs/RNAseq_snakemake/lib/python3.6/site-packages/ftputil/error.py", line 146, in __exit__
raise FTPOSError(*exc_value.args, original_exception=exc_value)
ftputil.error.FTPOSError: [Errno 111] Connection refused
Debugging info: ftputil 3.2, Python 3.6.7 (linux)
Could it be a problem ?会不会有问题? But I don't have this kind of error in snakemake, just missing file error.
但是我在snakemake中没有这种错误,只是缺少文件错误。 I don't understand why ftputil is not working.
我不明白为什么 ftputil 不起作用。
So I did a quick & dirty search in the source code of Snakemake, and Snakemake makes use of ftputil
which requires a username and password.所以我在 Snakemake 的源代码中做了一个快速而肮脏的搜索,Snakemake 使用了需要用户名和密码的
ftputil
。 When you do not provide a ssh-key path to Snakemake this password will default to None
, which then gets passed to ftputil
.当您不提供 Snakemake 的 ssh-key 路径时,此密码将默认为
None
,然后传递给ftputil
。
See Snakemake source .参见Snakemake 源。
I agree that the default behaviour should default to something more sensible like ~/.ssh/id_rsa
, but unfortunately it doesn't.我同意默认行为应该默认为更合理的东西,比如
~/.ssh/id_rsa
,但不幸的是它没有。
When you use SFTP on console you have to write当您在控制台上使用 SFTP 时,您必须编写
sftp myusername@adress:/path/to/file .
But in remote function of Snakemake you should delete the ":" between host and the path of the file.但是在Snakemake的远程功能中,您应该删除主机和文件路径之间的“:”。
I was mislead by the SFTP syntax but it was well written in the snakemake doc我被 SFTP 语法误导了,但它在 snakemake 文档中写得很好
# example from snakemake doc
SFTP.remote("example.com/path/to/file.bam")
# what I was doing badly
SFTP.remote("adress:path/to/filelocal.txt")
The right command is :正确的命令是:
from snakemake.remote.SFTP import RemoteProvider
SFTP = RemoteProvider(port=22, username="myusername", password="mypassword")
rule all:
input:
# "/" instead of ":" between host and the path of the file
SFTP.remote("adress/path/to/filelocal.txt")
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.