what's the best way to get file/directory information that's supported on Unix/Linux remote host

Question

I have a python code that needs to get file or directory information on a remote host that's Unix/Linux (OS names: HP-UX/RedHat/SunOS/AIX/Linux/etc).

The program SSH into the remote host (using paramiko library) and executes ls -l or ls -ld depending on if it's a file or a directory.

Information I need are:

1. Permission (user/group/other)
2. User owner
3. Group owner
4. Last Modified
5. File name

However, the problems with ls are:

1. Output is different from platform so special handling is needed which makes code verbose using checks.
2. File size unit may be different in the output depend on environment variable (for GNU coreutils, BLOCK_SIZE), or some may even not support this. This also requires platform-specific checks.

I am looking for a python library or simple portable executable if there is one.

Solutions I considered (but seems infeasible)

1. Use regex to check format of the output and process if it matches konwn format. However, this seems to be error prone due to try-checks.
2. Also try-check environment variables for the file size and found out the file size unit in the output. (eg echo few characters to a file and check the unit. If 4 char is written and size says 1 then I know for sure the unit is greater than 1 bytes. Repeat the steps). This seems to be also error prone
3. Install cross-platform compiler on each host, compile and then execute. Can't do it since if the host OS is reinstalled or restored to the point that doesn't have compiler, this installation process needs to be repeated.

Any suggestions?

Answer 1

Would stat do the trick? It seems like it has all the features you're looking for and should already be installed.

http://ss64.com/bash/stat.html

If you want to work in python, python has a built in library called stat which provides similar functionality:

https://docs.python.org/2/library/stat.html

Answer 2

SFTP is a standard file transport protocol built into SSH-2. So if you can SSH into the remote hosts, then most likely you can use SFTP in a standard way to list and stat files. SFTP is widely supported by SSH servers including OpenSSH.

paramiko is probably the most popular SSH/SFTP wrapper for Python. Here is an example script using paramiko to do an SFTP stat:

import paramiko

ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.WarningPolicy()) # Only warn on no known_hosts
ssh.connect("localhost", username="sam")
sftp = ssh.open_sftp()
try:
  listing = sftp.listdir_attr(".")
  print(listing[0:10])
  my_stat = sftp.stat(".")
  print(my_stat)
  print(my_stat.st_size, my_stat.st_mtime, my_stat.st_atime, my_stat.st_uid, my_stat.st_gid)
except IOError:
  pass
ssh.close()

Of course this is just a simple example and there is a lot more you can do with paramiko SFTP API - docs .

---

Update: Related post: SFTP in Python? (platform independent)

Answer 3

You probably don't have python installed on all those machines either, but I bet you have some ancient crusty version of perl on even the HPUX box.

~/tmp/t3 $perl -e 'print (join (",", (stat("t.awk"))), "\n");'
655368,160089,33204,1,16257,200,0,178,1480727842,1480710575,1480710575,4096,8

~/tmp/t3 $stat t.awk
  File: `t.awk'
  Size: 178             Blocks: 8          IO Block: 4096   regular file
Device: a0008h/655368d  Inode: 160089      Links: 1
Access: (0664/-rw-rw-r--)  Uid: (16257/mcgowan)   Gid: (  200/    users)
Access: 2016-12-02 17:17:22.000000000 -0800
Modify: 2016-12-02 12:29:35.000000000 -0800
Change: 2016-12-02 12:29:35.000000000 -0800

Here's the cross reference for the perl stat fields:

  0 dev      device number of filesystem
  1 ino      inode number
  2 mode     file mode  (type and permissions)
  3 nlink    number of (hard) links to the file
  4 uid      numeric user ID of file's owner
  5 gid      numeric group ID of file's owner
  6 rdev     the device identifier (special files only)
  7 size     total size of file, in bytes
  8 atime    last access time in seconds since the epoch
  9 mtime    last modify time in seconds since the epoch
 10 ctime    inode change time in seconds since the epoch (*)
 11 blksize  preferred I/O size in bytes for interacting with the
             file (may vary from file to file)
 12 blocks   actual number of system-specific blocks allocated
             on disk (often, but not always, 512 bytes each)