Pseudo terminal problems (Mac/Linux): SIGTTOU & Inappropriate ioctl

Question

I am working on a pseudo terminal library. The code is implemented in C code and the code is used by a web based terminal. The code works as long as I do not use sudo or login.

This is the error I get when I run the server on a Mac:

sh-3.2$ sudo ls
Password:
[1]+  Stopped(SIGTTOU)
sh-3.2$

The above works on Linux:

$ sudo ls
  readme.txt

However, I get the following on Linux with sudo bash:

$ sudo bash
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
]0;root@ubuntu: /tmproot@ubuntu:/tmp#

Note: the above works, but I have no job control.

I have probably forgot to set some controlling bits on the terminal, but Google has not been very helpful in finding this. Also, do you know of any good books that explains Pseudo terminal management in great detail.

I have the setsid call, but I am not using openpty. I use the following code when opening the pty:

static int createPty(lua_State* L, char* ttyName, int* pty)
{
   *pty = getpt();
   if (*pty < 0 || grantpt(*pty) < 0 || unlockpt(*pty) < 0)
      return lDoErr(L,"Cannot open PTY: %s",strerror(errno));
   if(ptsname_r(*pty, ttyName, PTY_NAME_SIZE-1))
      return lDoErr(L,"ptsname_r: %s",strerror(errno));
   return 0;
}

I have edited the code below and this code works. The reason my first version did not work was that I tried to create two PTY channels. I wanted to be able to differentiate between stdout and stderr, but the Linux kernel does not allow multiple TIOCSCTTY calls.

static int
childOpenTTY(const char* ttyName)
{
   struct termios termbuf;
   int fd=open(ttyName, O_RDWR);
   if(fd < 0)
      doClientError("open %s: %s",ttyName, strerror(errno));
   tcsetpgrp(fd, getpid());
   ioctl(fd,TIOCSCTTY,NULL);
   tcgetattr(fd, &termbuf);
   cfmakeraw(&termbuf); /* turn off NL to CR/NL mapping on output. */
   tcsetattr(fd, TCSANOW, &termbuf);
   return fd;
}

if( (ret = createPty(L, ttyName, &te->pty)) != 0)
   return ret;
if ((te->pid = zzbafork()) < 0)
   return lDoErr(L,"fork: %s",strerror(errno));
if(te->pid == 0)
{  /* Child process */
   static const char efmt[]={"Cannot set '%s' (dup2 err)"};
   int fd;
   if(setsid() < 0) /* make new process group */
      doClientError("setsid: %s",strerror(errno));
   fd=childOpenTTY(ttyName);
   if(dup2(fd, STDIN_FILENO) != STDIN_FILENO)
      doClientError(efmt,"stdin");
   if(dup2(fd, STDOUT_FILENO) != STDOUT_FILENO)
      doClientError(efmt,"stdout");
   if(dup2(fd, STDERR_FILENO) != STDERR_FILENO)
      doClientError(efmt,"stderr");
   if(fd != STDIN_FILENO && fd != STDOUT_FILENO && fd != STDERR_FILENO)
      close(fd);
   execve(cmd, (char**)cmdArgv, environ);
   /* execve should not return, unless error exec cmd */
   doClientError("Executing %s failed: %s",cmd,strerror(errno));
}

Answer 1

It's hard to be sure since there's no actual code shown here, but I suspect you're running into POSIX-style "session" management. You need to execute a setsid call, then open the pty (slave side) such that it becomes the controlling terminal. The openpty and login_tty routines do the low level grunge work for you; are you using those?