Windows上的GNU make 4.2（和4.2.1）截断配方中的行

Question

I want to move to make 4.2 so I can have the ability to do parallel jobs on Windows using --jobs=N .

make 4.2 is not available in binary form from SourceForge, only 3.81

So I had to build it from source - they included a VS studio project - and I was up and running.

Problem is, it appears to have a bug though it could be a bug with my makefile.

I ran make4.2 all --print-data-base --debug=vjm on my project and I got ERRORS. What's unfortunate is that the error is NOT related to syntax of rules as this project builds without error on make 3.81 using the same options.

Here is the command it fails on.

COPY_DIR      = cp -f -R
MKDIR         = mkdir -p

deploy_marketProperties: deploy_themeMappingConfig
    @echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/TargetDir" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/Source/*) "C:/Users/User1/Desktop/TargetDir"

1. Creates a directory
2. Copies bunch of files from one directory to that new directory

Used the same make call for both versions of make: make all --print-data-base --debug=vjm

However, I get the following error when using make4.2 :

Copying application data...
cp: target `C' is not a directory
makev42[3]: *** [Makefile:383: deploy_marketProperties] 

The command it tried running was:

echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/TargetDir" && cp -f -R C:/Users/User1/Desktop/Source/im.json "C:/Users/User1/Desktop/TargetDir"

There's actually 264 .json files but I only show one for readability.

Again, this runs fine on make v3.81 using exact same command-line options.

It appears make 4.2 has a problem parsing that huge list of files maybe??

The question

Is there something I can change with my makefiles to make it run on make v4.2 ?

We are highly motivated to make it work with a 4.0+ version of make . Otherwise we have to use a build tool that will cost $$.

UPDATE 1

Per the suggestion of @MadScientist, I have executed both versions of make and present the outputs.

C:\\release>make381 deploy_marketProperties --debug=vjm > 381.err 2>&1

C:\\release>make42 deploy_marketProperties --debug=vjm > 42.err 2>&1

The target deploy_marketProperties has the following rule:

deploy_marketProperties: 
    echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/A/Project/src/dir1/marketProperties/*) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"

The output of make v 3.81 . NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for Windows32
find_and_set_shell path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile `Makefile'...
Updating goal targets....
Considering target file `deploy_marketProperties'.
 File `deploy_marketProperties' does not exist.
 Finished prerequisites of target file `deploy_marketProperties'.
Must remake target `deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/2blank.json C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/3blank.json  "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 0x000000ac
Copying application data...
Successfully remade target file `deploy_marketProperties'.

The output of make v 4.2 . NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):

GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
 File 'deploy_marketProperties' does not exist.
 Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/im.json C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/da.json  "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 00000088
Copying application data...
cp: target `C' is not a directory
make_msvc.net2003: *** [Makefile:384: deploy_marketProperties] Error 1

Note that both make versions use the same shell:

C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe

C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage:  sh [GNU long option] [option] ...
        sh [GNU long option] [option] script-file ...
GNU long options:
        --debug
        --debugger
        --dump-po-strings
        --dump-strings
        --help
        --init-file
        --login
        --noediting
        --noprofile
        --norc
        --posix
        --protected
        --rcfile
        --restricted
        --verbose
        --version
        --wordexp
Shell options:
        -irsD or -c command or -O shopt_option          (invocation only)
        -abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.

NOTE THE msys @MadScientist; is that relevant?

UPDATE 2

I re-ran make 4.2 but I deleted all but ONE .json file in the directory and - surprise! - it works.

This is the ENTIRE output from make42 deploy_marketProperties --debug=vjm :

GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
 File 'deploy_marketProperties' does not exist.
 Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/subProject/marketProperties/2blank.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 000000B4
Copying application data...
Successfully remade target file 'deploy_marketProperties'.

UPDATE 3

I tried it with make 4.2.1 and I got the exact same behavior as on make 4.2 : fails when there are hundreds of .json files but succeeds if there less than some magic number of characters.

I got this message during one of my trials:

cp: target `C:/Users/User1/Des' is not a directory

Obviously it's getting cut-off but why? Same shell being used and only thing different is version of make .

Answer 1

You should use 4.2.1, it has some important fixes (I doubt it will make a difference for this problem).

As discussed in the comments, using POSIX tools on Windows requires a specialized environment. The normal "shell" on Windows is command.com, which doesn't accept POSIX sh syntax, such as && . The normal mkdir command on Windows doesn't accept the -p flag; that's a POSIX flag. There is no cp command on Windows.

So it's clear that your makefile is expecting to be run with some sort of POSIX environment.

Based on what you've described above, it's not clear how the problem is related to GNU make directly. GNU make is parsing the makefile and it's invoking the shell and passing along the command line properly: this can be inferred because the shell is running the echo command and running the mkdir command, and even running the cp command.

It's the cp command which is printing the message, not make. I believe you that it works with 3.81, but I suspect a non-obvious difference between the way 3.81 was compiled and the way you're building 4.2. It's even possible that there have been custom patches applied to the 3.81 version you're using, although I don't know.

To debug further I recommend the following:

• First remove the @ prefix from the recipe line: that's always the number one thing to do; if you can't see how make is invoking the command how can you debug?
• Then run the same makefile with the two different versions of make, from the same shell/prompt (so the path to make is the only difference). Is there any difference in the printed output? If so what is it?
• If not, then clearly there's something more mysterious going on. Try to see if maybe the QNX version of make is modifying the environment somehow: you can run the env command in your recipe to see it.

Maybe it's related to the length of the command line? Try temporarily moving some of the json files out of that directory so the command line is shorter.

Answer 2

I asked the make-w32@gnu.org mailing list and I have the answer.

To fix it, I had to uncomment the macro BATCH_MODE_ONLY_SHELL in config.h of the make source code.

/*
 * If you have a shell that does not grok 'sh -c quoted-command-line'
 * correctly, you need this setting. Please see below for specific
 * shell support.
 */
#define BATCH_MODE_ONLY_SHELL 1

Taking quote from reply from mailing list:

What that flag means (as I understand it: I'm not that familiar with this aspect of Windows support) is that make will never try to invoke the shell directly passing the recipe to be run on the command line.

Instead it is being forced to always write the recipe to a temporary file ("batch file") on your disk and invoke the shell such that it runs the recipe in the temporary file.

Apparently your shell is able to read and execute very long command lines from a file, which it cannot successfully process when passed in as arguments, even though these command line arguments are not exceeding Windows limits.