Bug 39191 - "make dist" fails on older versions of Bash due to >1000 duplicated entries in MonoDevelop.Ide EXTRA_DIST
Summary: "make dist" fails on older versions of Bash due to >1000 duplicated entries i...
Status: CONFIRMED
Alias: None
Product: Xamarin Studio
Classification: Desktop
Component: Installer ()
Version: 6.0.0 (C7)
Hardware: PC Linux
: --- normal
Target Milestone: master
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-02-29 07:20 UTC by Timotheus Pokorra
Modified: 2016-07-08 14:08 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report for Bug 39191 on Developer Community or GitHub if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: Developer Community HTML or GitHub Markdown
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:
Status:
CONFIRMED

Description Timotheus Pokorra 2016-02-29 07:20:29 UTC
The latest tarball for Cycle 7 Alpha has not been uploaded to http://download.mono-project.com/sources/monodevelop/ yet.

I looked at https://jenkins.mono-project.com/view/Packaging-MonoDevelop/job/build-source-tarball-monodevelop/ and that seems to fail for the past weeks.

15:53:20 [build-source-tarball-monodevelop] $ /bin/sh -xe /tmp/hudson1899508319479617037.sh
15:53:20 + rm -f tarballs/external
15:53:20 rm: cannot remove `tarballs/external': Is a directory

Would be great if this could be fixed, and the tarball uploaded!

Thanks!
Comment 1 Jo Shields 2016-02-29 10:56:10 UTC
I can trigger it, but I don't know entirely what to expect from the attempt to build it - MD6 seems to have a lot of problems building on my Jenkins infrastructure (e.g. lots of GLib errors)
Comment 2 Timotheus Pokorra 2016-02-29 13:12:07 UTC
I tried to run make dist on tag monodevelop-6.0.0.4520 

./configure --profile=stable
make dist

fails then on this:

 (cd MonoDevelop.Ide && make  top_distdir=../../../monodevelop-6.0 distdir=../../../monodevelop-6.0/src/core/MonoDevelop.Ide \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[5]: Entering directory '/root/monodevelop/main/src/core/MonoDevelop.Ide'
make[5]: execvp: /bin/sh: Argument list too long
Makefile:311: recipe for target 'distdir' failed

It seems that $(DISTFILES) contains too many files for bash?
Comment 3 Timotheus Pokorra 2016-02-29 13:31:44 UTC
for some reason, the icons, eg. remove-16~dark~disabled.png, are twice in DISTFILES.

Probably caused by this:

  ALL_FILES := $(EXTS) \
                $(addprefix */,$(EXTS))         \
                $(addprefix */*/,$(EXTS))       \
                $(addprefix */*/*/,$(EXTS))     \
                $(addprefix */*/*/*/,$(EXTS))   \
                $(addprefix */*/*/*/*/,$(EXTS)) \
                $(wildcard icons/*)             \
                $(wildcard templates/*)

the png file is included twice because of png in EXTS, and the file is in icons/*
Comment 4 Jo Shields 2016-02-29 15:33:06 UTC
OK, it looks like I can generate the tarball by using a builder with a newer OS - I'm told that it's a limitation of either the Bash version or libc version in Debian 7 vs. newer distributions, and that matches my local testing.

So I'm going to do a tarball using one of my Ubuntu 14.04 machines (ARM, so slower), but retitle this bug to address the duplicated ALL_FILES problem
Comment 5 Timotheus Pokorra 2016-02-29 20:51:17 UTC
I hope you are right, and it works with Ubuntu 14.04.
I tried with Fedora Rawhide, and I had the same issue.
Comment 6 Arsen.Shnurkov 2016-06-22 06:24:04 UTC
I have this error on Sabayon Linux

$ bash --version
GNU bash, version 4.3.42(1)-release (x86_64-pc-linux-gnu)
Comment 7 Arsen.Shnurkov 2016-06-22 07:11:12 UTC
the same with 
GNU bash, version 4.4.0(1)-rc1 (x86_64-pc-linux-gnu)

to remove duplicates:
http://unix.stackexchange.com/questions/40749
Comment 8 Arsen.Shnurkov 2016-06-22 08:23:37 UTC
variable WILDCARDED_FILES has length ~= 1363753
limit is 131072

because
$ find /usr -iname "binfmts.h" | xargs grep MAX_ARG_STRLEN
/usr/src/linux-4.3.0-sabayon/include/uapi/linux/binfmts.h:#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define PAGE_SIZE 4096
4096 * 32 = 131072

WILDCARDED_FILES has no duplicate filenames in it
Comment 9 Arsen.Shnurkov 2016-06-22 09:18:22 UTC
variable DISTFILES actually contain dupicated icon names.

it is possible to use $^ for deduplication of icon names as in
http://stackoverflow.com/a/33312941/6017919

distdir: $(DISTFILES)
        @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
        topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
        list='$^'; \
          dist_files=`for file in $$list; do echo $$file; done | \

but this will not help, because it's length after deduplication is still higher than the limit
Comment 10 Arsen.Shnurkov 2016-06-22 09:26:55 UTC
Ubuntu Trusty (14) have the same limit
http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/tree/include/uapi/linux/binfmts.h#n8
Comment 12 Arsen.Shnurkov 2016-06-22 17:44:07 UTC
source code which should be fixed:
http://git.savannah.gnu.org/cgit/automake.git/tree/lib/am/distdir.am#n146

i have no idea how to fix it in generic way
Comment 13 Arsen.Shnurkov 2016-06-22 18:08:02 UTC
It is possible to write $^ into file with
https://www.gnu.org/software/make/manual/html_node/File-Function.html#File-Function

i.e. add something like
distdir: $(DISTFILES)
        $(file >list.txt,$^)
        ...
into
https://github.com/mono/monodevelop/blob/master/main/src/core/MonoDevelop.Ide/Makefile.am