fileset with multiple source directories?

[Jacob Kjome]

I have a property that contains a colon separated list of base directories
for sending to javac so that it will compile from multiple source
directories. Now I need to copy any properties or xml files from each and
every one of those source directories to the build directory. However,
Filesets can only point to a single source directory the the copy fails
with the source directory given to the Fileset is this colon separated list
of source directories. Is there any way to do what I need without
resorting to ant-contrib <foreach> and things like that?

So, I have....

in build.properties...

contrib.path=contrib
src.contrib.projects=${contrib.path}/rollbackWithMementoPattern \
:${contrib.path}/jxpath \
:${contrib.path}/socketServer/client \
:${contrib.path}/socketServer/common \
:${contrib.path}/socketServer/example \
:${contrib.path}/socketServer/server \
:${contrib.path}/btree


in build.xml...

<target name="compile.contrib" depends="download.contrib.dependencies,
compile.core"
description="Compile contrib sources" >
<antcall target="-compile.sub">
<param name="srcdir" value="${src.contrib.projects}" />
<param name="includes" value="**/*.java" />
<param name="excludes" value="**/old**/**" />
</antcall>
</target>

<target name="-compile.sub" if="srcdir" >
<javac
srcdir="${srcdir}"
destdir="${build.home}"
includes="${includes}"
excludes="${excludes}"
debug="${build.debug}"
deprecation="${build.deprecation}"
optimize="${build.optimize}"
verbose="${build.verbose}">
<classpath refid="build.classpath" />
</javac>

<!-- Copy application resources -->
<copy todir="${build.home}" >
<fileset
dir="${srcdir}"
excludes="${excludes}"
includes="**/*.properties,
**/*.ini,
**/*.character-sets,
**/*.dtd,
**/*.xsd,
**/*.*ml" />
</copy>
</target>

Obviously the <copy task will fail because the fileset gets a ${srcdir}
property containing a colon separated list of source directories. Can
someone think of a nifty way of using <pathconvert> or something like that
to do what I need here? Is there another approach that will work? Again,
if it involves ant-contrib stuff, I can figure out how to do that just
fine. I'm just looking for a pure ant solution. I want to know whether it
is possible or not before I give up on Ant proper here.

thanks,

Jake

[Sebastien Blanc]

using a pathconvert to flatten ur different filesets and then concatenate the
different resulting properties to get one colom separated list ?
seb.

Post by Jacob Kjome
I have a property that contains a colon separated list of base directories
for sending to javac so that it will compile from multiple source
directories. Now I need to copy any properties or xml files from each and
every one of those source directories to the build directory. However,
Filesets can only point to a single source directory the the copy fails
with the source directory given to the Fileset is this colon separated list
of source directories. Is there any way to do what I need without
resorting to ant-contrib <foreach> and things like that?
So, I have....
in build.properties...
contrib.path=contrib
src.contrib.projects=${contrib.path}/rollbackWithMementoPattern \
:${contrib.path}/jxpath \
:${contrib.path}/socketServer/client \
:${contrib.path}/socketServer/common \
:${contrib.path}/socketServer/example \
:${contrib.path}/socketServer/server \
:${contrib.path}/btree
in build.xml...
<target name="compile.contrib" depends="download.contrib.dependencies,
compile.core"
description="Compile contrib sources" >
<antcall target="-compile.sub">
<param name="srcdir" value="${src.contrib.projects}" />
<param name="includes" value="**/*.java" />
<param name="excludes" value="**/old**/**" />
</antcall>
</target>
<target name="-compile.sub" if="srcdir" >
<javac
srcdir="${srcdir}"
destdir="${build.home}"
includes="${includes}"
excludes="${excludes}"
debug="${build.debug}"
deprecation="${build.deprecation}"
optimize="${build.optimize}"
verbose="${build.verbose}">
<classpath refid="build.classpath" />
</javac>
<!-- Copy application resources -->
<copy todir="${build.home}" >
<fileset
dir="${srcdir}"
excludes="${excludes}"
includes="**/*.properties,
**/*.ini,
**/*.character-sets,
**/*.dtd,
**/*.xsd,
**/*.*ml" />
</copy>
</target>
Obviously the <copy task will fail because the fileset gets a ${srcdir}
property containing a colon separated list of source directories. Can
someone think of a nifty way of using <pathconvert> or something like that
to do what I need here? Is there another approach that will work? Again,
if it involves ant-contrib stuff, I can figure out how to do that just
fine. I'm just looking for a pure ant solution. I want to know whether it
is possible or not before I give up on Ant proper here.
thanks,
Jake
---------------------------------------------------------------------

[Jacob Kjome]

Post by Sebastien Blanc
using a pathconvert to flatten ur different filesets and then concatenate the
different resulting properties to get one colom separated list ?
seb.

Hmm... I'm a bit confused. I'm already starting out with a colon separated
list which is what your solution seems to provide as an end result. Also,
I don't have pre-existing filesets. I am trying to create a fileset that
contains all the files in the various source directories. Or did you mean
create individual filesets and then create one fileset that contains all
those filesets? That may be possible and it is something I
considered. However, the issue is, can I do this with Ant without the help
of something like <foreach> from the ant-contrib package?

Jake

Post by Sebastien Blanc

Post by Jacob Kjome
I have a property that contains a colon separated list of base directories
for sending to javac so that it will compile from multiple source
directories. Now I need to copy any properties or xml files from each and
every one of those source directories to the build directory. However,
Filesets can only point to a single source directory the the copy fails
with the source directory given to the Fileset is this colon separated list
of source directories. Is there any way to do what I need without
resorting to ant-contrib <foreach> and things like that?
So, I have....
in build.properties...
contrib.path=contrib
src.contrib.projects=${contrib.path}/rollbackWithMementoPattern \
:${contrib.path}/jxpath \
:${contrib.path}/socketServer/client \
:${contrib.path}/socketServer/common \
:${contrib.path}/socketServer/example \
:${contrib.path}/socketServer/server \
:${contrib.path}/btree
in build.xml...
<target name="compile.contrib" depends="download.contrib.dependencies,
compile.core"
description="Compile contrib sources" >
<antcall target="-compile.sub">
<param name="srcdir" value="${src.contrib.projects}" />
<param name="includes" value="**/*.java" />
<param name="excludes" value="**/old**/**" />
</antcall>
</target>
<target name="-compile.sub" if="srcdir" >
<javac
srcdir="${srcdir}"
destdir="${build.home}"
includes="${includes}"
excludes="${excludes}"
debug="${build.debug}"
deprecation="${build.deprecation}"
optimize="${build.optimize}"
verbose="${build.verbose}">
<classpath refid="build.classpath" />
</javac>
<!-- Copy application resources -->
<copy todir="${build.home}" >
<fileset
dir="${srcdir}"
excludes="${excludes}"
includes="**/*.properties,
**/*.ini,
**/*.character-sets,
**/*.dtd,
**/*.xsd,
**/*.*ml" />
</copy>
</target>
Obviously the <copy task will fail because the fileset gets a ${srcdir}
property containing a colon separated list of source directories. Can
someone think of a nifty way of using <pathconvert> or something like that
to do what I need here? Is there another approach that will work? Again,
if it involves ant-contrib stuff, I can figure out how to do that just
fine. I'm just looking for a pure ant solution. I want to know whether it
is possible or not before I give up on Ant proper here.
thanks,
Jake
---------------------------------------------------------------------

---------------------------------------------------------------------

[Sebastien Blanc]

Post by Jacob Kjome
Or did you mean
create individual filesets and then create one fileset that contains all
those filesets?

yes, call pathconvert for all filesets (u can create these filesets rite ?) and
incrementally build the property that concatenates all of them to have a final
property containing all the files under all ur dirs:
<path id="path1">
<fileset dir="dir1">
</path>
<path id="path2">
<fileset dir="dir2">
</path>
...
<pathconvert property="p1" refid="path1">
<pathconvert property="p2" refid="path2">
...
assign global property to p1,p2,p3 ....
then convert ur global property into a fileset using the includes attribute.
if do so, u don't need to use the foreach since ur copy task will already be in
fileset format.
also note that if this list is directly the list of subdirs under one dir then u
can compute it dynamically with the same dirset/pathconvert strategy, if not then
hardcode the list (as u wd have to do anyway with for-each)
seb.

[Jacob Kjome]

Ok, but how do I derive the individual directories for each fileset without
looping through the colon-separated list of directories?

So, from...

contrib/subdir1 : contrib/subdir2 : contrib/subdir3

How to I create separate filesets corresponding to each of the dirs
above? Am I missing something? And why the extra <path> elements. The
refid for <pathconvert> is said to take a refid for a <fileset>, so why not
just use that directly by adding an id to each fileset?

I also can't hardcode the <fileset> or <pathconvert> statements. I need to
be able to run this on as many (or as few) paths as there exist
colon-separated directories.

Am I just being very dense today or is your solution not quite matching
what I need?

Jake

Post by Sebastien Blanc

Post by Jacob Kjome
Or did you mean
create individual filesets and then create one fileset that contains all
those filesets?

yes, call pathconvert for all filesets (u can create these filesets rite
?) and
incrementally build the property that concatenates all of them to have a final
<path id="path1">
<fileset dir="dir1">
</path>
<path id="path2">
<fileset dir="dir2">
</path>
...
<pathconvert property="p1" refid="path1">
<pathconvert property="p2" refid="path2">
...
assign global property to p1,p2,p3 ....
then convert ur global property into a fileset using the includes attribute.
if do so, u don't need to use the foreach since ur copy task will already
be in
fileset format.
also note that if this list is directly the list of subdirs under one dir
then u
can compute it dynamically with the same dirset/pathconvert strategy, if
not then
hardcode the list (as u wd have to do anyway with for-each)
seb.
---------------------------------------------------------------------

[Sebastien Blanc]

my bad, I thought this list could be computed through a dirset or fileset.
in such a case, I wd personnally use contrib foreach so I'd let other people try to
answer ur question.

however:
- one ANT way you could go for is to define ur build dependencies into an XML file
and then use xslt to generate ur build.xml file (for instance): this is usefull to
me to re-use these dependencies for javac, EJB deploy, clean in the correct order,
bean verify and so on.
- heaviest and simplest solution beeing to define one property for each subdir - I
know, not much help but looks like this list is a harcoded one and does not rely on
any logic but build dependencies: maybe this could be outputed by a tool that
compiles build depencies (jdepend for instance) and then re-used.

just to be sure, is there really no way for u to compute dynamically this
src.contrib.projects proprety, i.e. isn't this list the list of subdirectories
under ${contrib.path} ? maybe ur pb is the order of the resulting dirset for
compilation dependency so in such a case I understand.

seb.

[Sebastien Blanc]

u can look at Dominique Devienne 's answer on 'project management w/ ANT' thread' for
xml/xslt. no easy way to go but result is efficient.
seb.

DD wrote: "What you describe seems similar in some ways to what I've been doing around
my <subant> task, but it seems to me you still think the Makefile way rather
than the Ant way...

Anyhow, here's what I've done, so you can decide for yourself if you want to
take a similar approach. My goal was to build two dozens projects, all with
dependencies between them. Here's how I did it.

1) Defined an XML file gathering all the information about the projects,
detailed enough to be able to generate a build file for each. The XML file
contains the static dependency information between the projects. Also
defined a schema for that file to validate it before using it.

<target name="validate" depends="init"
description="Ensures dependencies.xml's is correct
(schema-valid)">
<schemavalid file="dependencies.xml"
externalNoNamespaceSchema="dependencies.xsd" />
<echo message="${dependencies} is schema-valid" />
</target>

2) XSLT this file into one build.xml per project (using the Xalan redirect
extension, similarly to the junit-frames.xsl of Ant).

<target name="buildfiles" depends="init, validate"
description="Builds all the modules directories and build files">
<style in="dependencies.xml"
out="build/dependencies-out.xml"
style="dependencies2build.xsl"
processor="trax" />

<style in="dependencies.xml"
out="build/dependencies-bat-out.xml"
style="dependencies2build-bat.xsl"
processor="trax" />
</target>

3) Use my <subant> task to call all the sub-projects build in the right
build/dependency order. The build order is inferred from the XML file
defined in (1) and a specific build path resolver (the <subant> posted in
BugZilla doesn't have this resolver extension I added later, and just builds
in the statically specified <buildpath> order).

<buildpath ident="buildpath"
resolverClassname="com.lgc.buildmagic.TahoeBuildPathResolver">
<param name="destdir" value="${destdir}" />
<param name="dependencies" value="${dependencies}" />
</buildpath>
<property name="buildpath" refid="buildpath" />

4) I have a master build.xml file which re-generates the sub-projects
build.xml files using regular Ant tasks like <uptodate> and normal
dependencies between targets (see dependency on buildfiles target below).
Targets that recurse to the subprojects simply look like this:

<target name="build" depends="buildfiles"
description="Builds all the modules">
<subant buildpathref="buildpath" />
</target>

<target name="jar" depends="buildfiles"
description="Builds the JARs of all modules">
<subant buildpathref="buildpath" />
</target>

In a nutshell, that's it. Note that recursion is limited to one level, and
I'm not trying to decide when to build and where to go (no multi-level
recursion). Everything is statically defined in the main XML file (called
dependencies.xml in my case).

This whole thing took me a long time to build, and evolves a few custom
tasks, but in the end it worked, and is fully Ant (1.5.1) driven. But this
is definitely not for the fainted of heart. My real purpose for building
this monster was to keep internal dependencies between the various 'modules'
of the same (huge) code base in check (too many developer writing too much
code too fast), but it sounds similar enough to your approach that I'm
taking the time to explain it here, in the hope that it might help you
decide how to proceed in your case. Cheers, --DD"

Post by Sebastien Blanc
my bad, I thought this list could be computed through a dirset or fileset.
in such a case, I wd personnally use contrib foreach so I'd let other people try to
answer ur question.
- one ANT way you could go for is to define ur build dependencies into an XML file
and then use xslt to generate ur build.xml file (for instance): this is usefull to
me to re-use these dependencies for javac, EJB deploy, clean in the correct order,
bean verify and so on.
- heaviest and simplest solution beeing to define one property for each subdir - I
know, not much help but looks like this list is a harcoded one and does not rely on
any logic but build dependencies: maybe this could be outputed by a tool that
compiles build depencies (jdepend for instance) and then re-used.
just to be sure, is there really no way for u to compute dynamically this
src.contrib.projects proprety, i.e. isn't this list the list of subdirectories
under ${contrib.path} ? maybe ur pb is the order of the resulting dirset for
compilation dependency so in such a case I understand.
seb.
---------------------------------------------------------------------

[Jacob Kjome]

Post by Sebastien Blanc
my bad, I thought this list could be computed through a dirset or fileset.
in such a case, I wd personnally use contrib foreach so I'd let other
people try to
answer ur question.
- one ANT way you could go for is to define ur build dependencies into an
XML file
and then use xslt to generate ur build.xml file (for instance): this is
usefull to
me to re-use these dependencies for javac, EJB deploy, clean in the
correct order,
bean verify and so on.
- heaviest and simplest solution beeing to define one property for each
subdir - I
know, not much help but looks like this list is a harcoded one and does
not rely on
any logic but build dependencies: maybe this could be outputed by a tool that
compiles build depencies (jdepend for instance) and then re-used.
just to be sure, is there really no way for u to compute dynamically this
src.contrib.projects proprety, i.e. isn't this list the list of subdirectories
under ${contrib.path} ? maybe ur pb is the order of the resulting dirset for
compilation dependency so in such a case I understand.

No, there really isn't. The reason is that the compileable base
directories under contrib can be structured in any way desired. For
instance, here is a sample structure...

contrib/
project1/
client/
server/
example/
project2/

In project1, there are actually 3 separate base directories for
compilation; "client", "server", and "example"). In project2, there is
only one based directory and that is just the "project2" directory
itself. Also, some projects have code that doesn't compile. This is either
because it is just there as a partial reference or because the contributor
is not actively updating the code to compile with the latest changes in the
project. The property in build.properties makes it so we can define all
the base directories of the compilable contrib projects (or sub projects of
contrib projects). So, this information is not attainable via any
particular logical pattern. The nice thing is that I can continue to use
the same <javac> target to compile files with a single srcdir or multiple
srcdirs. It is just this issue of copying all the resources such as
properties file and xml files that is giving me troubles.

The idea about the xml/xsl thing is interesting and I'll take a look at
that, although for this case, it seems a bit heavy-handed. I'm merely
trying to copy stuff from multiple src dirs. It is just too bad that this
seemingly simple task has to be so hard for Ant to do. If filesets did
something similiar to <javac> where it would take a delimited list of src
dirs, this wouldn't be an issue. Seems like this feature ought to get
added into Ant-1.6.

Jake

Post by Sebastien Blanc
seb.
---------------------------------------------------------------------

[Sebastien Blanc]

Post by Jacob Kjome
Seems like this feature ought to get
added into Ant-1.6.

looks like it is the subant target. it was added to the HEAD of ANT a while ago.
does anybody know if this is now available in 1.6 ?

from DD 2:
<buildpath id="buildpath">
<pathelement location="src/lib/utils/utils.xml" />
<pathelement location="src/lib/math/math.xml" />
<pathelement location="src/lib/geobase/geobase.xml" />
<pathelement location="src/lib/lines/lines.xml" />
<pathelement location="src/lib/surfaces/surfaces.xml" />
<pathelement location="src/lib/volumes/volumes.xml" />
<pathelement location="src/lib/tgobjs/tgobjs.xml" />
<pathelement location="src/lib/appli/appli.xml" />
<pathelement location="src/lib/ascii/ascii.xml" />
<pathelement location="src/lib/archive/archive.xml" />
<pathelement location="src/lib/gapi/gapi.xml" />
</buildpath>

<target name="compile">
<subant buildpathref="buildpath" />
</target>