Discussion:
p2 repositories ... we can do better
(too old to reply)
David M Williams
2012-03-02 06:05:08 UTC
Permalink
That is, we can do better at educating people how to make a good p2
repository! We have not done that education well. Hopefully this note, with
its links, can help provide the education you need .... then tell your
friends! ... tell your family! :)

This information actually applies to all projects and all releases at
Eclipse, not just the yearly train, but I am sending to cross-project list,
thinking that will educate most projects as a first step. This should be
considered a requirement of correct and effective use of the Eclipse
Foundation Infrastructure so mentors and PMCs should make sure their
projects know about it and making effective use of the infrastructure.

I think everyone knows about pack200, and while sometimes a pain, it can
help reduce bandwidth for faster installs, and I think most projects do a
good job of using that. And that is one of the hardest things to do! ...
which makes me think the current situation is more about education than
amount of effort.

There are two things which, as a whole, we are not doing well.

A. Use p2.mirrorsURL property in artifact repositories (artifacts.jar/xml
files).

See bug 368826. There are many reasons for not using it ... such as lack of
tools, but another reason is that some of us (e.g. me) thought it was
"common knowledge" that you should use it, but of course it is not to
projects that are new to Eclipse. In response, I have added a little
documentation to the main IT Doc [2], beefed up and corrected some of the
community maintained page that describes how to use p2.mirrorsURL [3] and
provided some documentation on how to use a custom Eclipse Application to
set, change, or remove the p2.mirrorsURL, courtesy of the Webtools Releng
project [4]. While an "unofficial" tool, I think most can take advantage of
it, as is. Just practice on some test repos first! :) But this type of tool
can make it easier to fix existing artifact repositories without
hand-editing a 2 Megabyte XML file. But again, practice on some test repos
first! You can make a copy of (only) the artifacts.jar file to some /temp
directory your local machine and "pretend" that is the artifact repository,
to see if it makes the changes as you intend. Then use your scripts on
production machine, being sure to copy original artifacts jar to a safe
location first in case you need to revert, and be sure to test its really
working, afterwards, as is well described on the p2.mirrorsURL wiki[3].

B. Use p2.index at repository locations.

p2 likes metafiles for its metafiles! As if content, artifacts,
composites, jars and xml were not enough, a while back p2 found the need to
use another file, called p2.index to help it figure out a few ambiguous
cases. Since few people need this functionality, it has gone ignored for a
long time but it is finally becoming noticeable, as Eclipse gets larger and
more popular (and, as a victim of its own success, as "p2 update" is
working well enough many people use that now, instead of downloading a
fresh zip file!). The "lucky accident" is that providing the p2.index file
can save p2 a few needless "round trips" to the download server, while it
figures out what is there, and what it should use, even if not truly needed
to resolve ambiguity. The long gory history is documented in bug 347448
[5]. In response, I have added a little documentation to the main IT Doc
[6] and beefed up and corrected the p2.index wiki [7]. The good news is
that it is easy to do. For 99% of you (I am guessing) it is simply a matter
of copying one of two files to your repository locations ... you do need to
pick which goes where (and most of you will need both, one at composite
sites, the other at simple sites). The two to pick from (for these
frequent, common cases) are documented on the wiki [7] (if you find other
common, or, even tricky cases, please add them to the wiki). This is
especially important to have for "composite sites" since p2 normally checks
for that last, but if the p2.index says that location is a composite site,
then that saves p2 several needless "peek attempts" at download server.
Note, recent versions of b3 aggregator [8] produce these p2.index files
automatically, though the default Equinox p2 publisher does not (bug 302909
[9]) (I do not know about other p2 publishers, if there are any).

While all these issues and p2 in general can be made better, faster,
easier, and some day even mow your lawn (over time, if someone commits to
it) we are where we are, right now. So these are two things we
can/should/need to do right now to improve the situation: add p2.index
files to repository directories, and add p2.mirrorsURL property to artifact
repositories.

I know, this is a long note. And even tons more information linked to go
off and read for details. Keep this note for reference; bookmark the links.
Discuss it with your team. I suspect only one person per project needs to
understand it enough to act one it, but the main point is that there are
things that can be done right now (over the new few weeks, month) to
improve our p2 repositories (even old, existing ones -- the stats say even
helios repositories are still getting LOTS of use!) and then we would of
course clearly want to continue to do these good practices for Juno and
Kepler. Maybe one person per project could spend one day on it within the
next 2 to 4 weeks? (And, yes, I know that is asking for a lot of total
effort ... but, you will learn a lot of cool things in the process! ... and
cleaning up and learning these things now will help us be ready for the
Juno release!

Doing these things will give our Eclipse users a little better update and
install experience. To be clear and honest, this is not gong to be like a
50% improvement, or anything ... I suspect it won't be noticeable most of
the time for most users ... but, some users will notice it a lot! if they
can get mirrors closer to their corner of the world (via p2.mirrorsURL
property), and it will be noticeable by many people during those "peak
times" then download.eclipse.org gets flooded by huge numbers of tiny
requests for files that will never exist (such as "peeking" for
content.jar, when p2 could look more directly for compositeContent.jar, if
p2.index was there). [If our webmasters didn't know about p2, they'd
probably think it as a DOS attack :) just kidding, I've no idea what a DOS
attack looks like, and am pretty sure our IT infrastructure is responding
well.]

Three years ago, who would have thought p2 would be a victim of its own
success?! :) I am sure it can be even more successful and more "enterprise
ready", so if anyone is interested, get involved! provide some patches!
make some tools!

As always, feel free to ask if questions ... and feel free to correct any
mistakes I make in my statements or my wiki updates.

Much thanks for your time and efforts.


p2.mirrorsURL related:

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=368826
[2]
http://wiki.eclipse.org/IT_Infrastructure_Doc#Enable_mirrors_.2F_use_mirrorsURL_for_my_p2_repo.3F
[3] http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL
[4] http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties

p2.index related:

[5] https://bugs.eclipse.org/bugs/show_bug.cgi?id=347448
[6]
http://wiki.eclipse.org/IT_Infrastructure_Doc#Include_a_p2.index_file_at_p2_repository_site.3F
[7] http://wiki.eclipse.org/Equinox/p2/p2_index
[8] http://wiki.eclipse.org/Eclipse_b3/aggregator/manual
[9] https://bugs.eclipse.org/bugs/show_bug.cgi?id=302909
Denis Roy
2012-03-02 13:51:02 UTC
Permalink
David,

Thanks for sending this out.

I've noticed that on the release repos you've added a rather large
comment block at the top of the file. This comment block gets
transferred to each client that fetches the file. And since p2 doesn't
ask for compressed content in its headers, the bytes go out full-sized.

If the only purpose is to maintain some notes, I think a p2.index.readme
file would be better, since it would not be transferred.

Just a thought.

Thanks,

Denis
Post by David M Williams
That is, we can do better at educating people how to make a good p2
repository! We have not done that education well. Hopefully this note, with
David M Williams
2012-03-02 17:54:43 UTC
Permalink
Well you are just never satisfied are you? :) Just kidding ... it is a good
idea ... but, for the record, that "rather large comment block" was only 5
lines, consisting of only 250 characters. But, granted, that does triple
the size of the minimum 130 character p2.index file. .... and when
requested 500,000 times -- in one day! -- I can see how that'd add up (if
everyone did it).

Besides fixing those files, I added a note to the wiki at
http://wiki.eclipse.org/Equinox/p2/p2_index#Examples

and opened two related bugs:

Bug 373121 - p2 should use request headers accepting compression from
server
Bug 373122 - p2.index files should have no comments


Thanks for helping us improve,





From: Denis Roy <***@eclipse.org>
To: cross-project-issues-***@eclipse.org,
Date: 03/02/2012 08:52 AM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
Sent by: cross-project-issues-dev-***@eclipse.org



David,

Thanks for sending this out.

I've noticed that on the release repos you've added a rather large
comment block at the top of the file. This comment block gets
transferred to each client that fetches the file. And since p2 doesn't
ask for compressed content in its headers, the bytes go out full-sized.

If the only purpose is to maintain some notes, I think a p2.index.readme
file would be better, since it would not be transferred.

Just a thought.

Thanks,

Denis
Post by David M Williams
That is, we can do better at educating people how to make a good p2
repository! We have not done that education well. Hopefully this note, with
Denis Roy
2012-03-02 19:00:37 UTC
Permalink
David, please do not confuse satisfaction (or lack thereof) with
desperation :)

Should anyone thing I'm blowing this out of proportion -- Yesterday
(March 1) we received 6,017,804 requests for various p2.index files
(including 1,765,385 under /releases) If each file contains 250 extra
bytes for comments, that add 1.4 GB of internet transfer for eclipse.org
PER DAY.

250 bytes is small, but when you scale it up to eclipse.org proportions,
things start getting real.

I wish p2 would just look for one file...
Post by David M Williams
Well you are just never satisfied are you? :) Just kidding ... it is a good
idea ... but, for the record, that "rather large comment block" was only 5
lines, consisting of only 250 characters. But, granted, that does triple
the size of the minimum 130 character p2.index file. .... and when
requested 500,000 times -- in one day! -- I can see how that'd add up (if
everyone did it).
Besides fixing those files, I added a note to the wiki at
http://wiki.eclipse.org/Equinox/p2/p2_index#Examples
Bug 373121 - p2 should use request headers accepting compression from
server
Bug 373122 - p2.index files should have no comments
Thanks for helping us improve,
Date: 03/02/2012 08:52 AM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
David,
Thanks for sending this out.
I've noticed that on the release repos you've added a rather large
comment block at the top of the file. This comment block gets
transferred to each client that fetches the file. And since p2 doesn't
ask for compressed content in its headers, the bytes go out full-sized.
If the only purpose is to maintain some notes, I think a p2.index.readme
file would be better, since it would not be transferred.
Just a thought.
Thanks,
Denis
Post by David M Williams
That is, we can do better at educating people how to make a good p2
repository! We have not done that education well. Hopefully this note,
with
_________________
Marcel Bruch
2012-03-03 09:08:16 UTC
Permalink
I'm evaluating how to make our repositories comply to these needs with our tycho build. Maybe some successful tycho user or tycho committer can comment on how to support pack200, p2.index files, and p2.mirrorsURL?

To me it sounds like these configuration options should be part of the repository packaging but at least I haven't found them. Are there any options to "switch on" these things? Or is there a set of plug-ins to use that will do the trick?

Thanks,
Marcel
Jesse McConnell
2012-03-03 13:02:08 UTC
Permalink
the eclipse-signing-maven-plugin takes care of some of it but reading
over the original mail I sort of glazed over and didn't really pick up
on where else it ought to have helped address things...

so from my view the index and mirror files goop is stuff in tycho land

if anyone picks up on a shortcoming of the signing plugin do chirp up
and let us know...past that igor will likely know more on the p2
details, I still find it a bit silly. :)

cheers,
jesse

--
jesse mcconnell
Post by Marcel Bruch
I'm evaluating how to make our repositories comply to these needs with our tycho build. Maybe some successful tycho user or tycho committer can comment on how to support pack200, p2.index files, and p2.mirrorsURL?
To me it sounds like these configuration options should be part of the repository packaging but at least I haven't found them. Are there any options to "switch on" these things? Or is there a set of plug-ins to use that will do the trick?
Thanks,
Marcel
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Denis Roy
2012-03-05 15:07:08 UTC
Permalink
Post by Jesse McConnell
if anyone picks up on a shortcoming of the signing plugin do chirp up
and let us know...past that igor will likely know more on the p2
details, I still find it a bit silly. :)
Jesse, I'm a bit intrigued by that last part -- could you elaborate?

Denis
Post by Jesse McConnell
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
I'm evaluating how to make our repositories comply to these needs with our tycho build. Maybe some successful tycho user or tycho committer can comment on how to support pack200, p2.index files, and p2.mirrorsURL?
To me it sounds like these configuration options should be part of the repository packaging but at least I haven't found them. Are there any options to "switch on" these things? Or is there a set of plug-ins to use that will do the trick?
Thanks,
Marcel
Jesse McConnell
2012-03-06 01:36:45 UTC
Permalink
please don't let my 'silly' comment derail the conversation here...it
was perhaps a careless word I threw out

anything that improves peoples experience with p2 would be a good
thing because right now for me it is still this black box that is a
bit hit or miss in terms of what is in it, what comes out of it, and
how its all organized

I am sure people can say the same about maven central, but 5 minutes,
a web browser and a bit of investigative spirit and clicking around
can sort that out...

by about the 5th jar file I have cracked open and sorting through
MANIFEST.MF files...that is where the eye glazing comes in from my
previous statement :)

so by all means please continue improving p2 repositories! and if the
eclipse-signing-maven-plugin is supposed to be doing anything from
this conversation please bring that to my attention directly!

cheers,
jesse

--
jesse mcconnell
Post by Jesse McConnell
the eclipse-signing-maven-plugin takes care of some of it but reading
over the original mail I sort of glazed over and didn't really pick up
on where else it ought to have helped address things...
so from my view the index and mirror files goop is stuff in tycho land
if anyone picks up on a shortcoming of the signing plugin do chirp up
and let us know...past that igor will likely know more on the p2
details, I still find it a bit silly. :)
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
I'm evaluating how to make our repositories comply to these needs with our tycho build. Maybe some successful tycho user or tycho committer can comment on how to support pack200, p2.index files, and p2.mirrorsURL?
To me it sounds like these configuration options should be part of the repository packaging but at least I haven't found them. Are there any options to "switch on" these things? Or is there a set of plug-ins to use that will do the trick?
Thanks,
Marcel
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Ian Bull
2012-03-06 05:59:21 UTC
Permalink
Sorry, a little late to this game, but another small thing. While David
mentioned the use of packed jars, please keep the unpacked jars around too.
We have found that the Java 7 pack/un-packer has changed (compared to
previous versions), and we are having trouble processing pack.gz files on
Java 7 Runtimes. If the un-packed jars are still around, p2 can fall back
to these.

Cheers,
Ian

On Mon, Mar 5, 2012 at 5:36 PM, Jesse McConnell
Post by Jesse McConnell
please don't let my 'silly' comment derail the conversation here...it
was perhaps a careless word I threw out
anything that improves peoples experience with p2 would be a good
thing because right now for me it is still this black box that is a
bit hit or miss in terms of what is in it, what comes out of it, and
how its all organized
I am sure people can say the same about maven central, but 5 minutes,
a web browser and a bit of investigative spirit and clicking around
can sort that out...
by about the 5th jar file I have cracked open and sorting through
MANIFEST.MF files...that is where the eye glazing comes in from my
previous statement :)
so by all means please continue improving p2 repositories! and if the
eclipse-signing-maven-plugin is supposed to be doing anything from
this conversation please bring that to my attention directly!
cheers,
jesse
--
jesse mcconnell
Post by Jesse McConnell
the eclipse-signing-maven-plugin takes care of some of it but reading
over the original mail I sort of glazed over and didn't really pick up
on where else it ought to have helped address things...
so from my view the index and mirror files goop is stuff in tycho land
if anyone picks up on a shortcoming of the signing plugin do chirp up
and let us know...past that igor will likely know more on the p2
details, I still find it a bit silly. :)
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
I'm evaluating how to make our repositories comply to these needs with
our tycho build. Maybe some successful tycho user or tycho committer can
comment on how to support pack200, p2.index files, and p2.mirrorsURL?
Post by Jesse McConnell
Post by Marcel Bruch
To me it sounds like these configuration options should be part of the
repository packaging but at least I haven't found them. Are there any
options to "switch on" these things? Or is there a set of plug-ins to use
that will do the trick?
Post by Jesse McConnell
Post by Marcel Bruch
Thanks,
Marcel
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
--
R. Ian Bull | EclipseSource Victoria | +1 250 477 7484
http://eclipsesource.com | http://twitter.com/eclipsesource
Ian Bull
2012-03-06 06:16:28 UTC
Permalink
Here is a link to the bug report:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=361628

cheers,
ian
Post by Ian Bull
Sorry, a little late to this game, but another small thing. While David
mentioned the use of packed jars, please keep the unpacked jars around too.
We have found that the Java 7 pack/un-packer has changed (compared to
previous versions), and we are having trouble processing pack.gz files on
Java 7 Runtimes. If the un-packed jars are still around, p2 can fall back
to these.
Cheers,
Ian
Post by Jesse McConnell
please don't let my 'silly' comment derail the conversation here...it
was perhaps a careless word I threw out
anything that improves peoples experience with p2 would be a good
thing because right now for me it is still this black box that is a
bit hit or miss in terms of what is in it, what comes out of it, and
how its all organized
I am sure people can say the same about maven central, but 5 minutes,
a web browser and a bit of investigative spirit and clicking around
can sort that out...
by about the 5th jar file I have cracked open and sorting through
MANIFEST.MF files...that is where the eye glazing comes in from my
previous statement :)
so by all means please continue improving p2 repositories! and if the
eclipse-signing-maven-plugin is supposed to be doing anything from
this conversation please bring that to my attention directly!
cheers,
jesse
--
jesse mcconnell
Post by Jesse McConnell
the eclipse-signing-maven-plugin takes care of some of it but reading
over the original mail I sort of glazed over and didn't really pick up
on where else it ought to have helped address things...
so from my view the index and mirror files goop is stuff in tycho land
if anyone picks up on a shortcoming of the signing plugin do chirp up
and let us know...past that igor will likely know more on the p2
details, I still find it a bit silly. :)
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
I'm evaluating how to make our repositories comply to these needs with
our tycho build. Maybe some successful tycho user or tycho committer can
comment on how to support pack200, p2.index files, and p2.mirrorsURL?
Post by Jesse McConnell
Post by Marcel Bruch
To me it sounds like these configuration options should be part of the
repository packaging but at least I haven't found them. Are there any
options to "switch on" these things? Or is there a set of plug-ins to use
that will do the trick?
Post by Jesse McConnell
Post by Marcel Bruch
Thanks,
Marcel
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
--
R. Ian Bull | EclipseSource Victoria | +1 250 477 7484
http://eclipsesource.com | http://twitter.com/eclipsesource
--
R. Ian Bull | EclipseSource Victoria | +1 250 477 7484
http://eclipsesource.com | http://twitter.com/eclipsesource
Matthias Sohn
2012-03-06 07:14:35 UTC
Permalink
Post by Jesse McConnell
please don't let my 'silly' comment derail the conversation here...it
was perhaps a careless word I threw out
anything that improves peoples experience with p2 would be a good
thing because right now for me it is still this black box that is a
bit hit or miss in terms of what is in it, what comes out of it, and
how its all organized
I am sure people can say the same about maven central, but 5 minutes,
a web browser and a bit of investigative spirit and clicking around
can sort that out...
by about the 5th jar file I have cracked open and sorting through
MANIFEST.MF files...that is where the eye glazing comes in from my
previous statement :)
so by all means please continue improving p2 repositories! and if the
eclipse-signing-maven-plugin is supposed to be doing anything from
this conversation please bring that to my attention directly!
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
--
Matthias
Jesse McConnell
2012-03-06 12:04:50 UTC
Permalink
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like? I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho

the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)

cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Marcel Bruch
2012-03-06 17:25:45 UTC
Permalink
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =

According to http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL just add:

<property name="p2.mirrorsURL" value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>

Since webmaster thinks that we have been hit by this issue recently (https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think even more about how to integrate this into our builds. As last means of resort I'll write a bash script that unzips artifacs.jar, adds the property to the artifacts.xml, and zips the file again.

But I wonder how much effort it takes to add this in Tycho's eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.


= Adding support for p2.index =

The file looks like this:

version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!

Whether it's "xml" or "jar" should depend on the "compress" property we already specify in the eclipse-repository.


= Enabling download stat in your repository =

And if we are already on defining properties: to enable download stats it's...

...for artifacts.xml/repository:
<property name='p2.statsURI' value='http://your.stats.server/stats'/>

...for bundles:
<property name='download.stats' value='test.plugin1.bundle'/>

http://wiki.eclipse.org/Equinox_p2_download_stats




So, in theory it's just adding properties and looks from outside like a simple thing to do. But how long it takes to implement it - at least the p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows better?
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
Eclipse Code Recommenders:
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
Jesse McConnell
2012-03-06 17:35:30 UTC
Permalink
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.

I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it

cheers,
jesse

--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently (https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think even more about how to integrate this into our builds. As last means of resort I'll write a bash script that unzips artifacs.jar, adds the property to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
 version = 1
 metadata.repository.factory.order = compositeContent.xml,\!
 artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a simple thing to do. But how long it takes to implement it - at least the p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows better?
Post by Jesse McConnell
 I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch
 g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
David M Williams
2012-03-06 18:10:00 UTC
Permalink
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See

http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties

Feel free to use/copy that as you'd like.

The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.

Good luck,





From: Jesse McConnell <***@gmail.com>
To: Cross project issues <cross-project-issues-***@eclipse.org>,
Date: 03/06/2012 12:38 PM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
Sent by: cross-project-issues-dev-***@eclipse.org



Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.

I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it

cheers,
jesse

--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Post by Marcel Bruch
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
Post by Marcel Bruch
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
Post by Marcel Bruch
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
 version = 1
 metadata.repository.factory.order = compositeContent.xml,\!
 artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
Post by Marcel Bruch
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Marcel Bruch
Post by Jesse McConnell
 I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch
 g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Marcel Bruch
2012-03-06 18:31:24 UTC
Permalink
If that code could be easily integrated into a maven-plugin, that would be nice. However, we publish all but the release update sites from the build server. In that case a simple parameterized script added as post build step to hudson would suffice I think (in my case). Is the tool already installed on build.eclipse.org somewhere? I guess some project (WTP?) is already using it, right? Maybe it makes sense to put it on /shared (or whatever conventions exist) and document how to integrate this on a wiki page?

I'd give it try and document it.
Post by David M Williams
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See
http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
Feel free to use/copy that as you'd like.
The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.
Good luck,
Date: 03/06/2012 12:38 PM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Post by Marcel Bruch
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
Post by Marcel Bruch
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
Post by Marcel Bruch
It wouldn't be specify to Eclipse; just a generic support for properties
- I think.
Post by Marcel Bruch
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
Post by Marcel Bruch
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats
it's...
Post by Marcel Bruch
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Marcel Bruch
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
Eclipse Code Recommenders:
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
Jesse McConnell
2012-03-06 18:37:28 UTC
Permalink
having to go to the cli to tweak this stuff is a non-starter, i have a
mechanism leveraging that for the packing process and I dislike that
immensely

anyway, i am already in java code and mucking with the xml fixing
checksums and the leftovers from the packing process so probably
easier to just add a couple of properties for these settings and
inject them into the xml at that point

not ideal but little about the situation is so will make do with what I have

jesse

--
jesse mcconnell
***@gmail.com



On Tue, Mar 6, 2012 at 12:10, David M Williams
Post by David M Williams
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See
http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
Feel free to use/copy that as you'd like.
The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.
Good luck,
Date:   03/06/2012 12:38 PM
Subject:        Re: [cross-project-issues-dev] p2 repositories ... we can do
           better
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Post by Marcel Bruch
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
Post by Marcel Bruch
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
Post by Marcel Bruch
It wouldn't be specify to Eclipse; just a generic support for properties
- I think.
Post by Marcel Bruch
= Adding support for p2.index =
 version = 1
 metadata.repository.factory.order = compositeContent.xml,\!
 artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
Post by Marcel Bruch
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats
it's...
Post by Marcel Bruch
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Marcel Bruch
Post by Jesse McConnell
 I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch
 g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
David Carver
2012-03-06 18:47:19 UTC
Permalink
It looks like some of this stuff should be doable using Tycho 0.14 and
the Tycho P2 Extras plugin. Included is the ability to run the various
Ant Tasks provided by eclipse.

http://wiki.eclipse.org/Tycho/Additional_Tools

Dave
Post by Jesse McConnell
having to go to the cli to tweak this stuff is a non-starter, i have a
mechanism leveraging that for the packing process and I dislike that
immensely
anyway, i am already in java code and mucking with the xml fixing
checksums and the leftovers from the packing process so probably
easier to just add a couple of properties for these settings and
inject them into the xml at that point
not ideal but little about the situation is so will make do with what I have
jesse
--
jesse mcconnell
On Tue, Mar 6, 2012 at 12:10, David M Williams
Post by David M Williams
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See
http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
Feel free to use/copy that as you'd like.
The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.
Good luck,
Date: 03/06/2012 12:38 PM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Post by Marcel Bruch
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
Post by Marcel Bruch
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
Post by Marcel Bruch
It wouldn't be specify to Eclipse; just a generic support for properties
- I think.
Post by Marcel Bruch
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
Post by Marcel Bruch
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats
it's...
Post by Marcel Bruch
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Marcel Bruch
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Jesse McConnell
2012-03-06 18:54:55 UTC
Permalink
Great! then someone that actually uses p2 and this stuff on a regular
basis should do this...maybe even replace the signing plugin all
together since there are eclipse ant plugins for signing and packing,
etc...

it would not break my heart at all to never look at the signing plugin again :)

jesse

--
jesse mcconnell
It looks like some of this stuff should be doable using Tycho 0.14 and the
Tycho P2 Extras plugin.  Included is the ability to run the various Ant
Tasks provided by eclipse.
http://wiki.eclipse.org/Tycho/Additional_Tools
Dave
having to go to the cli to tweak this stuff is a non-starter, i have a
mechanism leveraging that for the packing process and I dislike that
immensely
anyway, i am already in java code and mucking with the xml fixing
checksums and the leftovers from the packing process so probably
easier to just add a couple of properties for these settings and
inject them into the xml at that point
not ideal but little about the situation is so will make do with what I have
jesse
--
jesse mcconnell
On Tue, Mar 6, 2012 at 12:10, David M Williams
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See
http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
Feel free to use/copy that as you'd like.
The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.
Good luck,
Date:   03/06/2012 12:38 PM
Subject:        Re: [cross-project-issues-dev] p2 repositories ... we can do
           better
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties
- I think.
= Adding support for p2.index =
 version = 1
 metadata.repository.factory.order = compositeContent.xml,\!
 artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats
it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
 I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch
 g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
David Carver
2012-03-06 18:56:12 UTC
Permalink
BTW, in case people didn't know. As of 0.14, Tycho fully supports using
p2.inf during p2 repository creation.

http://wiki.eclipse.org/Tycho/Release_Notes/0.14

Dave
Post by David Carver
It looks like some of this stuff should be doable using Tycho 0.14 and
the Tycho P2 Extras plugin. Included is the ability to run the
various Ant Tasks provided by eclipse.
http://wiki.eclipse.org/Tycho/Additional_Tools
Dave
Post by Jesse McConnell
having to go to the cli to tweak this stuff is a non-starter, i have a
mechanism leveraging that for the packing process and I dislike that
immensely
anyway, i am already in java code and mucking with the xml fixing
checksums and the leftovers from the packing process so probably
easier to just add a couple of properties for these settings and
inject them into the xml at that point
not ideal but little about the situation is so will make do with what I have
jesse
--
jesse mcconnell
On Tue, Mar 6, 2012 at 12:10, David M Williams
Post by David M Williams
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See
http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
Feel free to use/copy that as you'd like.
The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.
Good luck,
Date: 03/06/2012 12:38 PM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Post by Marcel Bruch
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
Post by Marcel Bruch
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
Post by Marcel Bruch
It wouldn't be specify to Eclipse; just a generic support for properties
- I think.
Post by Marcel Bruch
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress"
property we
already specify in the eclipse-repository.
Post by Marcel Bruch
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats
it's...
Post by Marcel Bruch
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Marcel Bruch
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
David Carver
2012-03-06 19:14:03 UTC
Permalink
For p2.Mirrors functionality and getting it added to artifacts.xml, one
can use the maven xml plugin, and the p2.xsl file linked from here:

http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#via_site.xml

In addition you can use the Ant tasks, and the previously antrunner
plugin from tycho to run eclispe ant tasks.

Dave
Post by David Carver
BTW, in case people didn't know. As of 0.14, Tycho fully supports
using p2.inf during p2 repository creation.
http://wiki.eclipse.org/Tycho/Release_Notes/0.14
Dave
Post by David Carver
It looks like some of this stuff should be doable using Tycho 0.14
and the Tycho P2 Extras plugin. Included is the ability to run the
various Ant Tasks provided by eclipse.
http://wiki.eclipse.org/Tycho/Additional_Tools
Dave
Post by Jesse McConnell
having to go to the cli to tweak this stuff is a non-starter, i have a
mechanism leveraging that for the packing process and I dislike that
immensely
anyway, i am already in java code and mucking with the xml fixing
checksums and the leftovers from the packing process so probably
easier to just add a couple of properties for these settings and
inject them into the xml at that point
not ideal but little about the situation is so will make do with what I have
jesse
--
jesse mcconnell
On Tue, Mar 6, 2012 at 12:10, David M Williams
Post by David M Williams
I think your eyes glazed over reading my original note before you got to
the part where I said there is already such a tool. See
http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
Feel free to use/copy that as you'd like.
The advantage of having a "stand-alone app" or tool (not only as "part of a
build") is that is allows you to change the properties after the repo is
created, as is sometimes required, after moving or mirroring the repo
elsewhere.
Good luck,
Date: 03/06/2012 12:38 PM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do
better
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="
http://www.eclipse.org/downloads/download.php?file=
{repository_path}&amp;format=xml"/>
Post by Marcel Bruch
Since webmaster thinks that we have been hit by this issue recently (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
Post by Marcel Bruch
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
Post by Marcel Bruch
It wouldn't be specify to Eclipse; just a generic support for properties
- I think.
Post by Marcel Bruch
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
Post by Marcel Bruch
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats
it's...
Post by Marcel Bruch
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Marcel Bruch
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
David Carver
2012-03-06 18:40:24 UTC
Permalink
Honestly, the p2.Mirror URL and other items that get injected into the
artifacts.xml, p2.index, etc. Really need to go into the Tycho P2
repository creation support. The signing plugin shouldn't be the one
doing this stuff.

Dave
Post by Jesse McConnell
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL" value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently (https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think even more about how to integrate this into our builds. As last means of resort I'll write a bash script that unzips artifacs.jar, adds the property to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a simple thing to do. But how long it takes to implement it - at least the p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows better?
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Jesse McConnell
2012-03-06 18:46:14 UTC
Permalink
Dave,

I agree completely, ideally they would be doing the packing as well.

heck ideally we could sign with a key that was signed by the
foundation key for a web of trust and not have to do this build
machine signing kludge

anyway, we can add it to the signing plugin in the short term and hope
tycho handles it later (or they might now, I don't know)

or we can just see if tycho does it now and if not ask them nicely to
add support for it...i don't mess with tycho enough to know what they
do and don't do...I just figured that someone who really uses it would
have found the option and chimed in by now. There is time though, it
will be a while til I can get to the plugin to tweak it so if someone
did the research that would be grand

jesse

--
jesse mcconnell
Post by David Carver
Honestly, the p2.Mirror URL and other items that get injected into the
artifacts.xml, p2.index, etc.  Really need to go into the Tycho P2
repository creation support.   The signing plugin shouldn't be the one doing
this stuff.
Dave
Post by Jesse McConnell
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL"
value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
 version = 1
 metadata.repository.factory.order = compositeContent.xml,\!
 artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Jesse McConnell
 I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch
 g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Denis Roy
2012-03-06 18:48:16 UTC
Permalink
Post by Jesse McConnell
heck ideally we could sign with a key that was signed by the
foundation key for a web of trust and not have to do this build
machine signing kludge
Hey, that sounds like fun. Have you opened a bug? Perhaps even provide
a small clue as to what it is you envision?


Thanks,

Denis
Post by Jesse McConnell
anyway, we can add it to the signing plugin in the short term and hope
tycho handles it later (or they might now, I don't know)
or we can just see if tycho does it now and if not ask them nicely to
add support for it...i don't mess with tycho enough to know what they
do and don't do...I just figured that someone who really uses it would
have found the option and chimed in by now. There is time though, it
will be a while til I can get to the plugin to tweak it so if someone
did the research that would be grand
jesse
--
jesse mcconnell
Post by David Carver
Honestly, the p2.Mirror URL and other items that get injected into the
artifacts.xml, p2.index, etc. Really need to go into the Tycho P2
repository creation support. The signing plugin shouldn't be the one doing
this stuff.
Dave
Post by Jesse McConnell
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL"
value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
Jesse McConnell
2012-03-06 18:51:31 UTC
Permalink
yep, https://bugs.eclipse.org/bugs/show_bug.cgi?id=354756

cheers,
jesse

--
jesse mcconnell
Post by Jesse McConnell
heck ideally we could sign with a key that was signed by the
foundation key for a web of trust and not have to do this build
machine signing kludge
Hey, that sounds like fun.  Have you opened a bug?  Perhaps even provide
a small clue as to what it is you envision?
Thanks,
Denis
Post by Jesse McConnell
anyway, we can add it to the signing plugin in the short term and hope
tycho handles it later (or they might now, I don't know)
or we can just see if tycho does it now and if not ask them nicely to
add support for it...i don't mess with tycho enough to know what they
do and don't do...I just figured that someone who really uses it would
have found the option and chimed in by now.  There is time though, it
will be a while til I can get to the plugin to tweak it so if someone
did the research that would be grand
jesse
--
jesse mcconnell
Post by David Carver
Honestly, the p2.Mirror URL and other items that get injected into the
artifacts.xml, p2.index, etc.  Really need to go into the Tycho P2
repository creation support.   The signing plugin shouldn't be the one doing
this stuff.
Dave
Post by Jesse McConnell
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL"
value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me think
even more about how to integrate this into our builds. As last means of
resort I'll write a bash script that unzips artifacs.jar, adds the property
to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
 version = 1
 metadata.repository.factory.order = compositeContent.xml,\!
 artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property we
already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like a
simple thing to do. But how long it takes to implement it - at least the
p2.mirrorsURL feature - I've no idea. But maybe a tycho committer knows
better?
Post by Jesse McConnell
 I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
 w www.eclipse.org/recommenders
 tw www.twitter.com/marcelbruch
 g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Igor Fedorenko
2012-03-07 13:13:43 UTC
Permalink
p2.Mirror support is unlikely to happen in Tychp. p2.Mirror is about
published repositories and Tycho build simply does not cover repository
publishing.

In theory, artifact signing can be implemented as part of Tycho, but
current Eclipse signing process makes this not possible in practice due
to significant signing time lag.

--
Regards,
Igor
Post by David Carver
Honestly, the p2.Mirror URL and other items that get injected into the
artifacts.xml, p2.index, etc. Really need to go into the Tycho P2
repository creation support. The signing plugin shouldn't be the one
doing this stuff.
Dave
Post by Jesse McConnell
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
<property name="p2.mirrorsURL"
value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue recently
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes me
think even more about how to integrate this into our builds. As last
means of resort I'll write a bash script that unzips artifacs.jar,
adds the property to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for properties - I think.
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress" property
we already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download stats it's...
<property name='p2.statsURI' value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside like
a simple thing to do. But how long it takes to implement it - at
least the p2.mirrorsURL feature - I've no idea. But maybe a tycho
committer knows better?
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to have that
support in tycho
the signing plugin is really just a hack to support this aspect of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Roland Grunberg
2012-03-07 21:58:37 UTC
Permalink
I've filed https://bugs.eclipse.org/bugs/show_bug.cgi?id=373262 on the
p2.mirrorsURL issue and https://bugs.eclipse.org/bugs/show_bug.cgi?id=373445
for the p2.index proposal, at least for tracking these issues.

There's a rough draft on 373262 that attempts to implement the functionality.
--
Roland Grunberg
Aleksandar Kurtakov
2012-03-08 15:20:34 UTC
Permalink
----- Original Message -----
Sent: Wednesday, March 7, 2012 3:13:43 PM
Subject: Re: [cross-project-issues-dev] p2 repositories ... we can do better
p2.Mirror support is unlikely to happen in Tychp. p2.Mirror is about
published repositories and Tycho build simply does not cover
repository
publishing.
Igor, we speak about p2.mirrorUrl which is a property of the p2 repository not about the p2.mirror ant task which is what you seem to speak about.
This is a simple repository property e.g. <property name='p2.mirrorsURL' value='http://www.eclipse.org/downloads/download.php?file=/eclipse/updates/3.6/R-3.6.2-201102101200&amp;format=xml'/> so being able to get the property injected during build time from tycho is definetely the best option.

Alex
In theory, artifact signing can be implemented as part of Tycho, but
current Eclipse signing process makes this not possible in practice
due
to significant signing time lag.
--
Regards,
Igor
Post by David Carver
Honestly, the p2.Mirror URL and other items that get injected into
the
artifacts.xml, p2.index, etc. Really need to go into the Tycho P2
repository creation support. The signing plugin shouldn't be the
one
doing this stuff.
Dave
Post by Jesse McConnell
Not sure for tycho but given some time I could have that in the
signing plugin in an hour or so I would think.
I'll see if I can scrape some time together to get that support
added
in, at least in the interm until tycho could support it
cheers,
jesse
--
jesse mcconnell
On Tue, Mar 6, 2012 at 11:25, Marcel
Post by Marcel Bruch
Post by Jesse McConnell
Post by Matthias Sohn
could the eclipse-signing-maven-plugin provide a parameter to
inject the p2.mirrorsURL property into artifact repositories
and
parameters to generate the p2.index file ?
can you give me a specific example of what that xml (assuming
that
would be in some of the xml metadata) would look like?
= Support for p2.mirrorsURL =
According to http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL
<property name="p2.mirrorsURL"
value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>
Since webmaster thinks that we have been hit by this issue
recently
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=373352) this makes
me
think even more about how to integrate this into our builds. As
last
means of resort I'll write a bash script that unzips
artifacs.jar,
adds the property to the artifacts.xml, and zips the file again.
But I wonder how much effort it takes to add this in Tycho's
eclipse-repository packaging since tycho generates these files?
It wouldn't be specify to Eclipse; just a generic support for
properties - I think.
= Adding support for p2.index =
version = 1
metadata.repository.factory.order = compositeContent.xml,\!
artifact.repository.factory.order = compositeArtifacts.xml,\!
Whether it's "xml" or "jar" should depend on the "compress"
property
we already specify in the eclipse-repository.
= Enabling download stat in your repository =
And if we are already on defining properties: to enable download
stats it's...
<property name='p2.statsURI'
value='http://your.stats.server/stats'/>
<property name='download.stats' value='test.plugin1.bundle'/>
http://wiki.eclipse.org/Equinox_p2_download_stats
So, in theory it's just adding properties and looks from outside
like
a simple thing to do. But how long it takes to implement it - at
least the p2.mirrorsURL feature - I've no idea. But maybe a tycho
committer knows better?
Post by Jesse McConnell
I suspect is
possible but I also think it is probably more appropriate to
have that
support in tycho
the signing plugin is really just a hack to support this aspect
of the
eclipse requirements that is outside of the traditional tycho
workflow...having said that we can always put another hack or
two into
it :)
cheers,
jesse
Post by Matthias Sohn
--
Matthias
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Thanks,
Marcel
--
w www.eclipse.org/recommenders
tw www.twitter.com/marcelbruch
g+ www.gplus.to/marcelbruch
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Stéphane Bouchet
2012-03-16 10:46:28 UTC
Permalink
Hi,

i created some ant scripts to do so using tycho based build. see [1]

however, i also discovered that earlier builds using buckminster had
incorrectly pointing "p2.mirrorsURL" to a wrong location[2].
i don't know if other projects are affected, but Acceleo, ATL, EMF
Compare and EEF were impacted.

hope this will decrease the amount of p2 requests,

cheers,

Stéphane.

[1]http://sbouchet-eef.blogspot.com/2012/03/p2-repositories-we-can-do-better-eef.html
[2]https://bugs.eclipse.org/bugs/show_bug.cgi?id=314388#c69
Post by David M Williams
That is, we can do better at educating people how to make a good p2
repository! We have not done that education well. Hopefully this note, with
its links, can help provide the education you need .... then tell your
friends! ... tell your family! :)
This information actually applies to all projects and all releases at
Eclipse, not just the yearly train, but I am sending to cross-project list,
thinking that will educate most projects as a first step. This should be
considered a requirement of correct and effective use of the Eclipse
Foundation Infrastructure so mentors and PMCs should make sure their
projects know about it and making effective use of the infrastructure.
I think everyone knows about pack200, and while sometimes a pain, it can
help reduce bandwidth for faster installs, and I think most projects do a
good job of using that. And that is one of the hardest things to do! ...
which makes me think the current situation is more about education than
amount of effort.
There are two things which, as a whole, we are not doing well.
A. Use p2.mirrorsURL property in artifact repositories (artifacts.jar/xml
files).
See bug 368826. There are many reasons for not using it ... such as lack of
tools, but another reason is that some of us (e.g. me) thought it was
"common knowledge" that you should use it, but of course it is not to
projects that are new to Eclipse. In response, I have added a little
documentation to the main IT Doc [2], beefed up and corrected some of the
community maintained page that describes how to use p2.mirrorsURL [3] and
provided some documentation on how to use a custom Eclipse Application to
set, change, or remove the p2.mirrorsURL, courtesy of the Webtools Releng
project [4]. While an "unofficial" tool, I think most can take advantage of
it, as is. Just practice on some test repos first! :) But this type of tool
can make it easier to fix existing artifact repositories without
hand-editing a 2 Megabyte XML file. But again, practice on some test repos
first! You can make a copy of (only) the artifacts.jar file to some /temp
directory your local machine and "pretend" that is the artifact repository,
to see if it makes the changes as you intend. Then use your scripts on
production machine, being sure to copy original artifacts jar to a safe
location first in case you need to revert, and be sure to test its really
working, afterwards, as is well described on the p2.mirrorsURL wiki[3].
B. Use p2.index at repository locations.
p2 likes metafiles for its metafiles! As if content, artifacts,
composites, jars and xml were not enough, a while back p2 found the need to
use another file, called p2.index to help it figure out a few ambiguous
cases. Since few people need this functionality, it has gone ignored for a
long time but it is finally becoming noticeable, as Eclipse gets larger and
more popular (and, as a victim of its own success, as "p2 update" is
working well enough many people use that now, instead of downloading a
fresh zip file!). The "lucky accident" is that providing the p2.index file
can save p2 a few needless "round trips" to the download server, while it
figures out what is there, and what it should use, even if not truly needed
to resolve ambiguity. The long gory history is documented in bug 347448
[5]. In response, I have added a little documentation to the main IT Doc
[6] and beefed up and corrected the p2.index wiki [7]. The good news is
that it is easy to do. For 99% of you (I am guessing) it is simply a matter
of copying one of two files to your repository locations ... you do need to
pick which goes where (and most of you will need both, one at composite
sites, the other at simple sites). The two to pick from (for these
frequent, common cases) are documented on the wiki [7] (if you find other
common, or, even tricky cases, please add them to the wiki). This is
especially important to have for "composite sites" since p2 normally checks
for that last, but if the p2.index says that location is a composite site,
then that saves p2 several needless "peek attempts" at download server.
Note, recent versions of b3 aggregator [8] produce these p2.index files
automatically, though the default Equinox p2 publisher does not (bug 302909
[9]) (I do not know about other p2 publishers, if there are any).
While all these issues and p2 in general can be made better, faster,
easier, and some day even mow your lawn (over time, if someone commits to
it) we are where we are, right now. So these are two things we
can/should/need to do right now to improve the situation: add p2.index
files to repository directories, and add p2.mirrorsURL property to artifact
repositories.
I know, this is a long note. And even tons more information linked to go
off and read for details. Keep this note for reference; bookmark the links.
Discuss it with your team. I suspect only one person per project needs to
understand it enough to act one it, but the main point is that there are
things that can be done right now (over the new few weeks, month) to
improve our p2 repositories (even old, existing ones -- the stats say even
helios repositories are still getting LOTS of use!) and then we would of
course clearly want to continue to do these good practices for Juno and
Kepler. Maybe one person per project could spend one day on it within the
next 2 to 4 weeks? (And, yes, I know that is asking for a lot of total
effort ... but, you will learn a lot of cool things in the process! ... and
cleaning up and learning these things now will help us be ready for the
Juno release!
Doing these things will give our Eclipse users a little better update and
install experience. To be clear and honest, this is not gong to be like a
50% improvement, or anything ... I suspect it won't be noticeable most of
the time for most users ... but, some users will notice it a lot! if they
can get mirrors closer to their corner of the world (via p2.mirrorsURL
property), and it will be noticeable by many people during those "peak
times" then download.eclipse.org gets flooded by huge numbers of tiny
requests for files that will never exist (such as "peeking" for
content.jar, when p2 could look more directly for compositeContent.jar, if
p2.index was there). [If our webmasters didn't know about p2, they'd
probably think it as a DOS attack :) just kidding, I've no idea what a DOS
attack looks like, and am pretty sure our IT infrastructure is responding
well.]
Three years ago, who would have thought p2 would be a victim of its own
success?! :) I am sure it can be even more successful and more "enterprise
ready", so if anyone is interested, get involved! provide some patches!
make some tools!
As always, feel free to ask if questions ... and feel free to correct any
mistakes I make in my statements or my wiki updates.
Much thanks for your time and efforts.
[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=368826
[2]
http://wiki.eclipse.org/IT_Infrastructure_Doc#Enable_mirrors_.2F_use_mirrorsURL_for_my_p2_repo.3F
[3] http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL
[4] http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties
[5] https://bugs.eclipse.org/bugs/show_bug.cgi?id=347448
[6]
http://wiki.eclipse.org/IT_Infrastructure_Doc#Include_a_p2.index_file_at_p2_repository_site.3F
[7] http://wiki.eclipse.org/Equinox/p2/p2_index
[8] http://wiki.eclipse.org/Eclipse_b3/aggregator/manual
[9] https://bugs.eclipse.org/bugs/show_bug.cgi?id=302909
_______________________________________________
cross-project-issues-dev mailing list
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Loading...