Date: Fri, 29 Aug 2003 12:44:02 +0300
Subject: Re: Managing Debian packages with Subversion
From: Fabian Fagerholm <fabbe@paniq.net>
To: debian-mentors@lists.debian.org
<headers and mime junk shortened>

On Thu, 2003-08-28 at 22:24, Fabian Fagerholm wrote:
> How do I manage a new Debian package with Subversion?

Thanks to jcollins@asgardsrealm.net and joeyh@debian.org for their
insight into this issue. I will try to summarize everything in this
reply to myself.

                Managing Debian packages with Subversion

The following assumes you have basic knowledge of Subversion (SVN). If
you need to read up on its installation and usage, please consult the
Subversion Book by Ben Collins-Sussman, Brian W. Fitzpatrick, C. Michael
Pilato, available from the Subversion site at
http://subversion.tigris.org/ or the book's site
http://svnbook.red-bean.com/.

What are we trying to accomplish?

We want to manage Debian packages with Subversion.

In other words, we want to store all the files related to a Debian
"source package" in a Subversion repository, including the upstream
source. When making changes to the packaging, or when updating the
package to a new upstream version, we want to check out our "project"
(our package) from the repository, make the required changes, and then
commit the changes back into the repository. Also, when we are satisfied
with the state of the packaging and want to release a new version, we
want to automate certain steps such as tagging the current state of the
repository with an appropriate name and version number (corresponding to
the debian package version), exporting this tag and building the
package.

Where appropriate, we want to use the "Subversion way" of doing these
things, meaning we want to take full advantage of the features provided
by Subversion.

Choosing your repository structure

While there are several ways of structuring your repository, common
practise is to have trunk/, branches/ and tags/ directories and divide
the content among these. Two questions arise:
      * Should the repository have trunk/, branches/ and tags/ at the
        top level, each containing project directories -- or should the
        top level be divided by project, and each project directory
        contain trunk/, branches/ and tags/?
      * Should there be an additional top-level directory called
        vendor/, containing the upstream source -- or should the
        upstream source be treated as just another branch?

We thus have 2x2 choices:
     A. Use vendor/ for upstream source.
             A. Per-project trunk/, branches/, tags/ and vendor/.
             B. Repository-wide trunk/, branches/, tags/ and vendor/.
     B. Use a regular branch for upstream source.
             A. Per-project trunk/, branches/ and tags.
             B. Repository-wide trunk/, branches/ and tags/.

Choosing one is largely a matter of taste. The repository will look
clean and structured to newcomers if the top-level directory is divided
by project (sub-option A). It might also add to ease of comprehension to
have upstream sources in vendor/ (option A). But there are some
technical considerations if you want to use the svn:externals property
(see the Subversion Book for details) to pull in several project
directories at once. (TODO: explain this further)

Option A.A.: Per-project trunk/, branches/, tags/ and vendor/

The repository will have the following structure:

        projectA/
           trunk/
           branches/
           tags/
           vendor/
        projectB/
           trunk/
           branches/
           tags/
           vendor/
        ...

The idea of this layout is to reflect the logical structure in the
repository. Internally, Subversion makes no distinction between a
directory called trunk and a directory called tags. Only your use of
these directories dictate their meaning. To Subversion, it's all the
same.

Option A.B.: Repository-wide trunk/, branches/, tags/ and vendor/

The repository will have the following structure:

        trunk/
           projectA/
           projectB/
           ...
        branches/
           projectA/
           projectB/
           ...
        tags/
           projectA/
           projectB/
           ...
        vendor/
           projectA/
           projectB/
           ...

The advantage of this might be that the svn:externals property might
work better. (TODO: verify, why)

Option B.A.: Per-project trunk/, branches/ and tags/

The repository will have the following structure:

        projectA/
           trunk/
           branches/
           tags/
        projectB/
           trunk/
           branches/
           tags/
        ...

Here, the logical structure is reflected in the repository, but the
upstream sources are considered just another branch. Vendor tags are
considered just another set of tags. This approach is fine, but there is
no technical reason to do so. Subversion doesn't care.

Option B.B.: Repository-wide trunk/, branches/ and tags/

The repository will have the following structure:

        trunk/
           projectA/
           projectB/
           ...
        branches/
           projectA/
           projectB/
           ...
        tags/
           projectA/
           projectB/
           ...

As in B.A., upstream sources and tags are not treated specially. This
might also work better with svn:externals. (TODO: check, why)

As noted before, the choice is largely a matter of taste. However, take
into account that Subversion doesn't care how you structure your
repository. The only tehcnical consideration is when using the
svn:externals property. (TODO: verify this and add the explanation
above, then adjust this to reflect the explanation) In the following, we
will simply refer to the option name (A.A., A.B., etc.) when explaining
the differences between the options.

Populating your project

When starting to package a new Debian package, the following steps are
performed:
      * Download the upstream source and unpack it.
      * Enter the upstream source directory and import it (svn import)
        into the appropriate directory:
              * A.A. projectX/vendor/current
              * A.B. vendor/projectX/current
              * B.A. projectX/branches/upstream
              * B.B. branches/projectX/upstream
      * Tag the state of what you just imported (svn copy):
              * A.A. projectX/vendor/<version>
              * A.B. vendor/projectX/<version>
              * B.A. projectX/tags/upstream/<version>
              * B.B. tags/projectX/upstream/<version>
      * Populate trunk/ from the imported upstream source (svn copy):
              * A.A. projectX/trunk
              * A.B. trunk/projectX
              * B.A. projectX/trunk
              * B.B. trunk/projectX
      * Check out the trunk you just created into a directory called
        project-<version> (svn co).
      * Debianize by dh_make. Note for people who have just read the New
        Maintainer's Guide: skip the -f flag. If you have the
        environment variable EMAIL set up correctly, you can skip the -e
        flag as well. You just need to run dh_make in the newly created
        directory.
      * Add the new debian/ directory to subversion's control (svn add).
      * Adjust packaging scripts as needed. Don't do any big changes
        yet, you just want to modify the control and README.Debian
        files, and perhaps some of the most obvious files in debian/.
        Try to think about the most initial state of packaging you want
        to start with.
      * Finally, commit the changes (svn commit).

Working with the project

There is nothing special about working with the Debian package from a
Subversion point of view. You just follow these steps:
      * Check out HEAD (the latest version) of your trunk (svn co) or
        update the trunk directory (svn update).
      * Make the required changes.
      * Commit (svn commit).
Try to structure the changes. Think about what you are going to do, and
define one task. Carry out the task, then commit. Pick another task,
carry it out, commit. And so on. This way, each commit will take the
package from one defined state to the next, and the commit changelog
will help you find the change that broke your package. You'll be able to
say "I made a mistake in revision 123, I'll have to undo it". Then you
can go back to the state before the mistake was made, and undo it in the
next revision. Several Subversion commands help you here -- see the
Subversion book for details.

Updating to a new upstream version

Having populated your repository in the manner described above, the
following procedure will allow you to move your packaging to a new
upstream version, resolving any conflicts between your modified version
and the new upstream version. (This section is largely based on the
Subversion Book)
      * Download the new upstream source and unpack it.
      * Use the svn_load_dirs tool to update the current upstream
        version and tag it (this will be committed immediately):
              * A.A. svn_load_dirs -t <version>
                url://host/repos/projectX/vendor current
                /path/to/new/version
              * A.B. svn_load_dirs -t <version>
                url://host/repos/vendor/projectX current
                /path/to/new/version
              * B.A. svn_load_dirs -t tags/upstream/<version>
                url://host/repos/projectX/ branches/upstream
                /path/to/new/version
              * B.B. svn_load_dirs -t tags/projectX/upstream/<version>
                url://host/repos/ branches/projectX/upstream
                /path/to/new/version
              * (TODO: verify that all of the above are correct)
      * The above might produce conflicts between the versions. You must
        resolve these conflicts, so it may be necessary to familiarize
        yourself with the changes introduces by upstream between the
        versions. (You should do so in any case...)
      * Check out HEAD (the latest version) of your trunk (svn co).
      * Merge the differences between the previous upstream version and
        the current (new) upstream version into your trunk (svn merge):
        svn merge url://to/previous/version url://to/current/version
        trunk
      * You must again resolve any resulting conflicts.
      * Commit the new version (svn commit).

Remember, if you follow the advice below about tagging your releases,
then you have well-defined checkpoints to roll back to if anything goes
wrong.

Making a release

When you are ready to make a release of your package, you should tag the
current state of trunk with the appropriate version number:
      * A.A. projectX/tags/<version>
      * A.B. projectX/tags/<version>
      * B.A. tags/projectX/<version>
      * B.B. tags/projectX/<version>

Then export (svn export) the current tag and build the package. You also
need to export the upstream source and turn it into an .orig.tar.gz.

Jamin W. Collins has provided the following script to automate the build
(TODO: adjust the script to allow for different repository layouts and
make it do the exporting, tarring of .orig.tar.gz, building and cleaning
up all in one sweep. The script currently assumes that trunk is a
directory containing the SVN working copy and that .orig.tar.gz is
provided.)

--8<--
#!/bin/sh

if [ -d trunk ]; then

   read LINE < ./trunk/debian/changelog
 
   PROJ=`echo $LINE | cut -f 1 -d " "`
   VER=`echo $LINE | sed s/'[^(]*(\([^)]*\).*'/'\1'/ | cut -f 1 -d '-'`

   if [ -d "${PROJ}-${VER}" ]; then
      rm -fr "${PROJ}-${VER}"
   fi
  
   svn export trunk ${PROJ}-${VER}
  
   cd ${PROJ}-${VER}
  
   debuild -rfakeroot -us -uc

else

   echo "*** trunk directory does not exist ***"
   exit 1

fi
--8<--

Final words

I would welcome additions and corrections to this document.

Date: Fri, 29 Aug 2003 12:30:46 -0400
From: Joey Hess <joeyh@debian.org>
To: debian-mentors@lists.debian.org
Subject: Re: Managing Debian packages with Subversion
<headers and mime junk shortened>

Fabian Fagerholm wrote:
> Thanks to jcollins@asgardsrealm.net and joeyh@debian.org for their
> insight into this issue. I will try to summarize everything in this
> reply to myself.

Nice job writing this up, here are some corrections.

> The repository will have the following structure:
>=20
>         trunk/
>            projectA/
>            projectB/
>            ...
>         branches/
>            projectA/
>            projectB/
>            ...
>         tags/
>            projectA/
>            projectB/
>            ...
>         vendor/
>            projectA/
>            projectB/
>            ...
>=20
> The advantage of this might be that the svn:externals property might
> work better. (TODO: verify, why)

The reason I prefer this layout is I can check out the whole trunk/
directory here, and get a nice directory with all my debian packages
(projectA, projectB, projectC..) in it. No need for svn:externals at
all.

> Option B.A.: Per-project trunk/, branches/ and tags/
>=20
> The repository will have the following structure:
>=20
>         projectA/
>            trunk/
>            branches/
>            tags/
>         projectB/
>            trunk/
>            branches/
>            tags/
>         ...

Compare with a layout like this one. If I want a directory with all my
projects, I must either check out the whole repository (likely not an
option due to disk spack), or set up a directory with a svn:externals
propery like this:

projectA	svn://wherever/projectA
projectB	svn://wherever/projectB
projectC	svn://wherever/projectC

And maintain this. The maintenance burden, coupled with the limitations
of svn:externals (which some subversion developers don't like and would
prefer to just remove), make me prefer to avoid svn:externals when
possible.

Of course, there are advantages to the toplevel project layouts too. For
one thing, the tags and branches directories are closer to the trunk
directory, which can make it be less typing to tag projects, work with
branches, and so on. Just do a svn info, and paste the repo url,
changing the last word of it from trunk to tags.=20

I've lately been working on tools to automate this kind of thing, for
example I have a svnpath tool that I use to get directories for tags and
branches. It works with either repo layout, like this:

svn cp `svnpath` `svnpath tags`/new-tag

Will make a new tag of the project I'm cd'd to. This is in my subversion
repository in the bin directory along with other tools to do automated
committing and tagging based on the debian/changelog.

>              * B.B. svn_load_dirs -t tags/projectX/upstream/<version>
>                url://host/repos/ branches/projectX/upstream
>                /path/to/new/version

I've only ever had to do this once, and I tried a svn_load_dirs command
like that and watched it try to check out the root of my repository. 10+
gb of files (hundreds of versions of most packages) over dialup. The thing
is very badly documented, but I eventually worked out that I could run
it like this:

svn_load_dirs url://host/repos/branches/projectX upstream /dir

This leaves me doing the tag myself of course. Even when run this way
it apparently checked out all of url://host/repos/branches/projectX/,
which could be quite painful if you have a lot of branches. I hope this
tool gets improved.

> Then export (svn export) the current tag and build the package. You also
> need to export the upstream source and turn it into an .orig.tar.gz.

Please don't do that. Keep an archive of your .orig.tar.gz's and copy
the tarball into place before doing the build so we can have pristine
source. I personally use a local mini-dinstall repository that I upload
all my packages too, that way I have all the released packages handy,
along with their sources. Another approach is the check the orig
tarballs into svn, if you're optimistic about the ever-falling price of
disk space. :-)

