Synchronization with Subversion

From WhyNotWiki

Synchronization with Subversion  edit   (Category  edit)

Should I also move everything svn:externals-related in here since that could be considered "synchronization"?

Synchronization with Subversion  edit   (Category  edit)

[edit] How do I best keep track of a dependency on some vendor code (vendor drops)?

You can use svn:externals, if their code is in a public repository, but that system breaks down if you need to apply any custom changes to their code. In that case, you're better off importing a copy of their code into your repository.

The SVN book has a great how-to on a recommended procedure for keeping vendor branches (http://svnbook.red-bean.com/en/1.0/ch07s04.html). It looks like a very viable solution for keeping up-to-date with vendor releases as well as maintaining your own customizations. (It relies on svn merge, not surprisingly, to merge your changes and theirs... this means there is definitely a chance that there will be conflicts that require manual resolution. But hopefully most updates at least will be pain-free.)

excerpt from http://svnbook.red-bean.com/en/1.0/ch07s04.html :

Managing vendor branches generally works like this. You create a top-level directory (such as /vendor) to hold the vendor branches. Then you import the third party code into a subdirectory of that top-level directory. You then copy that subdirectory into your main development branch (for example, /trunk) at the appropriate location. You always make your local changes in the main development branch. With each new release of the code you are tracking you bring it into the vendor branch and merge the changes into /trunk, resolving whatever conflicts occur between your local changes and the upstream changes.

See also: How I put my customized MediaWiki installation under version control and started using vendor drops

[edit] Piston

(Vendor drops/branches, merges from remote repository...)

See Piston

[edit] SVK

http://svk.bestpractical.com/view/About

• svk uses the subversion fs library. the SVN::XD module is an svn_wc (working copy) replacement.
• SVN::XD does not use the .svn for checkout meta-data. it uses Data::Hierarchy for storing per-directory meta-data in a compact way.
• svk uses svm (svn mirror) for retrieving remote repositories (via ra) to a certain point of the local repository.
• ...

http://svk.bestpractical.com/view/SVKAntiFUD

svk is built on top of Subversion, isn't it over-engineered and fragile?

[...] The remaining over-engineered part in Subversion is probably the working-copy management code (libsvn_wc), which svk doesn't use at all. In fact, one of the reasons that SVK was invented was to work around that nasty spaghetti code.

http://svk.bestpractical.com/view/UsingSVKAsARepositoryMirroringSystem

e-Hernick: I have a need for partial decentralization. I have a 2.5GB repository, on a fast server on the Internet. But the connection here isn't that great, and three people here use that repository. Same thing for other users of this repository. They ci and up to that repository to which they've got a [slow] connection. But in the interest of easier management, I don't want to end up with 50 versions of the repository. But 2 or 3 would be a great improvement for a couple of people. You can have multiple people working on the same svk mirror repository, right?

clkao: Sure. At work we do that, because it's 1.3G repository with ~35k revs.

e-Hernick: [...] It's not a code project, it's a media repository. Used by about a dozen people in four locations. Lots of images, lots of media content. I really don't know how we'd have managed that without SVN. But the performance is an issue... When all those people have to update a few hundreds MBs. I'll need to set up and automate SVK as to make it transparent to the users, though. Most of the repository users are not programmers, and they don't know about version control. They use TortoiseSVN, and they've only been taught about very basic operations. Do you think it will be a problem ?

clkao: The only thing to note is not to commit directly to the mirrored path. Commit to local branch, and use svk push/pull.

e-Hernick: Let's take one location. 4 users. They'll all update and commit from the local branch, and nothing will change for them. However, I will have scripts to periodically sync that local branch to the central repository. The same principle will apply at most locations. However, some users will still work directly on the central repository. To avoid adding unnecessary latency, I could trigger the repository sync as a commit hook. So, at first, I'd be using svk as a single repository replicator. Once this simple setup works, I will explore the other possibilities of svk, including moving the programmers to use svk directly. Can this be done ?

clkao: Sure.

[edit] Mirroring a repository on your local machine

If you're running SVK on your local machine [...]...

# set up the local mirror
svk mirror http://repos/somewhere
svk cp //mirror/repos //local/repos

# now use the local branch as an SVN repository
svn co -p file:///home/user/.svk/local/repos repos

# pull new changes from the main repository
svk sync //mirror/repos
svk smerge //mirror/repos //local/repos 
#(update working copy)

# to commit changes from mirror to the main repository:
#(make changes to working copy and commit with SVN or [http://tortoisesvn.tigris.org/ TortoiseSVN])
svk sync //mirror/repos
svk smerge -C //local/repos //mirror/repos
#(resolve conflicts if any)
svk smerge -l //local/repos //mirror/repos

http://svk.bestpractical.com/view/SVKFAQ

[edit] Given an svk repository, do you have to use it via svk, or can you access it using svn?

Short answer: svn programs that only read the repository are working fine; those who actually write in the repository would bypass svk and make it fail (when writing to mirrored repositories, local branches should be ok. See UsingSVKAsARepositoryMirroringSystem).

...

The short answer is: Committing anything to mirrored paths with svn will surely lead to breakage; merging with "svn merge" will cause subsequent "svk smerge" to lost merge history, resulting in possibly duplicated merges; all other operations should be safe. --AutrijusTang)

A.S. Bradbury [3]

If you want to run a local repository, and keep it synced with an external repository, while being able to synchronize them, then you can just do this with SVK:

svk mkdir //local
svk mkdir //mirror
svk mirror REMOTE_PATH DEPOT_PATH
svk sync //mirror/project_name
svk cp //mirror/project_name //local/project_name
svk co ~/work/project_name

In case you’re not sure what the DEPOT_PATH is, that’s where svk stores your repository, you probably want to mirror to //mirror/project_name

Then edit your files, svk ci will commit to the local repository, svk pull will grab any changes from the remote repository, and svk push to push any local changes to the remote repository. It doesn’t offer the convenience of ‘proper’ distributed SCMs, but it’s great for hacking on your laptop, and being able to commit without internet access.

evan wrote about 6 hours later:

A.S., I’m trying to make this work, but I’m having trouble:

chloe:/tmp/svk eweaver$ svk mkdir local path /tmp/svk/local is not a checkout path.

I see that there is something special to the ”//path” syntax, but don’t understand what it is or where SVK intends to secretly store my repositories.

I would like to add some SVK-enabled subdirectories in my existing svn repository, as with the mirror above, so I could push and pull changesets through them to the remote repository. Is this possible with SVK?

A.S. Bradbury wrote about 8 hours later:

SVK works from a single local subversion repository, living at ~/svk/local by default (I believe). This is your “depot”. The // is how you refer to something in your “depot path”. How you manage it is up to you. I find that having //local and //mirror to be useful.

Let’s say that I’ve done svk mkdir //local, I now have a directory called local in my depotpath. I could checkout from it, but it’s empty. I want to import a project I’ve started on, so svk import ~/work/superproject //local/superproject.

This has just made a superproject directory in my repository, and imported the files. I can then check it out with svk co //local/superproject to get a working copy.

I hope this helps explain the fundamentals about the SVK way of thinking. Take a look at svk help intro, and svk help <command> for more information. I don’t think that SVK enabled subdirectories in an existing repository is going to work, but I’m really no expert. I just know enough about it work with SVK for my needs (which are minimal, I don’t even deal with edit conflicts and so on, it’s just handy for disconnected operating and simple svn interoperability).

evan wrote about 10 hours later:

Thanks for the walkthrough. I’m not sure if svk would help me or not. Offline check-in isn’t an issue because I have a bluetooth phone with a data account. I’d like to make an svk mirror on my server, and externalize that in my checkout so that I could push temporary commits and then choose when to move them upstream to Rubyforge, or to my master repository, for that matter.

But that’s pretty complexified, and it seems that svk doesn’t even work with a non-local depot.

[edit] Custom sync script using SVN::Mirror (Perl)

http://blog.evanweaver.com/articles/2007/02/24/how-to-make-a-changeset-preserving-svn-mirror

...

Why not some other tool?

• I couldn’t use svnsync because it requires a separate brand-new repository for every mirror source, and Trac doesn’t support external or multi-repository browsing.
• SVK is too much to learn and too invasive of my normal workflow.
• And Piston doesn’t preserve changesets.

Go Perl.

[edit] SVN::Pusher

http://search.cpan.org/dist/SVN-Pusher/

http://blog.evanweaver.com/articles/2007/02/24/how-to-make-a-changeset-preserving-svn-mirror

I spent some time trying with svm mergeback and also SVN::Pusher. Nothing even partially worked, and every attempt broke my mirror setup.

[edit] SVN::Mirror

http://search.cpan.org/dist/SVN-Mirror/bin/svm

http://blog.evanweaver.com/articles/2007/02/24/how-to-make-a-changeset-preserving-svn-mirror

I spent some time trying with svm mergeback and also SVN::Pusher. Nothing even partially worked, and every attempt broke my mirror setup.

    # the svn repository for svm to use
    % setenv SVMREPOS ~/svm

    # set the path mirror/svn to mirror official subversion trunk
    % svm init mirror/svn http://svn.collab.net/repos/svn/trunk

    # run the actual mirroring
    # flatten the changesets between revision 1 thru 6000
    % svm sync mirror/svn 6000

    # merge back changes in local branch
    % svn cp file://$SVMREPOS/mirror/svn file://$SVMREPOS/svn-local
    # make some changes and then merge back to source repository
    % svm mergeback mirror/svn svn-local

[edit] svnsync

> sudo svnadmin create /var/www/svn/target-test

> sudo echo <<End > /var/www/svn/target-test/hooks/pre-revprop-change
#!/bin/sh
exit 0
End

> sudo chmod +x /var/www/svn/target-test/hooks/pre-revprop-change
> sudo svnsync init file:///var/www/svn/target-test/ https://rake-completion.googlecode.com/svn/
Copied properties for revision 0 (svn:sync-* properties skipped).

[edit] "off-line"/"on-the-road" commits

http://bob.pythonmac.org/archives/2006/09/14/svnsync-mirror-your-svn-repository/

Can i use it as a ‘buffer server’ when I’m on the road and still want to commit different changes into the ‘mirror’ repository and then sync them with the master when i get online? Can concurrent changes happen on the master server? What’s the point exactly ?

for that, you probably want svk (bestpractical.com) though you still explicitly pull/push/sync “through” a local repository in that case, it looks about right for mobile-oriented use…

http://svk.bestpractical.com/view/SVKFAQ

Does svk allow you to make several commits to your local copy of the repository and then be able to merge all the commits back to the main repository (keeping all the log messages)?

Yes - this was its very first feature. However the merge of local changes will be one commit in the remote repository by default, unless you use smerge -I. Meanwhile, smerge -l will bring the logs to the commit made to remote repository.

[edit] Miscellaneous

http://blog.evanweaver.com/articles/2007/02/24/how-to-make-a-changeset-preserving-svn-mirror

It would be nice to be able to make commits to the local repository and push the changes back out to the remote. I spent some time trying with svm mergeback and also SVN::Pusher. Nothing even partially worked, and every attempt broke my mirror setup.

Aliases: Subversion / Synchronization, Subversion / Mirroring

Synchronization  edit   (Category  edit)

Subversion  edit   (Category  edit)