Distributed Version Control even for the smallest team

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,390
Reaction score
577
Points
153
Location
Vienna
Some readers of this blog series might have googled for DVCS, and found one of the IMHO cutest arguments against using a DVCS:
luckyluke.jpg

Still, I've never seen him without this one here, have you?
jollyjumper.jpg

But whatever...

I'm so bold now to proclaim that especially for the lonesome cowboy coder, a DVCS fits the needs. Let me show - as promised - what the last post's workflow looks like with a DVCS. Of course I'm using a specific one here, but at this point you can still substitute it with any other DVCS. The "D", however, is important.

You may remember from last time that the basic action in the workflow is the snapshotting of the project. Let's assume we are clever and hack together a small script that does that for us. Let's call it "commit".
Of course we could hard code "commit" to always zip together our complete Orbiter folder. But that would also zip up all those original Orbiter files, that we don't want to end up in our snapshot. So what to do?
Well, we could script "commit" in such a way that it reads a file with all the files we want to end up in the snapshot. We can then hand-edit this file to add new things to the snapshot. We could also create a script to add things to this file, let's call it "add".
Now we'd need a script to upload and download those ZIP archives to OHM. Sure we could use the browser, but let's just pretend our scripting language supports a HTML transfer easily, and creating such scripts is a piece of cake. So... I suggest we just call those scripts "push" for uploading to OHM, and "pull" for downloading something from OHM.

Yeah. Nice. Some scripts more for getting differences of two snapshots ("diff"), a small script for extracting a specific snapshot ("update"), and maybe something to tag a snapshot with a nice name ("tag"), and we are set.

So what do we have here? Don't be surprised now if I tell you we just created a very rough DVCS.
:eek:rlyflag:

We could polish the scripts now until all possible corner cases that might come up in our workflow are fixed, or we could just download such a bunch of scripts from somebody else who already thought similar about it. And there are many such men and women out there.

One of them is Matt Mackall, the creator of Mercurial(HG). BTW: the short form used for it is not an abbreviation, but the chemical symbol for Mercury(Hg).
Mercurial is written in Python, a scripting language. So you can indeed say that it is nothing but a bunch of scripts to do exactly what I have described above.
Of course it is doing it in a fast, efficient, <your_marketing_adverb> and convenient way. It takes care of differential ZIP creation, it takes care of the file "list", it takes care of the fancy arrows you've seen depicting the relation between archives before.

And it comes with a Windows-friendly GUI called TortoiseHg (well, actually it is the other way around: TortoiseHg comes with Mercurial included). From this tool I've taken screenshots to show you what the last post's workflow looks like with Mercurial.
TorotiseHG_logo.png


Engage...

  1. "Oh, doing this vessel/MFD/scenario/whatever could be a good idea".
    Opens Orbiter folder.
    Browses to /orbitersdk/samples/ or /scenarios/ or /mysubfolderhere/ .
    Initializes repository.
    Starts copying in some examples.
    Hacks around.
    Renames things.
    Goes back and forth from development environment to test in Orbiter.
    thg_init.png
  2. "Cool, looks good! Let's publish it!"

    "Hg adds" all important files.
    "Hg commits" a version.
    "Hg pushes" everything to remote location.
    Posts a thread with "ZOMG must see my add-on".
    Keeps on "hg adding" new things.
    "Hg commits" new versions.
    "Hg pushes" new versions to remote location.
    thg_commit_push.png


    In reality here, you'll also get the "arrows":
    thg_log.png
  3. "Dammit! I can hack on and on, and that stupid ADI-ball is still not working right! Yesterday it at least stayed put, I better use that code again."

    Checks the log of commits and finds "yesterday" by date, or maybe even easier by means of the commit comment.
    "Hg diffs" to current work. No need for safety points, no need for a second temporary folder. You could even use your favorite diff-tool here.
    Identifies the better code.
    "Hg commits" the current work as safe-point.
    "Hg updates" to the better code. No swearing.
    Hacks away.
    "Hg commits" the better version based on old code state.
    "Hg pushes" new versions to remote location.
    thg_diff_update.png

    Yeah, you know it already... those helpful "arrows" are STILL there...
    thg_log2.png
  4. O-F member finds a bug: "I hate to rain on the parade, but your vessel/MFD/scenario 1.2 does the CTD dance."

    "Hg diffs" version "1.2" with current work. It was tagged "1.2", so no need for a change-log file or whatnot.
    Bug is still there. Tough luck!
    "Hg commits" current work
    "Hg updates" to version "1.2"
    Fixes bug. Current features are not ready yet, but they are not present in the 1.2 snapshot, anyway.
    "Hm. Let's just release it as 1.2.1. I'll just commit and tag it here"
    "Hg pushes" new versions to remote location.
    Posts "Bug fixed! Please try"
    O-F member replies "Sorry, but..."
    Rinse. Repeat.
    thg_tag_log3.png


If you compare this here with what I've posted before, you can see some common aspects:

  1. Both are very flexible. You do not have to think ahead in order to lay out structures or anything. Under the hood, DVCS is a scripted snapshotting system to create compressed archives of a complete project at a given time.
  2. Everything is local at first. Only when YOU decide so, it will be transferred to a remote location.
  3. The central point is not something that is given by the system, it is a convention agreed upon by the community you work in. There is no server where you need to have contact to.
  4. Simple tools are used instead of complex setups. With the ZIP-method, you use Windows-Explorer... with HG, you can use Windows-Explorer, too.


But there are also differences:

  1. With DVCS, there is much more automation than manual work ("commit" vs. ZIP)
  2. With DVCS, the important "arrows" for understanding of snapshot relations are visible.
  3. With DVCS, the identification of a snapshot is not an archive file-name, but a hash-code. No need for a stable numbering scheme, the system is doing this for you.
  4. With DVCS, the style and completeness of system information (change-log, where the ZIPs are stored, how old versions are located, etc.) is always the same: the repository format.


I hope I've made it clear now, that using a DVCS even as a single developer even in the smallest possible project is nothing else but using an automated form of what you'd do intuitively manually, anyway. So why making your life more complicated than necessary? It is not like a DVCS costs you huge amounts of money. It is not like a DVCS will need a dedicated server-machine and a home-network. It is not like you'd have to commit yourself to a cloud-based system like SourceForge or even Github or BitBucket.

No. All you have to do is downloading a ca. 25MB MSI, install it, and issue "hg init" every time you start coding.
 

Xyon

Puts the Fun in Dysfunctional
Administrator
Moderator
Orbiter Contributor
Addon Developer
Webmaster
GFX Staff
Beta Tester
Joined
Aug 9, 2009
Messages
6,922
Reaction score
789
Points
203
Location
10.0.0.1
Website
www.orbiter-radio.co.uk
Preferred Pronouns
she/her
You don't have to, but I would wholeheartedly recommend bitbucket for the advantage of keeping your code externally backed up, if nothing else, should the worst happen to your development machine over and over again and you never develop any pattern-recognition....

I like mercurial. A lot. I'm using it in work along with the extensions for git and svn so I can interact with all three (almost) as if they were all mercurial repositories. Life is good with that, it really is, though I kinda miss working on pure mercurial repos when I have to communicate with an SVN backend. I've become an even bigger fan of mercurial after discovering how well its hooks and tags function, and how it branches has always enamoured me towards it.
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,390
Reaction score
577
Points
153
Location
Vienna
Xyon;bt5278 said:
You don't have to, but I would wholeheartedly recommend bitbucket for the advantage of keeping your code externally backed up, if nothing else, should the worst happen to your development machine over and over again and you never develop any pattern-recognition....

This is certainly true, and in fact a DVCS makes that process even more simple than it would be with a centralized version control system (because it is so easy to push to "just another" copy of the repository).

The reason I've put the last statement there is to not shy away beginners with the usage of a cloud service right from start. The point is that there really is no more excuse to stick to the folder/zip method (for add-on development) with DVCS around. You could even stick to your method of backing the system up (and suffer the consequences if the method is called "none at all").

Of course a DVCS immediately opens up the doors for all kind of different workflows once in use, but that is the topic for the next issue of this blog series (that will hopefully not take half a year to be done ;) ).
 

BruceJohnJennerLawso

Dread Lord of the Idiots
Addon Developer
Joined
Apr 14, 2012
Messages
2,585
Reaction score
0
Points
36
Good piece on this Face. I think I may make the move to version control eventually, but... ehh

Its just that its hard to see the use of it at this point. Most of my projects to date are really quite simple when you look under the hood, so I find it hard to motivate for the effort of archiving when there are so few features to archive in the first place.

On the other hand, my second gen projects like the Aquila & the Phoenix will probably drive me to this. Once you start getting into more complex code projects where everything is interrelated, a need for past versions definitely arises, but it all depends on how complex the project is. (and by extension, how many revisions need to be made)
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,390
Reaction score
577
Points
153
Location
Vienna
BruceJohnJennerLawso;bt5280 said:
Its just that its hard to see the use of it at this point. Most of my projects to date are really quite simple when you look under the hood, so I find it hard to motivate for the effort of archiving when there are so few features to archive in the first place.

Well, then I obviously failed to convey my main point to you: that the use of an DVCS makes sense even in the smallest project in the smallest team: the single user, single file case.

It is far better to have a repository for ONE file only instead of copying this file a ten times - only to get lost in versions later on. I even use the command line version of Mercurial on low spec servers for things like single config files, and it always was much nicer to simply do a commit than to have things like "trac.ini.old" or "trac.inibck" for one single INI file in my directory.

As I wrote in the blog entry before this one: version control comes naturally with a file system. The moment you made only a single copy of your single file to "just" have a save-point, you already made version control. And in the very moment you do that manually for that purpose, you've already acted like the Excel user that never used formulas: you've invested time in things that your computer can do many times better for you, if you only let him.

With this in mind, I don't think that there is an add-on consisting of less than a single file. Well, maybe a patch, but even there a DVCS does it many times better than you ;) .

Besides, what effort do you see here? 2 clicks for init, 1 click for starting the dialog, selecting files, enter commit message, 1 click for commit, done. How do you do it with ZIP? Zipping the whole Orbiter directory as often as you have save-points? Adding the files manually to a zip-archive? Just copying the individual files as you go?
I'd say the summary of your manual work will break even the needed "effort" of DVCS in - say - 4 save-points, and it will do it in a way that you actually are able to restore your save-points again without a major headache.
 

HarvesteR

Member
Joined
Apr 22, 2008
Messages
386
Reaction score
15
Points
18
Couldn't agree more. If I had it my way, a DVCS would come pre-installed with the OS, and children would be taught to use it in schools.

I use Git myself, but any will do. Once you learn to use it (it's really not that hard at all), you'll see that it's something every project, no matter how small, should use from the first.

Plus, you never know... That 'small' project you're starting now could very well become a big thing tomorrow!

Cheers
 

BruceJohnJennerLawso

Dread Lord of the Idiots
Addon Developer
Joined
Apr 14, 2012
Messages
2,585
Reaction score
0
Points
36
Face;bt5281 said:
Well, then I obviously failed to convey my main point to you: that the use of an DVCS makes sense even in the smallest project in the smallest team: the single user, single file case.

It is far better to have a repository for ONE file only instead of copying this file a ten times - only to get lost in versions later on. I even use the command line version of Mercurial on low spec servers for things like single config files, and it always was much nicer to simply do a commit than to have things like "trac.ini.old" or "trac.inibck" for one single INI file in my directory.

As I wrote in the blog entry before this one: version control comes naturally with a file system. The moment you made only a single copy of your single file to "just" have a save-point, you already made version control. And in the very moment you do that manually for that purpose, you've already acted like the Excel user that never used formulas: you've invested time in things that your computer can do many times better for you, if you only let him.

With this in mind, I don't think that there is an add-on consisting of less than a single file. Well, maybe a patch, but even there a DVCS does it many times better than you ;) .

Besides, what effort do you see here? 2 clicks for init, 1 click for starting the dialog, selecting files, enter commit message, 1 click for commit, done. How do you do it with ZIP? Zipping the whole Orbiter directory as often as you have save-points? Adding the files manually to a zip-archive? Just copying the individual files as you go?
I'd say the summary of your manual work will break even the needed "effort" of DVCS in - say - 4 save-points, and it will do it in a way that you actually are able to restore your save-points again without a major headache.

Hmmm, yes. But can I integrate my previous, crude system to a DVCS? I currently work each project from a version folder, ie

Shuttle-D Development folder
-Shuttle-D 1.0
-Shuttle-D 1.1
-Shuttle-D 1.2

once I go to release it, I copy over the relevant folder to the delivery memstick (most recent case was 1.1), and I cut away the stuff that doesn't need to be there, like Anim8or files, bitmaps, PDN files, whatever. The release ready version gets tested, zipped, etc. while the original folder stays in my development library so I have back-up records in case of a problem.

So up to this point, it sounds a bit like what you described, except for that I do one big push every release. That would be silly on a complicated project, but release 1.0 to release 1.1 only included a few changes, and I could probably list them off the top of my head right now, so documenting every change would be redundant.

Of course if you're talking about the meshing side of things, I might have need of a VCS. The reason why I dont is that I simply keep anything I create in Anim8or for a particular mesh in the same Anim8or project. That way, if I screw up a mesh royally, I can just click on the backup already open in Anim8or, copy it over and get started again. These methods have worked well enough for me so far, so its been hard to motivate for DVCS without actually experiencing a disaster along the way.

That being said, my projects cant stay simplistic forever, so I should make the switch sooner rather than later. What DVCS do you like the best?
 

Face

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 18, 2008
Messages
4,390
Reaction score
577
Points
153
Location
Vienna
BruceJohnJennerLawso;bt5283 said:
Hmmm, yes. But can I integrate my previous, crude system to a DVCS? I currently work each project from a version folder, ie

Shuttle-D Development folder
-Shuttle-D 1.0
-Shuttle-D 1.1
-Shuttle-D 1.2

once I go to release it, I copy over the relevant folder to the delivery memstick (most recent case was 1.1), and I cut away the stuff that doesn't need to be there, like Anim8or files, bitmaps, PDN files, whatever. The release ready version gets tested, zipped, etc. while the original folder stays in my development library so I have back-up records in case of a problem.

So up to this point, it sounds a bit like what you described, except for that I do one big push every release. That would be silly on a complicated project, but release 1.0 to release 1.1 only included a few changes, and I could probably list them off the top of my head right now, so documenting every change would be redundant.

Of course if you're talking about the meshing side of things, I might have need of a VCS. The reason why I dont is that I simply keep anything I create in Anim8or for a particular mesh in the same Anim8or project. That way, if I screw up a mesh royally, I can just click on the backup already open in Anim8or, copy it over and get started again. These methods have worked well enough for me so far, so its been hard to motivate for DVCS without actually experiencing a disaster along the way.

As I see it, you have already a very sophisticated versioning and release procedure. And it seems you do it all manually!

First, try to distinguish versioning and release packaging. ALL of your content should land in the repository, except big artifacts. The later would be things like generated IntelliSense databases, or debug builds, or maybe huge log files. Everything else, like textures, Animator projects, development PDFs, even binary modules, can land in the repository while you proceed forward.

If you do that, you do not have to clutter up your development folder with things like "Shuttle-D 1.0".

Next, the release packaging - with only the files in the right folders in a ZIP archive - runs parallel to the development. Everytime you want to do a release, you can either do what you did already (ZIP, manipulate, deploy), or you can simply run a release branch, where you merge in current development and do release fine-tuning (removing files that are superfluous, edit readmes, etc), all the while using the ZIP infrastructure that DVCS already offer with the click of a button. That can go to a point where you don't even have to deploy it, because a simple push to a cloud-based repo-copy already IS the deployment, even for release ZIPs.

In the next issue(s), I'm going to demonstrate how I use Mercurial to host an add-on project with Bitbucket. If you can give me a description of your current ZIP archive layout (what the development folder is, where the folder copies live, what files are stored where within those folders), I could add a section to show how such a manual versioning scheme can be converted retroactively to a repository history. I have done that quite often now, most prominently in my OVP repo, where I've converted e.g. Jarmonik's D3D9Client ZIP releases to version control history. Took me a day to gather all the ZIPs, and like 1 hour to convert it to history.

For the meshing process you have: did you never have a situation, where you would really like to see the version you had yesterday, but can't, because you closed all the save-points already and did not copy the project file, because it would have been silly to make another copy to clutter up the directory? I encounter this situation even with scenario files all the time, but for me the solution is easy: every time my work session is finished, I commit. If I need the version tomorrow, or in 2 weeks, I can go back easily, without having my working directory cluttered up with 14 copies of the file. In addition, I always see the current changes in a glimpse, because the difference is presented on commit in a simple way - but of course that only works well with text formats.

BruceJohnJennerLawso;bt5283 said:
What DVCS do you like the best?

Well, this is obvious, I'd say. From my experience, Mercurial in combination with its most prominent GUI frontend - TortoiseHg - is the best candidate for Orbiter add-on development. There are many more DVCS that can be used, but this one in particular is the most user-friendly on Windows, IMHO. Thus I have chosen to use this for my blog series here, although most of the concepts are valid for any DVCS out there.
 

wehaveaproblem

One step closer
Addon Developer
Donator
Joined
May 18, 2008
Messages
913
Reaction score
0
Points
16
Location
London
Website
wehaveaproblem.wordpress.com
Just an extra bonus of having a repo for your project... If you're someone like me who is protraction dev over years. I find it very handy when I come back to the project after time away to know exactly what the latest build is, what the, latest issues were, what was next on the roadmap, what face has done in the meantime etc etc.

For AU, the bitbucket/tortoise combo is invaluable to me. Admittedly AU is a big project with many components, but the theory still stands. It took me a while to get my head round it and face mends all my repo breaking header splitting pushes, but I still recommend it to people whole heartedly. I use it now for personal solo projects outside orbiter, regardless of size.
 
Top