sqlite_authors: add missing expansion

[sqlite-export.git] / README.txt

Exporting SQLite to Git
=======================

As of the [SQLite 3.28.0 release][1] there now exists an official Git
mirror of the SQLite software at <https://github.com/sqlite/sqlite>.

As of the [Fossil 2.9 release][2] on 2019-07-13, a new `fossil git
export` command provides the ability to export a Fossil repository
to git.

This project continues to provide an alternative mechanism to export
a fossil project to Git.  The new fossil 2.9 and later `fossil git export`
command produces a git repository that more-or-less matches one
created by using this project's `--trailer` and `--manifest` options
but without providing any mapping of user IDs to real user names.

Method        | User Names | Options    | Repository
:------------ | :--------- | :--------- | :----------
fossil 2.9    | left as ID | N/A        | FossilOrigin-Name trailers and manifests
sqlite-export | mapped     | <none>     | No notes, trailer lines or manifest files
sqlite-export | mapped     | --notes    | refs/notes/fossil records fossil check-in
sqlite-export | mapped     | --trailer  | FossilOrigin-Name trailer line added
sqlite-export | mapped     | --manifest | manifest and manifest.uuid files added

In fact, any of the three options (`--notes`, `--trailer`, `--manifest`)
may be used with this sqlite-export project in any combination to
produce the desired output repository with whatever "extras" are
desired or not.

Repository                 | Producer      | Extras
:------------------------- | :------------ | :-----
[$GH/sqlite/sqlite][3]     | fossil 2.9+   | no user mapping, always trailers and manifests
[$repo/sqlite][4]          | sqlite-export | users mapped, fossil origin in refs/notes/fossil
[$repo/sqlite-manifest][5] | sqlite-export | users mapped, trailers and manifests (no notes)
[$GH/mackyle/sqlite][6]    | sqlite-export | mirror of [$repo/sqlite][4]

The [$repo/sqlite][4] repository and its [$GH/mackyle/sqlite][6]
mirror are maintained using this project and the `--notes` option.

The [$repo/sqlite-manifest][5] repository is maintained using this
project and both the `--trailer` and `--manifest` options but _not_
the `--notes` option and should be substantially similar to [the
official git mirror of SQLite ($GH/sqlite/sqlite)][3] except that
user ids have been mapped to user names.

[1]: https://www.sqlite.org/releaselog/3_28_0.html
[2]: https://fossil-scm.org/home/doc/trunk/www/changes.wiki#v2_9
[3]: https://github.com/sqlite/sqlite.git
[4]: https://repo.or.cz/sqlite.git
[5]: https://repo.or.cz/sqlite/sqlite-manifest.git
[6]: https://github.com/mackyle/sqlite


Exporting to git with Fossil prior to 2.9
-----------------------------------------

_Reminder_

>  Fossil version 2.9 and later directly supports exporting the SQLite
   fossil repository to git.  There's no reason to use this project
   if that export process meets your needs.  See the [fossil 2.9
   release notes][2] for details.

Theoretically exporting with fossil prior to version 2.9 is as simple as:

  > `fossil export --git | git fast-import`

Unfortunately it doesn't work that way and that's what this project is all
about.


Quick Start
-----------

_Note_

>  Run the `build` script with the `-h` option to see some examples of
   possible arguments.  Any arguments passed to the `build` script are passed
   along to the fossil `configure` script during the build process.  Most
   systems will not require any arguments be passed to the `build` script.

1. Run the script `build` to fetch and build a suitable fossil tool and a
   `git-export-filter` tool.

2. Run the script `import` (maybe with the `--notes` and/or other options)
   to create an `sqlite.git` Git clone of the <https://sqlite.org/src> fossil
   sources.  (May take up to 60 minutes.)

3. See the "Building" section at the bottom of this README to "make" SQLite.


Fossil Issues
-------------

There are two problems with fossil export:

1. fossil versions starting with 1.18 mangle export branch and tag names to
   avoid including characters git does not allow.  The problem is that many
   more characters are mangled than needed so that a tag like `version-1.18`
   is converted to `version_1_18` unnecessarily.

2. fossil versions after 1.18 produce a Git fast-import data stream that
   causes `git fast-import` to fail with a fatal error.


The Tag Problem
---------------

The fossil change that introduces tag mangling is here:

  > <https://www.fossil-scm.org/index.html/info/b707622f29>

It was a well-intentioned change as previously invalid Git names would be
exported, but it went way, way too far.  In fact, the actual Git rules about
allowable characters in names are:

  1. Characters with an ASCII value less than or equal to 32 are not allowed
  2. The character with an ASCII value of 0x7F is not allowed
  3. The characters '~', '^', ':', '\', '*', '?', and '[' are not allowed
  4. The character '/' is a separator and is not allowed within a name
  5. The name may not start with '.' or end with '.'
  6. The name may not end with '.lock'
  7. The name may not contain the '..' sequence
  8. The name may not contain the '@{' sequence
  9. If multiple components are used (separated by '/'), no empty '' components

A patch is included in the file `patches/export_c_patch_diff.txt` that allows
the full diversity of git names to be used and should be applied to the fossil
`src/export.c` file of fossil version 2.1 before building fossil.  It also adds
an optional --notes option to the fossil export --git command that if given
will add a note in the refs/notes/fossil namespace to each commit giving the
original fossil check-in hash for the commit.  Furthermore, it also provides a
new --use-done-feature option (see `git help fast-import`) and makes sure there
aren't any whitespace issues with commit messages by transforming CRLF into
just LF and making sure the only whitespace at the end is a single LF.

There may be updates coming to the official fossil release to address this
name mangling problem, but as of fossil 2.1 they have yet to make it into any
official fossil release.


The Export Problem
------------------

The fossil change that introduces the export problem is here:

  > <https://www.fossil-scm.org/index.html/info/bc8d368b66>

There is even a ticket about this "timewarp" export issue here:

  > <https://www.fossil-scm.org/index.html/info/4013b0a81a>

This issue affects the `sqlite`, `sqlite_docsrc` and `fossil` repositories
making it impossible to export them from fossil and import them into Git with
a current version of fossil.


Let's do the Time Warp Again
----------------------------

The fossil ticket linked to in the above "The Export Problem" section talks
about "timewarps".  These are simply check-ins with a timestamp that is earlier
than at least one of their parents (merges have two parents, most others one).

Fossil doesn't much like these.  The Git fast-import format is a "streamy"
format that, while it allows back references to things earlier in the stream,
does not allow forward references to future, prospective data.  Fossil likes
to output its fast-import stream in check-in date order.  And there you see
the issue.  If a "timewarp" is present then children get put out before their
parents arrive, and Git rudely ends the fast-import operation when this occurs.

All three of the primary fossil repositories (SQLite, SQLite Docs, Fossil) have
at least one "timewarp" in them.

Fossil versions 1.18 and earlier produce a usable fast-import stream not
because it orders the output check-ins correctly in spite of the "timewarp",
but because it outputs all data for each check-in rather than outputting only
differences from the parent(s).  So while the output isn't really correct, it
is accepted by Git and when outside the "timewarp" portion of the history,
the converted Git commits have exactly the correct set of sources, so it's
really not much more than a minor annoyance when reviewing very tiny parts
of older history in the repository.

Starting with fossil version 1.19 this all changed.  Now, whenever possible,
the exported Git fast-import stream only includes "changes" from a check-in's
parent(s).  With a sloppy ordering based only on check-in timestamp and in
the presence of "timewarps", children get put out before their parent(s) arrive
with the ensuing Git rudeness.  While, on the surface, this seems like a good
change (and it brought the ability to do incremental exports), full exports
seem to take somewhat longer overall now.

Then on 2017-02-23, they "shattered it" <https://shattered.it/>.

Shortly thereafter fossil version 2.0 came out supporting additional hash
functions.  And on 2017-03-12 the official SQLite fossil repository got its
first check-in using the new hash function.  Versions of fossil prior to 2.0
cannot deal with these new hash function values.

Now you see the problem.  Fossil version 1.18 can no longer be used (even with
its technically incorrect output) as it cannot understand the new hash values.
But fossil versions 1.19 and later (including 2.0) cannot be used either since
they produce a completely unacceptable fast-import stream in the presence of
any "timewarps".

But, curiosity is a harsh mistress.  The topological ordering problem was
solved even for fossil 1.18 in a satisfactory way some time ago but never
published to avoid causing all the Git refs values to be force-updated.
Correcting the misordering caused by the "timewarps" alters the DAG (directed
acyclic graph) of check-in ancestry and that trickles down to all the children
causing all of their `commit` hash values to change even though the sources they
refer to remain completely unchanged.

As of 2017-03-12 there really isn't a choice anymore.

A GPL version 2 (or later) patch is included to address this in the file
`patches/export_topo_patch_diff.txt` that provides a guaranteed topological
ordering to the exported fast-import stream.  When it's built into fossil,
that version of fossil becomes also covered by the GPL.  The repository data
fossil maintains is unaffected by fossil's license(s) so having a GPL-covered
fossil binary should not really affect anyone.


Fossil Sources
--------------

A .tar.gz archive of the fossil 2.1 sources may be fetched from:

  > <https://fossil-scm.org/index.html/uv/fossil-src-2.1.tar.gz>
  
The downloaded .tar.gz file should have these size and hash values:

  > size:   4802504 bytes  
  > md5:    9f32b23cecb092d42cdf11bf003ebf8d  
  > sha1:   7c7387efb4c0016de6e836dba6f7842246825678  
  > sha256: 85dcdf10d0f1be41eef53839c6faaa73d2498a9a140a89327cfb092f23cfef05

The `archives` subdirectory contains a copy of this .tar.gz file and it will
be used by the `build` script to create a `fossil` executable that reports its
version as `2.1+export ` to confirm that it contains the export fixes.


Git Fast Import Issues
----------------------

The Git fast-import facility does not provide a means to filter the incoming
data stream to adjust user names (fossil export data only includes the user
login name as the email address) nor a means to adjust branch/tag names
(fossil exports a 'trunk' branch where Git expects a 'master' branch and fossil
also exports what are essentially lightweight tags as annotated tags).

To deal with these issues, the `git-export-filter` utility is used.

It can be found at:

  > <https://repo.or.cz/git-export-filter.git>

The included `sqlite_authors` file is used with the `git-export-filter` tool to
supply real user names and email addresses.  Also note that the `sqlite_authors`
file also works for the <https://sqlite.org/docsrc> fossil repository as well.


The Final Solution
------------------

After building a patched version of fossil 2.1 as described above and the
`git-export-filter` utility, a Git repository of the SQLite sources can be
created like so (which is what the `import --notes` script does):

  >     fossil clone https://sqlite.org/src sqlite.fsl
  >     git --git-dir=sqlite.git init --bare
  >     fossil export --git --notes sqlite.fsl |
  >     git-export-filter --authors-file sqlite_authors --require-authors \
  >       --trunk-is-master --convert-tagger tagger |
  >     git --git-dir=sqlite.git fast-import

The above will create the `sqlite.git` Git repository that is a clone of the
SQLite sources from the SQLite fossil respository <https://sqlite.org/src>
(note that only sources are cloned, not tickets or wiki pages or events).

The provided `build` script will attempt to download the necessary sources,
patch them and build suitable `fossil` and `git-export-filter` executable files.
It will pass along any arguments directly to the fossil `configure` script.
Run the `build` script with the `-h` option for examples (most systems will not
require any arguments be passed to the `build` script).

The provided `import` script will then attempt to clone the SQLite sources
and convert them into an `sqlite.git` repository.  It may be run again to update
the `sqlite.git` repository with new changes.  It accepts the `--notes` option
(which is recommended) to enable generation of the `refs/notes/fossil` notes
containing the original fossil check-in hash.  It also accepts the `--trailer`
and `--manifest` options which may be used in any combination with or without
the `--notes` option.

The initial run of the `import` script may take up to 60 minutes on a fast
machine, and subsequent runs of `import` even on a fast machine will still,
unfortunately, take some time.  The CPU will be pounded in either case.

**IMPORTANT**

Options passed to the `import` script are *not* remembered, so make sure to
pass the same options, (e.g. `--notes`) to the `import` script every time it's
run if it's being used to update a previously exported Git repository or you
may end up with out-of-date notes and/or mismatched trailer/manifest commits.


Options
-------

There are new options provided by the patch files for the `fossil export`
command.  As a convenience, they may be given to the `import` script which
will just pass them on to the `fossil export` command.

`--notes`

>   Included with the export tags patch a new fossil export option `--notes` is
>   provided that adds a Git commit note to the `refs/notes/fossil` namespace
>   which contains the original fossil check-in hash for each fossil checkin
>   exported to Git.  Use `git log --notes=fossil` to see these notes.

`--trailer`

>   Included with the export tags patch a new fossil export option `--trailer`
>   is provided that adds a "FossilOrigin-Name:" trailer line to each commit
>   created in the git repository that includes the original fossil check-in
>   hash for that commit.

`--manifest`

>   Included with the export tags patch a new fossil export option `--manifest`
>   is provided that causes every commit created in the git repository to
>   include a `manifest` and `manifest.uuid` file.  Use of this option will
>   increase the size of the generated git repository by approximately 25%.

`--use-done-feature`

>   Included with the export tags patch a new fossil export option
>   `--use-done-feature` is provided that includes the `feature done` and
>   `done` commands at the beginning and end respectively of the exported
>   fast-import stream.  This can help avoid partial imports.  See the
>   `git help fast-import` description of the ``--done`` option and the
>   `git help fast-export` description of the ``--use-done-feature`` option.


Building
--------

**QUICKLY**

 1. Clone/checkout the new `sqlite.git` repository into a new working tree

 2. Run the `create-fossil-manifest` script from this repository with the
    current working directory set to the new working tree created in (1)

 3. Now run the `configure` script in the new working tree created in (1)

 4. Now run `make` in the new working tree created in (1)

**DETAILS**

Ideally, simply cloning from the new `sqlite.git` repository would allow one to
then build SQLite by simply using `make` (or `configure` and `make`).

Unfortunately, this is not the case, the make will fail with a message about
no rule to make the files `manifest` and/or `manifest.uuid` unless the
`--manifest` option was passed to the `import` script.

Both the SQLite sources and the Fossil sources require two fossil vcs specific
files to be created (`manifest` and `manifest.uuid`) in order for make to be
successful.  When the `--manifest` option is passed to the `import` script
these files are added to every commit in the generated git repository which
increases the repository size by roughly 25%.

The `manifest.uuid` file simply contains the hash of the current checkout
and while a real `manifest` file contains a bunch of information, the only
thing that need be present is a line containing the UTC ISO date preceded
by 'D '.

The `create-fossil-manifest` script takes care of creating these files and
should be run with the current working directory set to the top-level of the
git clone's working directory if the `--manifest` option was _NOT_ passed to
the `import` script.

Any time the HEAD commit changes, the `create-fossil-manifest` script should
be run to update the `manifest` and `manifest.uuid` files (only if the
`--manifest` option was _NOT_ passed to the `import` script) before next
running make or the output of the `sqlite_source_id()` function will be
incorrect.