1 @c -*- coding: utf-8; mode: texinfo; -*-
4 @node Build system notes
5 @chapter Build system notes
7 @warning{This chapter is in high flux, and is being run in a
8 @qq{wiki-like} fashion. Do not trust anything you read in this
12 * Build system overview::
13 * Tips for working on the build system::
16 * Building an Ubuntu distro::
20 @node Build system overview
21 @section Build system overview
23 Build system is currently GNU make, with an extra "stepmake" layer
24 on top. Look at files in @file{make/} and @file{stepmake/} and
25 all @file{GNUmakefile}s.
27 There is wide-spread dissatisfaction with this system, and we are
28 considering changing. This would be a huge undertaking (estimated
29 200+ hours). This change will probably involve not using GNU make
30 any more -- but a discussion about the precise build system will
31 have to wait. Before we reach that point, we need to figure out
32 (at least approximately) what the current build system does.
34 Fundamentally, a build system does two things:
38 Constructs command-line commands, for example:
42 --tons --of --options \
45 --more --imperial --and --metric --tons --of --options \
50 If there was a previous build, it decides which parts of the
51 system need to be rebuilt.
55 When I try to do anything in the build system, it helps to remind
56 myself of this. The "end result" is just a series of command-line
57 commands. All the black magick is just an attempt to construct
60 @node Tips for working on the build system
61 @section Tips for working on the build system
73 to the build system files in various places. This will let you
74 track where the program is, in various points of the build.
77 First task: understand how @code{make website} works,
78 @emph{without} the translations. Looking at the english-only
79 website is the best introduction to the build system... it only
80 covers about 5% of the whole thing, but even that will likely take
91 * Building a bibliography::
94 @node Building a bibliography
95 @subsection Building a bibliography
97 Bibliography files contain a list of citations, like this:
101 author = @{Vinci, Albert C.@},
102 title = @{Fundamentals of Traditional Music Notation@},
103 publisher = @{Kent State University Press@},
108 There are a variety of types of citation (e.g. Book (as above),
109 article, publication). Each cited publication has a list of
110 entries that can be used to identify the publication.
111 Bibliograpies are normally stored as files with a .bib
112 extension. One part of the doc-build process is transforming the
113 bibliography information into @code{texinfo} files. The commands
114 to do this are in the @file{GNUmakefile} in the
115 @file{Documentation} directory.
117 A typical line of the makefile to translate a single bibliography
121 $(outdir)/colorado.itexi:
122 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
123 -s $(top-src-dir)/Documentation/lily-bib \
124 -o $(outdir)/colorado.itexi \
125 $(src-dir)/essay/colorado.bib
131 $(outdir)/colorado.itexi:
134 We're making the file @file{colorado.itexi} and so this is the
138 BSTINPUTS=$(src-dir)/essay $(buildscript-dir)/bib2texi \
141 It's in the @file{essay} directory and we want to run the
142 bib2texi.py script against it.
145 -s $(top-src-dir)/Documentation/lily-bib \
148 The style template is @file{lily-bib.bst} and is found in the
149 @file{Documentation} directory.
152 -o $(outdir)/colorado.itexi \
155 The output file in @file{colorado.itexi}.
158 $(src-dir)/essay/colorado.bib
161 The input file is @file{colorado.bib} in the @file{essay}
164 The @code{bib2texi} Python script used to be used with a variety
165 of options, but now is always called using the same options, as
166 above. Its job is to create the file containing the options for
167 @code{bibtex} (the program that actually does the translation),
168 run bibtex, and then clean up some temporary files. Its main
169 "value add" is the creation of the options file, using this code:
172 open (tmpfile + '.aux', 'w').write (r'''
175 \bibstyle@{%(style)s@}
176 \bibdata@{%(files)s@}''' % vars ())
179 The key items are the style file (now always lily-bib for us) and
182 The style file is written in its own specialised language,
183 described to some extent at
186 @uref{http://amath.colorado.edu/documentation/LaTeX/reference/faq/bibtex.pdf}
189 The file @file{lily-bib.bst} also has fairly extensive commenting.
192 @section Website build
194 Start here: @file{make/website.make}
196 Typing make website runs the file @file{GNUmakefile} from the
197 build directory. This only contains 3 lines:
201 include config$(if $(conf),-$(conf),).make
202 include $(configure-srcdir)/GNUmakefile.in
205 The variable @code{depth} is used throughout the make system to
206 track how far down the directory structure the make is. The first
207 include sets lots of variables but doesn't "do" anything. The
208 second runs the file @file{GNUmakefile.in} from the top level
211 This sets another load of variables, and then includes (i.e.
212 immediately runs) @file{stepmake.make} from the @file{make}
213 subdirectory. This sets a load of other variables, does some
214 testing to see if SCONS (another build tool?) is being used, and
215 then runs @file{make/config.make} - which doesn't seem to exist...
217 Next, it runs @file{make/toplevel-version.make}, which sets the
218 version variables for major, minor, patch, stable, development and
219 mypatchlevel (which seems to be used for patch numbers for
220 non-stable versions only?).
222 Next - @file{make/local.make}, which doesn't exist.
224 Then a few more variable and the interesting comment:
227 # Don't try to outsmart us, you puny computer!
228 # Well, UGH. This only removes builtin rules from
231 and then tests to see whether BUILTINS_REMOVED is defined. It
232 appears to be when I run make, and so
233 @file{stepmake/stepmake/no-builtin-rules.make} is run. The
234 comment at the head of this file says:
237 # UGH. GNU make comes with implicit rules.
238 # We don't want any of them, and can't force users to run
242 I've not studied that file at length, but assume it removes all
243 make's build-in rules (e.g. @file{*.c} files are run through the
244 GNU C compiler) - there's a lot of them in here, and a lot of
245 comments, and I'd guess most of it isn't needed.
247 We return to @file{stepmake.make}, where we hit the make rule all:
248 The first line of this is:
251 -include $(addprefix $(depth)/make/,$(addsuffix -inclusions.make, $(LOCALSTEPMAKE_TEMPLATES)))
254 which, when the variables are substituted, gives:
257 ./make/generic-inclusions.make
258 ./make/lilypond-inclusions.make.
261 (Note - according to the make documentation, -include is only
262 different from include in that it doesn't produce any kind of
263 error message when the included file doesn't exist).
265 And the first file doesn't exist. Nor the second. Next:
268 -include $(addprefix $(stepdir)/,$(addsuffix -inclusions.make, $(STEPMAKE_TEMPLATES)))
271 which expands to the following files:
274 /home/phil/lilypond-git/stepmake/stepmake/generic-inclusions.make
275 /home/phil/lilypond-git/stepmake/stepmake/toplevel-inclusions.make
276 /home/phil/lilypond-git/stepmake/stepmake/po-inclusions.make
277 /home/phil/lilypond-git/stepmake/stepmake/install-inclusions.make.
280 One little feature to notice here - these are all absolute file
281 locations - the line prior to this used relative locations. And
282 none of these files exist, either. (Further note - I'm assuming
283 all these lines of make I'm following are autogenerated, but
284 that'll be something else to discover.)
286 Next in @file{stepmake.make}:
289 include $(addprefix $(stepdir)/,$(addsuffix -vars.make, $(STEPMAKE_TEMPLATES)))
295 /home/phil/lilypond-git/stepmake/stepmake/generic-vars.make
296 /home/phil/lilypond-git/stepmake/stepmake/toplevel-vars.make
297 /home/phil/lilypond-git/stepmake/stepmake/po-vars.make
298 /home/phil/lilypond-git/stepmake/stepmake/install-vars.make.
301 Woo. They all exist (they should as there's no - in front of the
302 include). @file{generic-vars.make} sets loads of variables
303 (funnily enough). @file{toplevel-vars.make} is very short - one
304 line commented as @code{# override Generic_vars.make:} and 2 as
309 include $(stepdir)/documentation-vars.make
312 I assume the urg comment refers to the fact that this should
313 really just create more variables, but it actually sends us off to
314 @file{/home/phil/lilypond-git/stepmake/stepmake/documentation-vars.make}.
316 That file is a 3 line variable setting one.
318 @file{po-vars.make} has the one-line comment @code{# empty}, as
319 does @file{install-vars.make}.
321 So now we're back to @file{stepmake.make}.
326 # ugh. need to do this because of PATH :=$(top-src-dir)/..:$(PATH)
327 include $(addprefix $(depth)/make/,$(addsuffix -vars.make, $(LOCALSTEPMAKE_TEMPLATES)))
330 and the include expands to:
333 include ./make/generic-vars.make ./make/lilypond-vars.make.
336 These again set variables, and in some cases export them to allow
337 child @code{make} processes to use them.
339 The final 4 lines of @file{stepmake.make} are:
342 include $(addprefix $(depth)/make/,$(addsuffix -rules.make, $(LOCALSTEPMAKE_TEMPLATES)))
343 include $(addprefix $(stepdir)/,$(addsuffix -rules.make, $(STEPMAKE_TEMPLATES)))
344 include $(addprefix $(depth)/make/,$(addsuffix -targets.make, $(LOCALSTEPMAKE_TEMPLATES)))
345 include $(addprefix $(stepdir)/,$(addsuffix -targets.make, $(STEPMAKE_TEMPLATES)))
348 which expand as follows:
351 include ./make/generic-rules.make ./make/lilypond-rules.make
353 /home/phil/lilypond-git/stepmake/stepmake/generic-rules.make
354 /home/phil/lilypond-git/stepmake/stepmake/toplevel-rules.make
355 /home/phil/lilypond-git/stepmake/stepmake/po-rules.make
356 /home/phil/lilypond-git/stepmake/stepmake/install-rules.make
357 include ./make/generic-targets.make ./make/lilypond-targets.make
359 /home/phil/lilypond-git/stepmake/stepmake/generic-targets.make
360 /home/phil/lilypond-git/stepmake/stepmake/toplevel-targets.make
361 /home/phil/lilypond-git/stepmake/stepmake/po-targets.make
362 /home/phil/lilypond-git/stepmake/stepmake/install-targets.make
365 @file{lilypond-rules.make} is @code{#empty}
366 @file{generic-rules.make} does seem to have 2 rules in it. They
370 $(outdir)/%.ly: %.lym4
371 $(M4) $< | sed "s/\`/,/g" > $@
375 cat $< | sed $(sed-atfiles) | sed $(sed-atvariables) > $@
378 I believe the first rule is for *.ly files, and has a prerequisite
379 that *.lym4 files must be built first. The recipe is @code{m4 |
380 sed "s/\`/,/g" >}. Perhaps someone with more Unix/make knowledge
381 can comment on exactly what the rules mean/do.
383 @file{toplevel-rules.make} is @code{#empty}
384 @file{po-rules.make} is @code{#empty}
385 @file{install-rules.make} is @code{#empty}
386 @file{generic-targets.make} contains 2 lines of comments.
387 @file{lilypond-targets.make} contains only:
390 ## TODO: fail dist or web if no \version present.
392 grep -L version $(LY_FILES)
395 @file{stepmake/generic-targets.make} contains lots of rules - too
396 many to list here - it seems to be the main file for rules. (FWIW
397 I haven't actually found a rule for website: anywhere, although
398 it clearly exists. I have also found that you can display a rule
399 in the terminal by typing, say @code{make -n website}. This is
400 probably common knowledge.
402 @file{stepmake/toplevel-targets.make} adds a load of other (and
403 occasionally the same) rules to the gernric-targets.
405 @file{stepmake/po-targets.make} is rules for po* makes.
407 @file{stepmake/install-targets.make} has rules for local-install*.
409 And that's the end of stepmake.make. Back to
410 @file{GNUmakefile.in}. More some other time.
412 Another alterative approach to understanding the website build
413 would be to redirect @code{make -n website} and @code{make website}
414 to a text file and work through a) what it does and b) where the
415 errors are occurring.
417 Website build includes @ref{Building a bibliography}.
419 @node Building an Ubuntu distro
420 @section Building an Ubuntu distro
424 Install ubuntu, reboot
426 Run all updates, reboot if asked
428 Enable src repos, refresh package lists
430 Install LilyPond build deps:
432 sudo apt-get build-dep lilypond
435 Install git and autoconf:
437 sudo apt-get install git-core gitk autoconf
441 TEST TO SEE WHETHER EVERYTHING WORKS NOW:
444 Use lily-git.tcl to grab source files
446 Go to source dir and do "./autogen.sh" ; make ; make doc
448 If all compiles, move on to iso creation...
453 Download & install "remastersys":
455 http://sourceforge.net/projects/remastersys/
458 Copy lily-git.tcl script file into /etc/skel/
460 Modify /etc/remastersys.conf as desired (change .iso name, default
461 live session username, etc)
463 Remove non-essential desktop software as desired
465 Create iso: sudo remastersys dist
467 New iso is in /home/remastersys/remastersys/
469 Test iso by installing in VM and repeating steps above for
470 getting source files and building lp and docs