Software building
This presentation is the property of its rightful owner.
Sponsored Links
1 / 48

Software building PowerPoint PPT Presentation


  • 61 Views
  • Uploaded on
  • Presentation posted in: General

Software building. Tomáš Kalibera, Tomáš Pop. Plan. SW building Requirements Typical scenarios Dependency problems …. Tools Make Ant Make – advanced options Autotools { conf , make, scan,..} Ivy, Maven, SCons …. This lesson. Next lesson.

Download Presentation

Software building

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Software building

Software building

Tomáš Kalibera, Tomáš Pop


Software building

Plan

  • SW building

    • Requirements

    • Typical scenarios

    • Dependency problems

    • ….

  • Tools

    • Make

    • Ant

    • Make – advanced options

    • Autotools{conf, make, scan,..}

    • Ivy, Maven, SCons…

This lesson

Next lesson

Tomáš Kalibera, Tomáš Pop


What is for this lecture software building

What is (for this lecture) software building

  • Transforming software sources (tree of source files) into executable binary code

  • In general, it is simmilar to other transformations of files that are carried out by independent applications, such as

    • Transforming LaTeX document into PDF

    • Re-sizing and publishing a tree of photographs

    • Running functional tests

    • Updating a web site, translating XML source documents into XHTML by XSLT transformer

Tomáš Kalibera, Tomáš Pop


Typical requirements on software building

Typical requirements on software building

Try to find some

Tomáš Kalibera, Tomáš Pop


Typical requirements on software building1

Typical requirements on software building

  • Automation

    • Require no or minimal input from user

    • Run regularly or based on source code modifications

  • Portability

    • Run on all systems supported by the software

    • Do not require (too many) additional tools, compared to the software built

    • Allow programmers to implement/configure building easily for multiple platforms

Tomáš Kalibera, Tomáš Pop


Typical requirements on software building2

Typical requirements on software building

  • Efficiency

    • Do not repeat what has already been done

    • Employ multiple cores if available

  • Robustness

    • Build as much as possible – if there is any error, try to build everything what is not affected by that error

Tomáš Kalibera, Tomáš Pop


Requirements on tools for software building

Requirements on tools for software building

  • Generality

    • The same tool should be usable for a wide range of applications

  • Easy to use

    • Programmers – easy to define how building should be done, extensible

    • End users – easy to build a program with minimal (or ideally, without any) human intervention

Tomáš Kalibera, Tomáš Pop


Use scenarios for build tools

Use scenarios for build tools

Try describe typical scenarios (developer, end user)

Tomáš Kalibera, Tomáš Pop


One shot build from scratch

One-shot build from scratch

  • Usage pattern

    • Download sources including the build tool/configuration

    • Run the tool to build everything from scratch

    • If something goes wrong, fix the problem, delete everything and start again

  • Requirements

    • Portability

    • For simple programs, a (shell) script would probably suffice

    • Sometimes: include tests of resulting binary

Tomáš Kalibera, Tomáš Pop


Requirements on tools for software building1

Requirements on tools for software building

  • Usage pattern

    • Build from scratch

    • Modify some sources and/or configuration

    • Build again, reusing previously built fragments not affected by modifications

  • Requirements

    • Correctness – re-build everything what has to be re-built

    • Efficiency – do not re-build (much) more than that

Tomáš Kalibera, Tomáš Pop


Problems to solved in build tools

Problems to solved in build tools

Try to find some

Tomáš Kalibera, Tomáš Pop


Dependencies

Dependencies

  • Building consists of running (many) external tools operating on files, in correct order

  • Correct order must follow inter-dependencies of the source files

    • i.e. first creating binary object files (.o), then the library (.so, .a)

    • Or, recompiling a source file (.c) when header it depends on changes (.h)

Tomáš Kalibera, Tomáš Pop


Build dependencies in c programs

(Build) Dependencies in C Programs

  • .h C-preprocessor header files

  • Pre-compiled header files

  • .c C source files

  • .o object files

  • .so, .a library files

  • Binary executables

preprocessor

preprocessor

compiler

archiver

linker

Tomáš Kalibera, Tomáš Pop


Dependency detection header deps

Dependency detection – header deps

#include <stdio.h>

#ifdef __LINUX__

#include <unistd.h>

#endif

  • Dependencies can be detected by C preprocessor (typically via the compiler, i.e. gcc –M)

  • Dependencies depend on preprocessor directives

    • This typically maps to different dependencies for different platforms

    • Detectors therefore sometimes only detect a superset of possible dependencies

Tomáš Kalibera, Tomáš Pop


Dependency detection other c deps

Dependency detection – other C deps

  • Mission impossible

    • The information is not explicitly there, developers must provide it

  • Some reverse engineering is still possible, if needed (legacy systems..)

    • Debug information in binary object files (.o) may include source files

    • Object files in a static library (.a) may be listed by their names

    • In theory, listing symbols in executables might help

Tomáš Kalibera, Tomáš Pop


Build dependencies in java programs

(Build) Dependencies in Java Programs

  • .java source files

  • .class bytecode files

  • .jar archiver files

compiler

archiver

This becomes much harder with native code….

Tomáš Kalibera, Tomáš Pop


Dependency detection in java packages

Dependency detection in Java (packages)

import java.util.HashSet;

import org.sdmt.build.JavaDependencyDetector;

  • Dependencies can be detected by (some) Java compilers (i.e. Jikes)

  • Dependencies can be circular

    • This can only be resolved by Java compiler – files forming a dependency cycle have to be compiled together

  • Some of these dependencies can also be detected from compiled .class files

Tomáš Kalibera, Tomáš Pop


Which class files are needed

which .class files are needed ?

  • which .class files are needed ?

  • JAR is in fact a ZIP file

    • Try jar -xf foo.jar or use WinZip

  • Detecting dependencies is easy

    • Listing the files

    • Contains Classpath entry

Tomáš Kalibera, Tomáš Pop


C assemblies

C# Assemblies

  • Assembly (EXE, dll) contains metadata

    • Reference to other assemblies

    • Used by JIT compiler

  • The DLL Hell (Windows XP a Windows 2003 Server)

    • Several versions of a same assembly can be installed at the same time on one computer

    • At app start time Windows looks for application manifest – describes required versions of used assemblies (in application.exe.manifest file or in RT_MANIFEST resource embedded inside executable)

Tomáš Kalibera, Tomáš Pop


Other problems solved in build tools

Other problems solved in build tools

Tomáš Kalibera, Tomáš Pop


Getting the sources

Getting the sources

  • Support for downloading (sources, binary dependencies)

    • HTTP, FTP, SCP, …

    • e. g. Apache Ivy (next lesson)

  • Support for version control systems

    • Checkout/export

    • Or even commit of the changes

Tomáš Kalibera, Tomáš Pop


Packaging

Packaging

  • Creating a software package

    • Including metadata

    • Including documentation

    • Including binaries for a given system…

    • Compression of package content

  • Purpose of the packages

    • Distribution to end users

    • Deployment into other applications (plugins, modules…), i.e. EJB, Eclipse plugins

Tomáš Kalibera, Tomáš Pop


And much more

And much more…

  • Generating documentation

  • Running or creating tests

  • Bugs checking by code analysis

  • Calculating code complexity/good-design metrics

Tomáš Kalibera, Tomáš Pop


Existing tools

Existing tools

Tomáš Kalibera, Tomáš Pop


Existing tools time line

Existing tools - Time-line

1970 (Unix)

2003 (Maven)

1977 (Make)

2010 (Now)

1994 (Autotools)

2000 (Scons, Ant)

Image source: http://commandline.org.uk/python/ohloh-and-popularity-programming-languages-free-and-open-source-software/

Tomáš Kalibera, Tomáš Pop


The non java mostly c world

The non-Java (mostly C) world

  • Make and its derivatives

    • Originally by Stuart Feldman, Bell Labs, 1977

    • GNU Make originally by Richard Stallman and Roland McGrath, 1993

  • GNU Build System (Autotools)

    • Autoconf – configuration detector

    • Automake – makefile generator

    • Libtool – library creation abstraction tool

  • Scons

    • Python

    • Similar functionality as autotools.

Tomáš Kalibera, Tomáš Pop


The java world

The Java world

  • Apache Ant

    • James Duncan Davidson, 2000 (initially intended only for Tomcat)

    • Today de-facto standard for open source Java projects

  • Maven

    • Apache Software Foundation

    • Support for remote repositories with software components to build

  • Continuum, Cruise Control

    • Tools for continuous building

Tomáš Kalibera, Tomáš Pop


Software building

Make

Tomáš Kalibera, Tomáš Pop


Make overview

Make overview

  • Expert system

    • On input it gets a set of rules and a target to accomplish (build)

    • Based on the rules only, (not any specific algorithm for the program), it finds a way to construct the target

  • Target

    • Something to be done/built, i.e. program binary, object file foo.o, distribution package

  • Rules

    • Descriptions that can be converted to dependencies between two files and to commands that transform the files

Tomáš Kalibera, Tomáš Pop


N ot specifying build algorithm is advantage

Not specifying build algorithm is advantage

  • Support for parallel builds

    • Make can automatically decide what can be built in parallel (without any changes to earlier well written makefiles)

  • Support for “building as much as possible”

    • Make can be run in a mode where it compiles everything that does not depend on a target that cannot be built (not stopping on first error it encounters)

Tomáš Kalibera


Make from user s perspective

Make from user’s perspective

  • Make file

    • Textual file with description of rules, commands are command line utilities (compiler, shell commands, etc)

  • Build tree

    • Make decides on what has to be modified by examining modification dates

    • If file foo.o depends on foo.c, and foo.c is newer than foo.o, foo.o has to be recompiled..

  • Make tool

    • Run “make” to build the default target

Tomáš Kalibera


Make file example

Make file example

OBJ = ../ant/main.o parse.o

all: prog

prog: $(OBJ)

$(CC) -o $@ $(OBJ)

parse.o: parse.c parse.h

$(CC) -c parse.c

target

prerequisity

command

Example from Peter Miller’s article “Recursive Make Considered Harmful”

Tomáš Kalibera, Tomáš Pop


List of gnu make file features

List of (GNU) Make file features

  • Built in variables (CC, FC, …)

  • Implicit rules

    • Typical compiler invocations, based on built-in variables (CC, CFLAGS, CPPFLAGS,…)

  • User functions, conditionals, variables

  • Internal functions

    • File-based, text and path manipulation

  • Remaking make files

    • Make files themselves can be re-made by make

    • This is very useful for automated generation of dependencies, that can be described within the make file itself

Tomáš Kalibera, Tomáš Pop


Make limitations

Make limitations

  • Portability of make files

    • Make is standardized by POSIX, but not all systems are POSIX compliant…

    • GNU Make runs everywhere

    • Writing make files to run both on non-UNIX systems (such as Windows) and UNIX is however a challenge

  • Inability to describe more complex dependencies than between two files

    • This matters i.e. for CORBA idl compiler that generates 2 files from a single one

  • Some argue that Make syntax is not user friendly

    • … and propose XML syntax that is much worse

Tomáš Kalibera


Make limitations1

Make limitations

  • Especially with older make implementations, specifying all the dependencies by hand was too much work

    • Implicit rules help

    • Automake (and other makefile generators) overcome this issue

    • More later…

Tomáš Kalibera, Tomáš Pop


Apache ant

Apache Ant

Tomáš Kalibera, Tomáš Pop


Ant summary

Ant summary

  • Java based

    • Primarily for compilation of Java programs

    • Wide support for programs and tools common in the Java world (EJBs, JSPs, JUnit, …)

    • Easily extensible in Java

  • Main features

    • Allows very portable scripting of building for Java applications

  • Weaknesses

    • Syntax hard to read (XML)

    • Limited extensibility in dependencies

Tomáš Kalibera, Tomáš Pop


Ant definitions

Ant definitions

  • Project

    • Represents the application to be built

  • Target

    • A name for a goal required for building (compilation, packaging, generating documentation)

    • Can depend on other targets

  • Task

    • An executable code fragment (i.e. execution of Java compiler)

  • Property

    • Name/value pair (i.e. for the base directory)

Tomáš Kalibera


Ant build file structure

Ant build file structure

<project name="MyProject" default="dist" basedir=".">

<description> simple example build file </description>

<property name="src" location="src"/>

<property name="build" location="build"/>

<property name="dist" location="dist"/>

<target name="init">

<mkdir dir="${build}"/>

</target>

<target name="compile" depends="init“ description="compile the source " >

<javac srcdir="${src}" destdir="${build}"/>

</target>

</project>

This is a shortened version of an example from Apache Ant documentation

Tomáš Kalibera, Tomáš Pop


Ant tasks

Ant tasks

  • Compilation

    • Javac, jspc, rmic, depend, …

  • Archiving

    • Bzip2, zip, gzip, jar, rpm, tar, cab, …

  • Remote communication

    • ftp, scp, sshexec, sending emails, …

  • Testing

    • JUnit

  • Scripting support

    • JavaScripts, Java 1.6 based, …

  • File-based operations

    • Platform independent manipulation with files and directories

Tomáš Kalibera, Tomáš Pop


Source dependency resolution

Source dependency resolution

  • Recompile everything from scratch

    • Just run the compiler on all sources all the time

  • Re-use class files compiled previously

    • “depend” task, deletes .class files obsoleted by modified sources

    • Dependencies are read from class files

    • Some dependencies are however not discovered

Tomáš Kalibera


Links

Links

  • Ant

    • http://ant.apache.org/manual/index.html

  • Make

    • http://www.gnu.org/software/make/manual/make.pdf

    • http://aegis.sourceforge.net/auug97.pdf

Tomáš Kalibera, Tomáš Pop


Problems

Problems

  • So far, so good…

Tomáš Kalibera, Tomáš Pop


Problems 1 build dependencies in fortran

Problems: 1) Build dependencies in Fortran

  • .h C-preprocessor header files

  • .F90 module sources

  • .mod compiled module info

  • .F90, .F fortran sources

  • .o object files

  • .so, .a library files

  • Binary executables

preprocessor

compiler

compiler

archiver

linker

Tomáš Kalibera, Tomáš Pop


Problems 1 build dependencies in fortran1

Problems: 1) Build dependencies in Fortran

  • C sources have headers

    • Something not generated, but needed for compilation of sources

    • header dependencies influence what has to be recompiled

  • Java sources have packages

    • Something generated by compiler from sources, needed for compiling other sources

    • package dependencies influence the order of compilation

  • Fortran sources have both

    • Headers and modules

Hardly could be handled

by tools like Ant

Tomáš Kalibera, Tomáš Pop


Problems 2 changes in source structure

Problems: 2) Changes in source structure

  • Make

    • It is necessary to modify Makefile when changing structure

      • Building too much

      • Not all dependencies covered

Tomáš Kalibera


Problems 3 recursive makefiles

Problems: 3) Recursive makefiles

  • Idea and figures from paper: Petter Miller Recursive Make Considered Harmful

    • Unix Word often using recursive makefiles

    • Consider following structure:

MODULES = App Parse

all:

For dir in $(MODULES); do \

(cd $$dir; ${MAKE} all; \

done

  • MyApp

    • Makefile

    • App

      • Makefile

      • main.c

    • Parse

      • Makefile

      • parse.c

      • parse.h

      • parse.y

all: prog

prog: parse.o ../App/main.o

$(CC) -o $@ $<

parse.o: parse.cparse.h

$(CC) -c parse.c

parse.cparse.h: parse.y

$(YACC) -d parse.y

all: main.o

main.o: main.c\ ../Parse/parse.h

$(CC) -I../Parse -c main.c

Tomáš Kalibera, Tomáš Pop


Problems 3 recursive makefiles1

Problems: 3) Recursive Makefiles

MODULES = App Parse

all:

For dir in $(MODULES); do \

(cd $$dir; ${MAKE} all; \

done

  • MyApp

    • Makefile

    • App

      • Makefile

      • main.c

    • Parse

      • Makefile

      • parse.c

      • parse.h

      • parse.y

all: prog

prog: parse.o ../App/main.o

$(CC) -o $@ $<

parse.o: parse.cparse.h

$(CC) -c parse.c

parse.cparse.h: parse.y

$(YACC) -d parse.y

all: main.o

main.o: main.c\ ../Parse/parse.h

$(CC) -I../Parse -c main.c

  • Q: What happen if parse.yis changed and incremental build (top level Makefile) is run?

Tomáš Kalibera, Tomáš Pop


  • Login