schema based program synthesis and the autobayes system part ii
Skip this Video
Download Presentation
Schema-based Program Synthesis and the AutoBayes System Part II

Loading in 2 Seconds...

play fullscreen
1 / 29

Schema-based Program Synthesis and the AutoBayes System Part II - PowerPoint PPT Presentation

  • Uploaded on

Schema-based Program Synthesis and the AutoBayes System Part II. Johann Schumann SGT, NASA Ames. Example. Generate a program that finds the maximum value of a function f(x): max f(x) wrt x. univariate. multivariate. Note: the function might be given as a formula or a vector of data.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Schema-based Program Synthesis and the AutoBayes System Part II' - kueng

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Generate a program that finds the maximum value of a function f(x): max f(x) wrt x



Note: the function might be given as a formula or a vector of data

schema s for univariate optimization
Schemas for univariate optimization

schema(max F wrt X, C) :- ... as before

schema(max F wrt X, C) :-

length(X, 1),

% F is a vector of data points F(0..n)

C = let(sequence([



if(select(F,I) > mymax,

assign(mymax, select(F,I)), skip)...

]), comment([‘The maximum is found by iterating...’]),


schema(max F wrt X, C) :-

length(X, 1),

% instantiate numeric solution algorithm

% e.g., golden section search

C = ...

schema(max F wrt X, C) :-




schema for univariate optimization
Schema for univariate optimization

schema(max F wrt X, C) :- % INPUT (Problem), OUTPUT (Code fragment)

% guards

length(X, 1),

% calculate the first derivative

simplify(deriv(F, X), DF),

% solve the equation

solve(true, x, 0 = DF, S),

% possibly more checks

% is that really a maximum?

simplify(deriv(DF, X), DDF),

(solve(true, x, 0 > DDF, _)

-> true ;

writeln(‘Proof obligation not solved automatically’)


XP = [‘The maximum for‘, expr(F), ‘is calculated ...’],

V = pv_fresh,

C = let(assign(V, C, [comment(XP)]), V).



  • build the derivative: df/dx
  • set it to 0: 0 = df/dx
  • solve that equation for x
  • the solution is the desired maximum
  • Generation of multiple programs
    • -maxprog
    • -maxprog N -fastest (coarse approximation)
  • Control for numeric solvers
    • pragma schema_control_arbitrary_init_values
    • pragma schema_control_use_generic_optimize
  • Tracing pragmas
  • The necessity of constraints
multivariate optimization
Multivariate Optimization
  • Task: minimize function F(X) wrt X
  • Algorithm:
  • start somewhere
  • go down along the steepest slope
  • when you come to a flat area, return that (local) minimum
  • Many design decisions
    • where to start?
    • how to move?
    • when to stop?

double* minimze(F){

double* x0 = pick_start();

int converging = 1;

while (converging){

double step_length = 0.1;

double step_dir = -gradient(F,x0);

x1 = x0 + step_length * step_dir;

if (fabs(F(x1) - F(x0)) < 0.001)

converging = 0;


x0 = x1;



multivariate optimization1
Multivariate Optimization

schema(max F wrt X, C) :- % IN, OUT

% guards: here none


Y > 1,

% divide and solve subproblems

schema(getStartValue(F,X), C_Start), % recursive schema calls

schema(getStepDirection(F,X), C_Dir),

schema(getStepSize(F,X), C_Size),

% assemble code segment

X0=pvar_new(X), % get a new PROGRAM variable

C = block([local(X0,double)],


[ assign(X0, C_start),


assign(X0, +([X0, *([C_Dir, C_Size])))



multivariate optimization ii
Multivariate optimization II

generated code for max sin(v) wrt v


C = block([local(X0,double)],


[ assign(X0, C_start),


assign(X0, +([X0, *([C_Dir, C_Size])))



double v_0;

double E;

v_0 = -99;

E = 1e10;

while (E > 0.001){

y = sin(v_0);

v_0 = V_0 - cos(v_0) * 0.01;

E = fabs(y - sin(v_0));


  • The schemas generate code in an intermediate language
  • procedural elements
  • local variables, lambda blocks
  • sum(..), while_converging(..) --> loops

Important: variables in specification or program are NOT Prolog variables

why schema based synthesis
Why schema-based synthesis?

some possibilities for getStepDir

Multiple algorithm variants can be automatically constructed

The “best” one is chosen by the user or selected via constraints

ab schema hierarchies
AB Schema Hierarchies
  • Schemas to break down statistical problem
    • Bayesian independence theorems -- works on Bayesian graphs
  • Schemas to solve complex statistical problems
    • instantiate (iterative) clustering algorithms
    • handling of time series problems
  • Schemas to solve atomic problems
    • instantiate PDF and maximize (symbolically)
    • instantiate numerical solvers (see last slides)
  • auxiliary schemas
    • initialization of clustering algorithms
    • data pre-processing (e.g., [0..1] normalization)
ab schema hierarchy
AB Schema Hierarchy
  • Static tree structure
  • AB uses two kinds of schemas
    • schemas for probabilistic problems
    • schemas for formula
schemas and ab model
Schemas and AB Model
  • The AB schemas have to use all information from the input specification, which is stored in the Prolog data base (AB model)
  • Problem: schemas can modify the model, which must be undone during backtracking
    • add new statistical variables
    • remove dependencies for subproblems
  • Solutions:
    • add model as parameters: schema(Prob, C, M_in, M_out) and everywhere else
    • keep a model stack (similar to the dynamic calling environments in procedural languages) and use backtrackable asserts/retracts
backtrackable global stuff
Backtrackable Global Stuff
  • Global data in Prolog are handled using assert/retract or flags. All other data are local to each clause

p(X) :- q(X,Z), r(Z). % X, Y, Z local to clause

  • Asserts are not backtrackable

p(X) :- assert(keep(X)), ..., fail.

The “keep(X)” is kept in the data base even after backtracking

  • Work-around: add global variables as parameter to all predicates (impractical)

p(X, GL_in, GL_out) :- GL_out = [keep(X)|GL_in], ...

  • Backtrackable bassert/bretract requires some low-level additional C-programs (but has clean semantics)
schema control
Schema Control
  • schema applicability is controlled via guards
  • order of application: order in Prolog file
  • How to enforce/avoid certain schemas
    • autobayes pragmas, but that’s not really fun
    • doesn’t work for nested applications:
      • inner loop: symbolic solutions only
      • outer loop: enable numeric loop
    • generate them all and decide later or pick “fastest”
  • schema control language is a research topic
    • extend declarative AB language
    • how to talk about selection of iterative algorithm in a purely declarative language?
the ab infra structure
The AB Infra Structure
  • term utilties
  • rewriting engine
  • symbolic system:
    • simplifier
    • abstraction (range, sign, definedness)
    • solver
  • pretty printer (code, intermediate language)
  • comment generation
term utilities
Term utilities
  • implemented on top of Prolog a lot of functional-programming style predicates for
    • lists, sets, bags, relations
    • terms, AC-terms
  • operations
    • term_substitute, subsumption, differences between term sets
  • ...
rewriting engine
Rewriting Engine
  • A lot of stuff in AB is done using rewriting (but not all)
  • small rewriting engine implemented in Prolog
    • rewriting rules are Prolog clauses
    • conditional rewriting, AC-style rewriting
    • Evaluation:
      • eager: apply first top-down
      • lazy: apply bottom up
    • continuation: pure bottom-up or dove-tailing
    • handle for attachment of prover/constraint solver
    • compilation of rewriting rules for higher efficiency
rewriting rules
Rewriting Rules


trig_simplify('sin-of-0', [eval=lazy|_] ,_,_, sin(0), 0) :- !.

trig_simplify('sin-of-pi-over-6',[eval=lazy|_],_,_,sin(*([1/6, pi])),1/2)

:- !.

trig_simplify('cos^2+sin^2',[eval=eager|_],_,_, +(Args),+([1|Args3])) :-

select(cos(X)**2, Args, Args2),

select(sin(X)**2, Args2, Args3),


  • Can combine pure rewriting with Prolog programming in the body of the rewrite rule
compilation and rewriting
Compilation and Rewriting
  • Group and compile rewrite rules (statically)

?- rwr_compile(my_simplifications,

[trig_simplify, remove_const_rules ]


  • Call the rewriting engine

rwr_cond(my_simplifications, true, S, T).

  • Calling with time-out
symbolic system
Symbolic System
  • Symbolic system implemented on top of the rewriting engine + Prolog code for solvers, etc
  • assumption-based rewriting
    • X/Y -- (not(Y = 0)) --> X
  • simplification (lots of rules)
  • calculation of derivatives (deriv(F,X) as operator)
  • Taylor-series expansion, ...
  • equation solver
    • polynomial solver
    • Gauss-elimination for sets of linear equations
    • sequentialization of equation systems
the ab intermediate language
The AB Intermediate language
  • strict separation between synthesis and code generation
  • small procedural intermediate language with some extensions
    • sum(..), prod(..), simul_assign(..), while_converging(...)
    • Annotations for comments, and pre/post/inv formulas
  • code generator for different languages/targets
    • C++/Octave
    • C/Matlab, C/standalone
    • ADA/SparkADA, Java (both “unsupported/in work/bad shape”)
  • Pretty-printer to ASCII, HTML, LaTeX
extending autobayes
Extending AutoBayes
  • some extensions are straight-forward: add text-book formulas
  • additional symbolic simplification rules might be required
  • adding schemas requires substantial work
    • “hard-coded” schema as first step
    • applicability constraints and control
    • functional mechanisms to handle scalar/vector/matrix cases are available
    • support for documentation generation
    • no schema language, Prolog syntax used
non gaussian pdf
Non-Gaussian PDF
  • Data characteristics are modeled using probability density functions (PDFs)
  • Example: Gaussians, exponential, ...
  • AB contains a number of built-in PDFs, which can be extended (hands-on demo)
  • Having multiple PDFs adds a lot of power over libraries
  • For clustering, often Gaussian distribution of data is used.
  • How about angles: 0 == 360
  • you get 5 clusters
  • A different distribution (vonMises-Fisher) automatically solves this problem
  • In AutoBayes: just replace the “gauss” by “vonmises1” -- no programming required
  • multiple PDFs in one spec
sample generation
Sample Generation
  • We have used:
    • MODEL ---> P ---(data)--> parameters
  • The model can be read the other way round: generate me random data, which are consistent with the model
    • MODEL ---> P ---(parameters)--> data
  • Very useful for
    • model debugging/development
    • debugging and assessment of synthesized algorithms
autobayes and correctness
AutoBayes and Correctness
  • practical synthesis: forget about correct-by-construction, but
  • detailed math derivations, which can be checked externally (e.g., by Mathematica)
  • literature references in documentation/comments
  • generation of test harness and sample data
  • checking of safety properties (“AutoCert”) [Cade2002 slide set]
autobayes as a prolog program
AutoBayes as a Prolog Program
  • AutoBayes is a pretty large program
    • ~180 prolog files, 100,000LoC (with AutoFilter)
  • Heavy use of
    • meta-programming (call, etc.)
    • rewriting (using an engine implemented in Prolog)
    • functional programming elements for all sorts of list/vector/array handling
    • backtracking and backtrackable global data structures
    • procedural (non-logical) elements, e.g., file I/O, flags, etc.
  • no use of modules but naming conventions
  • everything SWI Prolog + few C extensions to handle backtrackable global counters and flags
autobayes weak points
AutoBayes Weak Points
  • The input parser is very inflexible (uses Prolog operators)
  • Very bad error messages–often just “no”
  • no “schema language”: AutoBayes extension only by union of Prolog/domain specialist
  • Only primitive control of schema selection: need for a schema-selection mechanism
  • not all schemas are fully documented
  • large code-base, which needs to be maintained
  • AutoBayes suitable for a wide range of data analysis tasks
  • AutoBayes generated customized algorithms
  • AutoBayes schema-based program synthesis + symbolic
  • logic + functional + procedural elements used
  • AutoBayes extension: easy to very hard
  • AutoBayes debugging: a pain, but explanations and LaTeX output very helpful
  • AutoBayes is NASA OpenSource: bugfixes/extensions always welcome
  • AutoBayes has a 160+ pages Users manual
  • AutoBayes useful for classroom projects to PhD projects