Application-specific constraints for multimedia presentation generation

1. Application-specific constraints for multimedia presentation generation Joost Geurts, Jacco van Ossenbruggen and Lynda Hardman CWI Amsterdam email: Joost.Geurts@cwi.nl

2. Talk overview • Generating multimedia automatically • Cuypers multimedia generation engine • Multimedia and constraints • Quantitative constraints • Qualitative constraints • Cuypers demo • Conclusion, future directions

3. Generating adaptive multimedia • Content • Large multimedia database • System profile • PC, PDA, WAP • Network profile • Modem, Gigabit • User profile • Language, Interests, Abilities, Preferences Too costly to author manually

4. Cuypers multimedia generation engine

5. Automatic multimedia generation • Designer does not specify complete presentation……but only specifies requirements • System automatically finds a solution which meets requirements • How should the requirements be specified? • Declarative constraints

6. Traditional use of constraints • Constraint solving used for problems with: • Many variables • Large domains • Based on domain reduction paradigm • Quantitative constraints • Integer domain • Reduction by arithmetic relations • Greater than (>) • Less than (<) • Equals (=)

7. Drawbacks of quantitative constraints • Too many (trivial) solutions that differ by: • 1 pixel position, or • 1 milliseconds in timing • Not sufficiently expressive e.g. cannot specify “no overlap” constraint • Too low level e.g. A.X2  B.X1

8. Solution: qualitative constraints • Example “A not overlap B”, “B after C” • Advantages: • More intuitive • More expressive • Smaller domains • Qualitative solutions translate automatically to lower level quantitative constraints • New problem: What if constraints are insoluble?

9. Solution: Constraint Logic Programming • Combine Prolog unification and backtracking with constraint solving • Use Prolog rules to generate constraints • Backtrack when constraints are insoluble

10. Cuypers generation engine • Multiple layers: • Prolog rules to generate constraints • Qualitative constraints translate to quantitative constraints • Solution of both constraints provides sufficient information for final presentation

11. Cuypers demo: scenario • Client:User is interested in Rembrandt and wants to know about about the “chiaroscuro” technique • Server: Query database • Server: Generate constraints according to: • System profile • User profile • Network profile • Server: Solve constraints / revise constraints • Server: Generate SMIL presentation • Client: Play presentation

12. Conclusions • Quantitative constraintsare insufficient for automatic multimediapresentation generation. Also need • Qualitative constraintsto allow intuitive and effectivehigh level specification, and • Backtrackingfor revising specific constraintsthat cause the entire set to fail

13. Future directions • Best-first instead of depth-first • Choose “best” among possible solutions • Needs evaluation criteria • Improve knowledge management • Make design knowledge declarative and explicit • Preserve metadata in final presentation • Use standardized and reusable profiles

15. Need to make trade-offs • Semantics • Convey message • Aesthetics • Clear / nice layout • Resources • Screen size, bandwidth • Dimension may result in conflicting goals

16. Quantitative Constraints % csp(+Ids, -Boxes) csp([IdA,IdB],[box(IdA,[x1:AX1, …]), box(IdB,[x1:BX1,…])]) :- % get values maxX(MaxX), maxY(MaxY), height(IdA,HeightA), widtht(IdA,WidthA), … % define domains [AX1,AX2,BX1,BX2]::[0..MaxX], [AY1,AY2,BY1,BY2]::[0..MaxY], % set width & height AX2 – AX1 #= WidthA, AY2 – AY1 #= HeightA, … % constraints AX2 #< BX1, % A left-of B AY1 #= BY1, % A top-align B, … true.

17. Multimedia and Constraints • Constraint Logic Programming • Domain reduction • Backtracking • Unification (matching rules) • Qualitative Constraints • Non-integer domain • Allen’s 13 temporal interval relations in three dimensions

18. Qualitative Constraints • Example: • Two images, A,B • A left or right of B • A not above or below B

19. Qualitative Constraints % csp(+Ids, -Graph) csp([IdA, Idb], [edge(IdA,IdB,x,NoOverlap),…]) :- % define domains NoOverlap :: [b,b-,m,m-], Overlap :: [d,d-,s,s-,f,f-,e], % constraints edge(IdA,IdB,x,NoOverlap), % B not-overlap A edge(IdA,IdB,y,Overlap), % B overlap A true.

20. Qualitative Constraints • Reasoning • Inverse: edge(A,B,D,Value) <=> inverse(Value,RValue),edge(B,A,D,RValue). • Equality edge(A,B,D,V1), edge(A,B,D,V2) => V1 #= V2 • Transitive edge(A,B,D,VAB), edge(B,C,D,VBC) => tr(VAB,VBC,VAC), % rule generation algorithm edge(A,C,D,VAC). • Translation rules to quantitative domain edge(A,B,D,b) => node(A,D/2,V2), node(B,D/1,V1) V1 #< V2.

21. Problems in generating multimedia • Text documents are flexible • Add page, scrollbar, • Template models • Wrap text around images • Multimedia documents are less flexible • No pages or scrollbars, no line-breaking or hyphenation • Not based on text-flow • Feedback needed • Linear process model does not work for multimedia

22. Quantitative Constraints • Example: • Two images, A,B • A left-of B • A top-align B

23. Cuypers generation engine • Rhetoric/Semantic • Sequence, Example • Communicative devices • Bookshelf, Slideshow • Qualitative Constraints • A before B • Quantitative Constraints • A.X2 < B.X1 • Presentation • SMIL