Sparse Semidefinite Programming Relaxations for Large Scale Polynomial Optimization and Their Applications to Differential Equations Martin Mevissen Submitted in partial fulfillment of the requirements for the degree of DOCTOR OF SCIENCE Department of Mathematical and Computing Sciences Tokyo Institute of Technology August 2010 2 Acknowledgements The endeavor for the PhD in Tokyo and to complete this thesis would not have been possible without the support of a number of great people. My greatest appreciation goes to my advisor Masakazu Kojima for his encouragement, guidance and continued support. I am grateful for his interest in my studies and the insight I gained by collaborating with him. I am indebted to him for providing me with an environment to enjoy research to the fullest ever since I joined his group for writing my Diploma thesis back in 2006. Moreover, I would like to express my gratitude to Nobuki Takayama, who hosted me twice at Kobe university. I am thankful for his sincere interest and his engagement in our joint work with Kosuke Yokoyama. My gratitude extends to Yasumasa Nishiura and Takashi Teramoto for inviting me to Hokkaido University in 2008. They offered me a great environment to learn more about reaction-diffusion equations. I am also thankful to Jean Lasserre and Didier Henrion for hosting me at LAAS in 2009, and I am looking forward to continuing our joint work in the future. I would like to thank Sunyoung Kim, Makoto Yamashita and Jiawang Nie for our fruitful collaborations and exciting discussions. In particular, I would like to express my gratitude to Makoto for his constant and patient technical support. Many thanks go to Hans-Jakob Lüthi for supporting this venture in Japan from early on and his encouraging advice. Also, I would like to thank the German Academic Exchange Service for enabling me to pursue this journey with its Doctoral Scholarship for three years. The stay at Tokyo Institute of Technology would have been inconceivable without the people I shared this time, many thoughts and the bond of friendship. In particular, I would like to thank Paul Sheridan, for his unshakable optimism, Ken Shackleton, for his large-heartedness, Matthias Hietland Heie, for his open mind, Hiroshi Sugimoto, for his noble heart, Kojiro Akiba, for all our conversation, Mikael Onsjö, for being a great host, and Tomohiko Mizutani, for helping me out with a lot of things, when I was a newcomer. In my life in Japan, I was glad to find many friends who I can count on and who gave me the chance to call this place home. I enjoyed greatly everything I shared with them. Thank you so much, Shuji, Yoko, Jif, Azra, Masa, Mari, Moe, Hiroshi, Shota, Chiaki, Soji, Naomi, Tomoko. Finally, my deepest gratefulness goes to my parents. Their encouragement and love have been with me all the time. They stood by me every day of my life. I dedicate this thesis to them. 3 4 To my parents Contents 1 Introduction 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 9 10 11 2 Semidefinite Programming and Polynomial Optimization 2.1 Positive polynomials and polynomial optimization . . . . . . . . . . . . . 2.1.1 Decomposition of globally nonnegative polynomials . . . . . . . . . 2.1.2 Decomposition of polynomials positive on closed semialgebraic sets 2.1.3 Dense SDP relaxations for polynomial optimization problems . . . 2.1.4 Sparse SDP relaxations for polynomial optimization problems . . . 2.2 Exploiting sparsity in linear and nonlinear matrix inequalities . . . . . . . 2.2.1 An SDP example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Positive semidefinite matrix completion . . . . . . . . . . . . . . . 2.2.3 Exploiting the domain-space sparsity . . . . . . . . . . . . . . . . . 2.2.4 Duality in positive semidefinite matrix completion . . . . . . . . . 2.2.5 Exploiting the range-space sparsity . . . . . . . . . . . . . . . . . . 2.2.6 Enhancing the correlative sparsity . . . . . . . . . . . . . . . . . . 2.2.7 Examples of d- and r-space sparsity in quadratic SDP . . . . . . . 2.3 Reduction techniques for SDP relaxations for large scale POP . . . . . . . 2.3.1 Transforming a POP into a QOP . . . . . . . . . . . . . . . . . . . 2.3.2 Quality of SDP relaxations for QOP . . . . . . . . . . . . . . . . . 2.3.3 Numerical examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 13 13 15 18 25 28 29 31 33 37 40 42 46 49 50 57 60 3 SDP Relaxations for Solving Differential Equations 3.1 Numerical analysis of differential equations . . . . . . . . . . . . . . 3.1.1 The finite difference method . . . . . . . . . . . . . . . . . . . 3.1.2 The finite element method and other numerical solvers . . . . 3.2 Differential equations and the SDPR method . . . . . . . . . . . . . 3.2.1 Transforming a differential equation into a sparse POP . . . 3.2.2 The SDPR method . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Enumeration algorithm . . . . . . . . . . . . . . . . . . . . . 3.2.4 Discrete approximations to solutions of differential equations 3.3 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 A nonlinear elliptic equation with bifurcation . . . . . . . . . 3.3.2 Illustrative nonlinear PDE problems . . . . . . . . . . . . . . 3.3.3 Reaction-diffusion equations . . . . . . . . . . . . . . . . . . . 3.3.4 Differential algebraic equations . . . . . . . . . . . . . . . . . 3.3.5 The steady cavity flow problem . . . . . . . . . . . . . . . . . 3.3.6 Optimal control problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 67 68 72 74 74 80 83 86 86 87 89 96 103 105 114 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 CONTENTS 4 Concluding Remarks and Future Research 123 4.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 4.2 Outlook on future research directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Notation N Nn Z R Rn Rn×m Sn Sn+ Sn (E, ?) Sn+ (E, ?) Sn (E, 0) Sn+ (E, 0) R[x] R[x]d R[x, P A]2 R[x] Λ(d) u(x, A) G(N, E) < ≻ • det(·) rank(·) Tr(·) deg(·) supp(·) natural numbers n- dimensional vector space with entries in N integers real numbers n-dimensional vector space with real entries vector space of n by m matrices with real entries vector of symmetric matrices in Rn×n cone of symmetric, positive semidefinite matrices in Rn×n partial symmetric matrices with entries specified in E matrices in Sn (E, ?) that can be completed to positive semidefinite matrices symmetric matrices with nonzero entries on the diagonal and in E positive semidefinite matrices in Sn (E, 0) ring of multivariate polynomials in the n dimensional variable x with coefficients in R set of polynomials of degree less or equal d set of polynomials supported on A ⊂ Nn , R[x, A] = {p ∈ R[x] | supp(p) ⊂ A} set of squares polynomials, P of sums P R[x]2 := p ∈ R[x] | p = ri=1 p2i , pi ∈ R[x] for some r ∈ N set of multivariate indices of degree less or equal d, Λ(d) = {α ∈ Nn : | α |≤ d} monomial vector for A ⊂ Nn , u(x, A) = (xα | α ∈ A) graph with vertex set N and edge set E positive semidefinite matrix positive definite matrix Pn Pn interior product on Sn , A • B = i=1 j=1 Ai,j Bi,j determinant of a matrix rank of a matrix P trace of a matrix, Tr(A) := ni=1 Ai,i degree of a polynomial support of a polynomial, supp(p) := {α ∈ Nn | pα 6= 0} 7 8 CONTENTS Mw (y) Mw (y, I) Mw (y g) Mw (y g, I) imaginary unit, i.e., I 2 = −1 basic, closed semialgebraic set generated by g1 , . . . , gm ∈ R[x], K(g1 , . . . , gm ) := {x ∈ Rn | g1 (x) ≥ 0, . . . , gm (x) ≥ 0} quadraticP module defined P by g1 , . . . , gm , M (K) = R[x]2 + g1 R[x]2 + . . . + gm R[x]2 approximation M (K) Pmof order d forP Md (K) = σ g | σ ∈ R[x]2 , deg(σi gi ) ≤ 2d i i i i=0 multiplicative convex cone generated by R[x]2 and g1 , . . . , gm , P P P2 hg1 , . . . , gm i = M (K) + g1 g2 R[x]2 + . . . + g1 g2 · · · gm R[x]2 multiplicative monoid Qr generated by g1 , . . . , gm , O(g1 , . . . , gm ) = { i=1 ti | ti ∈ {g1 , . . . , gm } for r ∈ N} ideal generated by g1 , . . . , gm , I(g1 , . . . , gm ) = R[x] + g1 R[x] + . . . + gm R[x] moment matrix of order w for the vector y partial moment matrix, contains only those components yα of y with α ∈ I localizing matrix of order w for vector y and g ∈ R[x] partial localizing matrix MkS Mk α ki,j k0 tC ǫsc ǫobj ω me Nx hx ui,j higher monomial set of a POP higher monomial list of a POP dividing coefficient number of substitutions required by an algorithm to transform a given POP into a QOP total computation time of an algorithm scaled feasibility error of numerical solution for a POP optimality error of SDP relaxation solution for a POP relaxation order of dense or sparse SDP relaxation maximum eigenvalue of a Jacobian of a system of polynomial equations number of grid points in x-direction in a discretized domain distance of two grid points in x-direction in a discretized domain approximation of u at grid point (xi , yj ) in a finite difference scheme max min sup inf lbd ubd maximum minimum supremum infimum lower bound upper bound SDP POP QOP ODE PDE OCP FDM FEM FVM SQP QSDP semidefinite program polynomial optimization problem quadratic optimization problem ordinary differential equation partial differential equation optimal control problem finite difference method finite element method finite volume method sequential quadratic programming quadratic semidefinite program I K(g1 , . . . , gm ), K M (K) Md (K) P2 hg1 , . . . , gm i O(g1 , . . . , gm ) I(g1 , . . . , gm ) Chapter 1 Introduction 1.1 Motivation A wide variety of problems arising in mathematics, physics, engineering, control and computer science can be formulated as optimization problems, where all functions in objective and constraints are multivariate polynomials over the field of real numbers - so called polynomial optimization problems (POP). In general, polynomial optimization problems are severely non-convex and NP-hard to solve. In recent years there has been active research in semidefinite programming (SDP) relaxation methods for POPs. That is, a general, non-convex POP is relaxed to a convex optimization problem, an SDP. Solving the SDP provides either a lower bound to the minimum, or approximations for minimum and global minimizers of the POP. First convexification techniques for general POPs have been proposed by Shor [91] and Nesterov [70]. Since the pioneering work of Shor, convexification and in particular SDP relaxation techniques have been used in an ever increasing number of applications and problems. One of the classical examples is the SDP relaxation for a non-convex quadratic programming formulation of the NP-hard max-cut problem [26]. Other NPhard problems, that can be formulated as POPs are {0, 1}− linear programming, or testing whether a symmetric matrix is copositive [67]. A breakthrough in this field was Lasserre’s seminal paper [51]. Given the feasible set of a general POP is a basic, compact semialgebraic set, a hierarchy of SDP relaxations can be constructed, which provides a monoton increasing sequence of lower bounds for the minimum of the POP. Lasserre showed this sequence converges to the minimum of the POP under a fairly general condition. Lasserre’s relaxation and also the approach [79] by Parrilo do rely on the representation of nonnegative polynomials as sum of squares of polynomials and the dual theory of moments. Despite being a powerful theoretical tool to approximate minimum and minimizer of general polynomial optimization problems, the Lasserre relaxation is not practical even for medium scale POPs. In fact, the size of the matrix inequality constraints in the SDP relaxations grows rapidly for increasing order of the hierarchy. Thus, in the case of a medium or large scale POPs the SDP relaxations becomes intractable for current SDP solvers as SeDuMi [95] or SDPA [104], even for small choices of the relaxation order. However, large scale POP arise from challenging problems and efficient methods to solve them are in high demand. For instance one problem, which received lots of attention recently, is the sensor network localization problem [6, 71, 43]. A first approach to reduce the size of SDP relaxations for POPs has been the concept of correlative sparsity of a POP [47, 102, 52]. Exploiting structured sparsity enables to attempt POP of larger dimension by a hierarchy of sparse SDP relaxations. Still, the capacity of current SDP solvers is limiting the applicability of sparse SDP relaxations for large scale POP. As one way to take advantage of sparsity more efficiently, we develop a general notion of sparsity in linear and nonlinear matrix inequalities and show how to exploit this sparsity via positive semidefinite matrix completion. We demonstrate how so called domain-space and range-space sparsity can be used to decrease the size of SDP relaxations for large scale POPs substantially. Another technique to attempt large scale POPs is based on the idea to reduce the size of the sparse SDP relaxations by transforming a general POP into an equivalent quadratic optimization problem (QOP). For an important class of large scale POPs the size of sparse SDP relaxations for the equivalent QOPs is far 9 10 CHAPTER 1. INTRODUCTION smaller than the size of sparse SDP relaxations for the original POPs. The second topic of this thesis is to investigate how to efficiently apply sparse SDP relaxation techniques for an important class of challenging problems, the numerical analysis of differential equations. For most problems involving ordinary or partial differential equations it is not possible to find analytic solutions - in particular if the equations are nonlinear in the unknown function. Even the problem to find approximations to the solutions of ODEs or PDEs by numerical methods is well known to be a hard problem, which begins to attract attention by researchers in moment, SDP and numerical algebra techniques. On the one hand a moment based approach [5] has been proposed to find tight bounds for linear functionals defined on linear PDEs. On the other hand, a homotopy continuation based approach [2, 33, 34] has been proposed to find all solutions of a discretized, possibly nonlinear PDE. We will show how to transform a problem involving differential equations into a POP by using standard finite difference schemes. The dimension of these POPs is determined by the discretization of the domain of the differential equation. Thus, for fine discretizations we obtain a challenging class of large scale POPs. These POPs satisfy both, correlative and domain-space sparsity, which enables us to apply sparse SDP relaxation techniques efficiently. The sparse SDP relaxation method is of particular interest for PDE problems with several solutions. We propose different algorithms based on the sparse SDP relaxation method to detect several or even all solutions of a system of nonlinear PDE. It is a strength of this method, that a wide variety of nonlinear PDE problems can be solved: Nonlinear elliptic, parabolic and hyperbolic equations, reaction-diffusion equations, steady state Navier-Stokes equations in fluid dynamics, differential algebraic equations or nonlinear optimal control problems. 1.2 Contribution This thesis is largely based on the content of prior publications of the author. Its contributions can be summarized as follows. • We present a general framework to detect, characterize and exploit sparsity in an optimization problem with linear and nonlinear matrix inequality constraints via positive semidefinite matrix completion. We distinguish two types of sparsity, domain-space sparsity for the symmetric matrix variable in objective and constraint functions of the problem, and range-space sparsity. Four conversion methods are proposed to exploit these two types of sparsity. We demonstrate the efficiency of these conversion methods on SDP relaxations for sparse, large-scale POP derived from discretizing partial differential equations and the sensor network localization problem. This result is based on our work [42]. • Based on the observation dating back to Shor [92], that any POP can be written as an equivalent QOP, we develop four heuristics for transforming a POP into a QOP. We show, that sparsity of the POP is maintained under our transformation procedures, and propose different techniques to improve the quality of the sparse SDP relaxations for the QOP, which are weaker than the more expensive sparse SDP relaxations for the equivalent POP. This technique is shown to be very efficient for large-scale POP: We are able to obtain highly accurate approximations to the global optimizers of the POP by solving SDP relaxations of vastly reduced size. This work is presented in detail in [62]. • We are the first to introduce a method based on sparse SDP relaxations to solve systems of linear and partial differential equations [61, 63]. Unlike the approach [5] we are able to approximate the actual solutions of an ordinary or partial differential equation. Moreover, our approach is applicable to nonlinear differential equations, whereas the technique [5] is limited to linear PDEs. Furthermore, compared to the numerical algebraic approach [2, 33, 34], we can solve a system of polynomial equations derived from a PDE for a much finer discretization by exploiting sparsity. Also, we do not aim at finding all complex solutions, but we detect the real solutions to that system of equations one by one. • Comparing the sparse SDP relaxation method to solve differential equations to existing PDE solvers, our approach has the following advantages: (a) We can add polynomial inequality constraints for the unknown solutions of the differential equations to the system of equations obtained by the finite 1.3. OUTLINE OF THE THESIS 11 difference discretization of the PDE, which can be understood as restricting the space of functions we are searching for solutions. (b) We can detect particular solutions of a PDE, by choosing an appropriate objective function for the sparse POP derived from the PDE problem or by adding inequality constraints to that POP. (c) We are able to systematically enumerate all solutions of a discretized PDE problem by iteratively applying the SDP relaxation method. (d) We exploit the fact, that the sparse SDP relaxations provide an approximation to the global optimizer of a POP. Thus, even if the accuracy of the solution of the SDP relaxation is not high, it is a good initial guess for locally convergent solvers for many PDE problems. This fact is in particular interesting for PDE problems with many solutions. These results are based on our work in [61, 63, 62]. • The sparse SDP relaxation method for solving large scale POP derived from differential equations can be applied to solve nonlinear, optimal control problems. Unlike the moment method in [54] our method yields approximations to the optimal control, trajectories and value of a control problem in addition to provide lower bounds for the optimal value of the control problems. 1.3 Outline of the thesis This thesis consists of two main parts. In the first part given by Chapter 2 we introduce the approaches to use methods from convex optimization to solve general, nonconvex polynomial optimization problems. We begin in 2.1 with introducing the historical background of characterizing positive polynomials, the problem of minimizing multivariate polynomials over basic, closed semialgebraic sets and the dense Lasserre relaxation, a sequence of semidefinite programs whose minima converge to the minimum of a polynomial optimization problem under fairly general conditions. Finally, we review the method of exploiting correlative sparsity of a POP to construct a sequence of sparse SDP relaxations. In 2.2 we present a general framework to exploit domain- and range space sparsity in problems involving linear or nonlinear matrix inequalities. This technique can be applied to the large scale SDP relaxations for large scale POP. In 2.3 we introduce the approach to reduce the size of dense or sparse SDP relaxations for large scale POP, which is based on the idea to transform a general POP into an equivalent QOP. In the second part presented by Chapter 3 we show how to use the methods and techniques from Chapter 2 for the numerical analysis of ordinary and partial differential equations. First we give an overview over existing numerical methods for solving partial differential equations, in particular the two most common approaches, the finite difference method and the finite element method, in 3.1. In 3.2 we introduce our method to transform a problem involving partial differential equations into a POP via finite difference discretization, and to solve the resulting large scale POP by the SDP relaxation techniques from Chapter 2. In 3.3 we apply our SDP relaxation method to a variety of different PDE problems such as nonlinear elliptic, parabolic and hyperbolic equations, differential algebraic equations, reaction-diffusion equations, fluid dynamics and nonlinear optimal control. Finally, we summarize the thesis in Chapter 4 with some concluding remarks and give an outlook on possible future research directions based on the methods and results presented here. 12 CHAPTER 1. INTRODUCTION Chapter 2 Semidefinite Programming and Polynomial Optimization 2.1 Positive polynomials and polynomial optimization Polynomial optimization and the problem of global nonnegativity of polynomials are active fields of research and remain in the focus of researchers from various areas as real algebra, semidefinite programming and operator theory. Shor [91] was the first who introduced the idea of applying a convex optimization technique to minimize an unconstrained multivariate polynomial. Also, Nesterov [70] was one of the first who discussed to exploit the duality of moment cones and cones of nonnegative polynomials in a convex optimization framework. He showed the characterization of a moment cone by linear matrix inequalities, i.e., semidefinite constraints, if the elements of the corresponding cone of nonnegative polynomials can be written as sum of squares. The next milestone in minimizing multivariate polynomials was given by Lasserre [51], who applied recent real algebraic results by Putinar [81] to construct a sequence of semidefinite program relaxations whose optima converge to the optimum of a polynomial optimization problem. Another approach to apply real algebraic results to attempt the problem of nonnegativity of polynomials was introduced by Parrilo [79]. We attempt to solve the following polynomial optimization problem: min p(x) s.t. gi (x) ≥ 0 (2.1) ∀i = 1, . . . , m where p, g1 , . . . , gm ∈ R [x]. Problem (2.1) can also be written as minx∈K p(x) (2.2) where K the basic, closed semialgebraic set that is defined by the polynomials g1 , . . . , gm . Let p⋆ denote the optimal value of problem (2.2) and K ⋆ := {x⋆ ∈ K | ∀x ∈ K : p(x⋆ ) ≤ p(x)}. In the case K compact, K ⋆ 6= ∅, if K 6= ∅. 2.1.1 Decomposition of globally nonnegative polynomials The origin of research in characterizing nonnegative and positive polynomials lies in Hilbert’s 17th problem, whether it is possible to express a nonnegative rational function as sum of squares of rational functions. This question was answered positively by Artin in 1927. Moreover, the question arises, whether it is possible to express any nonnegative polynomial as sum of squares of polynomials. In the case of univariate polynomials the answer to this question is yes, as stated in the following theorem. Theorem 2.1 Let p ∈ R [x], x ∈ R. Then, p(x) ≥ 0 for all x ∈ R if and only if p ∈ 13 P R[x]2 . 14 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Proof ” ⇐ ” : Trivial. ” ⇒ ” : Let p(x) ≥ 0 for all x ∈ R. It is obvious that deg(p) = 2k for some k ∈ N. Then, the real roots of of p(x) should have even multiplicity, otherwise p(x) would alter its sign in a neighborhood of a root. Let λi , i = 1, . . . , r be its real roots with corresponding multiplicity 2mi . Its complex roots can be arranged in conjugate pairs, aj + Ibj , aj − Ibj , j = 1, . . . , h. Then, h r Y Y 2mi ((x − aj )2 + b2j ). (x − λi ) p(x) = C j=1 i=1 Note that the leading coefficient C needs to be positive. Thus, by expanding the terms in the products, we see that p(x) can be written as a sum of squares of polynomials, of the form 2 k k X X vij xj . p(x) = i=0 j=0 However, Hilbert himself already noted that not every nonnegative polynomial can be written as sum of squares. For instance the Motzkin form M , M (x, y, z) = x4 y 2 + x2 y 4 + z 6 − 3x2 y 2 z 2 is nonnegative but not sum of squares. In fact Hilbert gave a complete characterization of the cases where nonnegativity and the existence of a sum of squares decomposition are equivalent. Definition 2.1 A form is a polynomial where all the monomials have the same total degree m. Pn,m denotes the set of nonnegative forms of degree m in n variables, Σn,m the set of forms p such that p = Σk h2k , where hk are forms of degree m 2. There is a correspondance between forms in n with power m and polynomials in n − 1 variables with degree less or equal to m. In fact, a form in n variables of degree m can be dehomogenized to a polynomial in n − 1 variables by fixing any of the n variables to the constant value 1. Conversely, given a polynomial in n − 1 variables in can be homogenized by multiplying each monomial by powers of a new variable such that the degree of all monomials equals m. Obviously, Σn,m ⊆ Pn,m holds for all n and m. The following Theorem is due to Hilbert. Theorem 2.2 Σn,m ⊆ Pn,m holds with equality only in the following cases: (i) Bivariate forms: n = 2, (ii) Quadratic forms: m = 2, (iii) Ternary quartic forms: n = 3, m = 4. We interprete the three cases in Theorem 2.2 in terms of polynomials. The first one corresponds to the equivalence of nonnegativity and sum of squares condition in the univariate case as in Theorem (2.1). The second one is the case of quadratic polynomials, where the sum of squares decomposition follows from an eigenvalue/eigenvector factorization. The third case corresponds to quartic polynomials in two variables. Relevance of sum of squares characterizations Recall that the constraints of our original polynomial optimization problem are nonnegativity constraints for polynomials of the type gi (x) ≥ 0 (i = 1, . . . , m). The question, whether a given polynomial is globally nonnegative is decidable, for instance by the TarskiSeidenberg decision procedure [7]. Nonetheless, regarding complexity, the general problem of testing global nonnegativity of a polynomial function is NP-hard [67], if the degree of the polynomial is at least four. Therefore it is reasonable to substitute the nonnegativity constraints by expressions that can be decided easier. It was shown by Parrilo that the decision whether a polynomial is sum of squares is equivalent to a semidefinite program as stated in the following theorem. 15 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION Theorem 2.3 The existence of a sum of squares decomposition of a polynomial in n variables of degree 2d can be decided by solving a semidefinite programming [79]. feasibility problem If the polynomial is dense, n+d n+d the dimensions of the matrix inequality are equal to × . d d αn 1 α2 Proof Let p ∈ R [x] with degree 2d. Recall u(x, Λ(d)) denotes vector of monomials xα 1 x2 · · · xn the ordered Pn n+d with i=1 αi ≤ d. The length of u(x, Λ(d)) is s := s(d) = . d P Claim: p ∈ P R[x]2 if and only if ∃V ∈ Ss+ such that p = u(x, Λ(d))T V u(x, Λ(d)). Pf: ⇒: p ∈ R[x]2 , i.e. p= r X qi2 = i=1 r X (wiT u(x, Λ(d)))2 T = u(x, Λ(d)) r X wi wiT i=1 i=1 ! u(x, Λ(d)). Pr Thus, V = i=1 wi wiT and V ∈ Ss+ . ⇐: As V ∈ Ss+ there exists a Cholesky factorization V = W W T , where W ∈ Rs×s and let wi denote the ith column of W . We have p = u(x, Λ(d))T V u(x, Λ(d)) = s X i=1 wi wiT u(x, Λ(d)) = s X (wiT u(x, Λ(d)))2 , i=1 i.e., p ∈ R[x]. Thus, the claim follows. P Expanding the quadratic form gives p = si,j=1 Vi,j u(x, Λ(d))i u(x, Λ(d))j . Equating the coefficients in this expression with the coefficients of the corresponding monomials in the original form for p generates a set of linear equalities for the variables Vi,j (i, j = 1, . . . , s). Adding the constraint VP∈ Ss+ to those linear equality constraints, we conditions for p which are equivalent to claiming p ∈ R[x]2 . Therefore, to decide P obtain 2 whether p ∈ R[x] is equivalent to solving a semidefinite programming feasibility problem. 2.1.2 Decomposition of polynomials positive on closed semialgebraic sets Real algebraic geometry deals with the analysis of the real solution set of a system of polynomial equations. The main difference to algebraic geometry in the complex case lies in the fact that R is not algebraically closed. One of the main results of real algebra are the Positivstellensätze which provide certificates in the case a semialgebraic set is empty. Improved versions of the Positivstellensätze can be obtained in case of compact semialgebraic sets. General semialgebraic sets The Positivstellensatz below is due to Stengle; a proof can be found in [7]. Theorem 2.4 (Stengle) Let (fj )j=1,...,t , (gk )k=1,...,m , (hl )l=1,...,k be finite families of polynomials in R [x]. The following properties are equivalent: gj (x) ≥ 0, j = 1, . . . , m (i) x ∈ Rn | fs (x) 6= 0, s = 1, . . . , t = ∅. hi (x) = 0, i = 1, . . . , k (ii) There exist g ∈ Σ2 hg1 , . . . , gm i, f ∈ O(f1 , . . . , ft ), the multiplicative monoid generated by f1 , . . . , ft , h ∈ I(h1 , . . . , hk ), the ideal generated by h1 , . . . , hk , such that g + f 2 + h = 0. To understand the differences between the real and the complex case, and the use of the Positivstellensatz 2.4 consider the following example. 16 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Example 2.1 Consider the very simple quadratic equation x2 + ax + b = 0. By the fundamental theorem of algebra, the equation has always solutions in C. For the case when the solution is required to be real, the solution set will be empty if and only if the discriminant D satisfies D := b − In this case taking g := 2 1 a √ (x + ) , 2 D a2 > 0. 4 f := 1, h := − 1 2 (x + ax + b), D the identity g + f 2 + h = 0 is satisfied. It is to remark, the Positivstellensatz represents the most general deductive system for which inferences from the given equations can be made. It guarantees the existence of infeasibility certificates given by the polynomials f , g and h. For complexity reasons these certificates cannot be polynomial time checkable for every possible instance, unless NP=co-NP. Parrilo showed that it is possible that the problem of finding infeasibility certificates is equivalent to an semidefinite program, if the degree of the possible multipliers is restricted [79]. Theorem 2.5 Consider a system of polynomial equalities and inequalities as in Theorem 2.4. Then, the search for bounded degree Positivstellensatz infeasibility certificates can be done using semidefinite programming. If the degree bound is chosen to be large enough , then the SDPs will be feasible, and the certificates are obtained from its solution. Proof: Consequence of the Positivstellensatz and Theorem 2.3, c.f. [79]. As the feasible set of (2.2) is a closed semialgebraic set, we are interested in characterizations for these sets and polynomials positive on semialgebraic sets. The Positivstellensatz allows to deduce conditions for the positivity or the nonnegativity of a polynomial over a semialgebraic set. A direct consequence of the Positivstellensatz is the following corollary [7], pp. 92. Corollary 2.1 Let g1 , . . . , gm ∈ R [x] , K = {x ∈ Rn | g1 (x) ≥ 0, . . . , gm (x) ≥ 0} and f ∈ R[x]. Then: (i) ∀x ∈ K f (x) ≥ 0 ⇔ ∃ s ∈ N ∃ g, h ∈ Σ2 hg1 , . . . , gm i s.t. f g = f 2s + h. (ii) ∀x ∈ K f (x) > 0 ⇔ ∃ g, h ∈ Σ2 hg1 , . . . , gm i s.t. f g = 1 + h. Proof (i) Apply the Positivstellensatz to the set {x ∈ Rn | g1 (x) ≥ 0, . . . , gm (x) ≥ 0, −f (x) ≥ 0, f (x) 6= 0} . (ii) Apply the Positivstellensatz to the set {x ∈ Rn | g1 (x) ≥ 0, . . . , gm (x) ≥ 0, −f (x) ≥ 0} . These conditions for the nonnegativity and positivity of polynomials on semialgebraic sets can be improved under additional assumptions. We present these improved conditions for compact semi-algebraic sets in the following section. 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION 17 Compact semialgebraic sets It is our aim to characterize polynomials that are positive or nonnegative on compact semialgebraic sets. A first characterization is a theorem due to Schmüdgen [87]: Theorem 2.6 (Schmüdgen) Let K = {x ∈ Rn ; g1 (x) ≥ 0, . . . , gm (x) ≥ 0} be a compact semialgebraic subset of Rn and let p be a positive polynomial on K. Then p ∈ Σ2 hg1 , . . . , gm i. It was Putinar [81] who simplified this characterization under an additional assumption. P Definition 2.2 A quadratic module M (K) is called archimedean if N − ni=1 x2i ∈ M (K) for some N ∈ N. Theorem 2.7 (Putinar) Let p be a polynomial, positive on the compact semialgebraic set K and M (K) archimedian, then p ∈ M (K). Thus, under the additional assumption of M (K) being archimedian, we obtain the stricter characterization p ∈ M (K) ⊆ Σ2 hg1 , . . . , gm i instead of p ∈ Σ2 hg1 , . . . , gm i. The original proof of Theorem 2.7 is due to Putinar [81]. In this proof Putinar applies the separation theorem for convex sets and some arguments from functional analysis. A new proof due to Schweighofer [88] avoids the arguments from functional analysis and requires only results from elementary analysis. A further theorem by Schmüdgen [87] provides equivalent conditions for M (K) being archimedian. Theorem 2.8 The following are equivalent: (i) There exist finitely many t1 , . . . , ts ∈ M (K) such that the set {x ∈ Rn | t1 (x) ≥ 0, . . . , ts (x) ≥ 0} (which contains K) is compact and Q i∈I ti ∈ M (K) for all I ⊂ {1, . . . , s}. (ii) There exists some p ∈ M (K) such that {x ∈ Rn | p(x) ≥ 0} is compact. Pn (iii) There exists an N ∈ N such that N − i=1 x2i ∈ M (K), i.e., M (K) is archimedian. (iv) For all p ∈ R [x], there is some N ∈ N such that N ± p ∈ M (K). Thus, for any polynomial p positive on K, p ∈ M (K) holds, if one of the conditions in Theorem 2.8 is satisfied. Whether it is decidable that one of the equivalent conditions hold is not known and subject of current research. However, for a given polynomial optimization problem with compact feasible set K, it is easy to make the Pncorresponding quadratic module M (K) archimedian. We just need to add a redundant constraint N − i=1 x2i ≥ 0 for a sufficiently large N . Example 2.2 Consider the compact semialgebraic set K = x ∈ R2 | g(x) = 1 − x21 − x22 ≥ 0 . The quadratic module M (K) is archimedian, as 1 − x21 − x22 = 02 + 12 · g(x) ∈ M (K). The polynomials f1 (x) := x1 + 2 and x31 + 2 are positive on K. Thus f1 , f2 ∈ M (K) with Theorem 2.7. Their decomposition can be derived as f1 (x) f2 (x) = x1 + 2 = 2x31 + 3 = 12 (x1 + 1)2 + 21 x22 + 1 + 12 (1 − x21 − x22 ), = (x31 + 1)2 + (x21 x2 )2 + (x1 x2 )2 + x22 + 1 + (x41 + x21 + 1) (1 − x21 − x22 ). The next example demonstrates that in general not every polynomial nonnegative on a compact semialgebraic set K is contained in M (K) even if M (K) is archimedian. 18 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Example 2.3 Consider the compact semialgebraic set K = x ∈ R | g1 (x) := x2 ≥ 0, g2 (x) := −x2 ≥ 0 . It is obvious that M (K) is archimedian. Also, it is easy to see that there are no q, r, s ∈ p(x) := x = q(x) + r(x) x2 + s(x) (−x2 ), P R[x]2 such that although p is nonnegative on K. However, the polynomial pa ∈ R[x] defined by pa (x) = x + a for a > 0 can be decomposed as 1 1 (x + 2a)2 − x2 . pa (x) = x + a = 4a 4a Thus pa ∈ M (K) for all a > 0. Remark 2.1 Given a compact semialgebraic set K, it apparently holds, any positive polynomial on K belongs to the cone M (K) if and only if M (K) is archimedian. Theorem 2.7 is called Putinar’s Positivstellensatz. Obviously, it does not really characterize the polynomials positive on K since the polynomials in M (K) must only be nonnegative on K. Also, it does not fully describe the polynomials nonnegative on K since they are not always contained in M (K). However, it is Theorem 2.7 that is exploited by Lasserre in order to attempt the polynomial optimization problem. 2.1.3 Dense SDP relaxations for polynomial optimization problems The idea to apply convex optimization techniques to solve polynomial optimization problems was first proposed in the pioneering work of Shor [91]. Shor introduced lower bounds for the global minimum of a polynomial function p. These bounds are derived by minimizing a quadratic function subject to quadratic constraints. Also Nesterov discussed the minimizaion of univariate polynomials and mentioned the problem of minimizing multivariate polynomials in [70]. It was Lasserre [51] who first realized the possibility to apply Putinar’s Positivstellensatz, Theorem 2.7, to solve a broader class of polynomial optimization problems, that goes beyond the case where p − p⋆ can be described as sum of squares of polynomials. At first, we introduce Lasserre’s approach to derive semidefinite relaxations for minimizing a polynomial over a semialgebraic set, as Putinar’s theorem is applied directly there. At second, we present the unconstrained case. Since semialgebraic sets enter through the backdoor, in order to be able to apply Putinar’s Positivstellensatz, we present it after the constrained case. Lasserre’s relaxation in the constrained case After studying positivity and nonnegativity of polynomials and the related problem of moments, we attempt the inital polynomial optimization problem (2.2) over a compact semialgebraic set K, minx∈K p(x). One of the major obstacles for finding the optimum p⋆ is the fact that the set K and the function p are far from being convex. The basic idea of Lasserre’ s approach [51] is to convexify problem (2.2). We outline this procedure of convexification. It has to be emphasized that Lasserre’s approach is based on two assumptions. First we require the semi-algebraic set K to be compact, and second we assume M (K) is archimedian. These assumptions imply, we are able to apply Putinar’s Positivstellensatz to polynomials positive on K. At first we note, p⋆ = sup {a ∈ R | p − a ≥ 0 on K} = sup {a ∈ R | p − a > 0 on K} . Since we assume that M (K) archimedian, we apply Theorem 2.7 to (2.3). Thus p⋆ ≤ sup {a ∈ R | p − a ∈ M (K)} ≤ sup {a ∈ R | p − a ≥ 0 on K} = p⋆ . (2.3) 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION 19 Finally we obtain p⋆ = sup {a ∈ R | p − a ∈ M (K)} . As a second approach, we note for the minimum p⋆ of (2.1) holds Z ⋆ p = inf p dµ | µ ∈ MP (K) , (2.4) (2.5) where MP (K) ⊆ M(K) denotes the set R of all Borel measures on K which are also probability measures. ≤′ holds since p(x) ≥ p⋆ on K implies p dµ ≥ p⋆ . And ′ ≥′ follows as each x feasible in (2.1) corresponds to a µ = δx ∈ M(K), where δx the Dirac measure at x. In order to get rid of the set M(K) in (2.5) we exploit the following theorem by Putinar [81]. ′ Theorem 2.9 For any map L : R [x] → R, the following are equivalent: (i) L is linear, L(1) = 1 and L(M (K)) ⊆ [0, ∞) . (ii) L is integration with respect to a probability measure µ on K, i.e., Z ∃ µ ∈ MP (K) : ∀ p ∈ R [x] : L(p) = p dµ. Proof C.f. [88], pp. 10. This theorem does not really characterize MP (K), but all real families (yα )α∈Nn that are sequences of moments of probability measures on K, i.e., Z yα = xα dµ ∀ α ∈ Nn , αn 1 where xα = xα 1 · · · xn . This statement is true, as every linear map L : R [x] → R is given uniquely by its α values L(x ) on the basis (xα )α∈Nn of R [x]. With Theorem 2.9 we obtain p⋆ = inf {L(f ) | L : R [x] → R is linear , L(1) = 1, L(M (K)) ⊆ [0, ∞)} . (2.6) Recall (2.4) as p⋆ = sup {a ∈ R | f − a ∈ M (K)} . Thus (2.6) can be understood as a primal approach to the original problem (2.1) and (2.4) as a dual approach. Due to complexity reasons it is necessary to introduce relaxations to these primal-dual pair of optimization problems, in order tonsolve the problem (2.1). Therefore we approximate M (K) by the sets o P Pm 2 R [x] , deg(σi gi ) ≤ 2ω for an Mω (K) ⊆ R [x], where Mω (K) := i=0 σi gi | σi ∈ ω ∈ N := {s ∈ N | s ≥ ωmax := max {ω0 , ω1 , . . . , ωm }} , degp i ωi := ⌈ degg 2 ⌉ (i = 1, . . . , m), ω0 := ⌈ 2 ⌉. Replacing M (K) by Mω (K) motivates to consider the following pair of optimization problems for a ω ∈ N : (Pω ) min (Dω ) max L(p) subject to a subject to L : R [x]2ω → R is linear, L(1) = 1 and L (Mω (K)) ⊆ [0, ∞) . a ∈ R and p − a ∈ Mω (K). (2.7) The optimal values of (Pω ) and (Dω ) are denoted by Pω⋆ and Dω⋆ , respectively. The parameter ω ∈ N is called the relaxation order of (2.7). It determines the size of the relaxations (Pω ) and (Dω ) to (2.2) and therefore also the numerical effort that is necessary to solve them. 20 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Theorem 2.10 (Lasserre) Assume M (K) is archimedian. (Pω⋆ )ω∈N and (Dω⋆ )ω∈N are increasing sequences that converge to p⋆ and satisfy Dω⋆ ≤ Pω⋆ ≤ p⋆ for all ω ∈ N . Moreover, if p − p⋆ ∈ M (K), then Dω⋆ = Pω⋆ = p⋆ for a sufficiently large relaxation order ω, i.e. strong duality holds. Proof Since the feasible set of (2.6) is a subset of the feasible set of (Pω ), Pω⋆ ≤ p⋆ . Moreover, if L feasible for (Pω ) and a for (Dω ), L(p) ≥ a holds since p−a ∈ Mω (K) implies L(p)−a = L(p)−aL(1) = L(p−a) ≥ 0. Thus Dω⋆ ≤ Pω⋆ . Obviously, a feasible a for (Dω ) is feasible for (Dω+1 ), and every feasible L of (Pω+1 ) is feasible for (Pω ). This implies (Pω⋆ )ω∈N and (Dω⋆ )ω∈N are increasing. Furthermore, as for any ǫ > 0 there exists a sufficiently large ω ∈ N such that p − p⋆ + ǫ ∈ Mω (K) by Theorem 2.7, i.e. p⋆ − ǫ feasible for (Dω ), the convergence follows. If p − p⋆ ∈ M (K), p − p⋆ ∈ Mω (K) for ω sufficiently large. Thus p⋆ feasible for (Dω ) und therefore Dω⋆ = Pω⋆ = p⋆ . If M (K) not archimedian, we are still able to exploit Schmuedgen’s Positivstellensatz to characterize p − a in (Dω ). As a next step we follow Lasserre’s observation and translate (Dω ) and (Pω ) into a pair of primal-dual semidefinite programs. Definition 2.3 Let L : R[x] → R be linear functional, a sequence y = (yα )α∈Nn be given by yα := L(xα ), and d ∈ N fixed. The moment matrix Md (y) of order d is the matrix with rows and columns indexed by u(x, Λ(d)), such that Md (y)α,β = L(xα xβ ) = yα+β ∀α, β ∈ Nn with | α |, | β |≤ d. n+d The size of Md (y) is given by the | u(x, Λ(d)) |= , the number of components of y needed for d constructing Md (y) is given by n + 2d | (yα )|α|≤2d |= . 2d In the case n = d = 2, the moment matrix is given by y00 y10 y01 y20 y11 y02 y10 y20 y11 y30 y21 y12 y01 y11 y02 y21 y12 y03 . M2 (y) = y20 y30 y21 y40 y31 y22 y11 y21 y12 y31 y22 y13 y02 y12 y03 y22 y13 y04 P Let g ∈ R[x] with g(x) = α∈Nn gα xα . The localizing matrix Md (y) of order d associated with g and y is the matrix with rows and columns indexed in ud (x), obtained from the moment matrix by X Md (g y)α,β := L(g(x)xα xβ ) = hγ yγ+α+β ∀α, β ∈ Nn , | α |, | β |≤ d. γ∈Nn For g(x) = x21 + 2x2 + 3, n = 2 and d = 1, the localizing matrix is given by y20 + 2y01 + 3y00 y30 + 2y11 + 3y10 y21 + 2y02 + 3y01 M1 (g y) = y30 + 2y11 + 3y10 y40 + 2y21 + 3y20 y31 + 2y12 + 3y11 y21 + 2y02 + 3y01 y31 + 2y12 + 3y11 y22 + 2y03 + 3y02 . We will exploit the following key lemma [88]. Lemma 2.1 Suppose L : R [x] → R is a linear map. Then L(Mω ) ⊆ [0, ∞) if and only if the m+ 1 matrices Mω−ωi (y gi ) < 0, ∀ i ∈ {0, . . . , m} , 21 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION where g0 := 1, ω0 = 0 and the sequence y defined by yα := L(xα ). Moreover, ) (m X s(ω−ωi ) . hMω−ωi (u2ω−2ωi (x) gi ), Gi i | G0 , . . . , Gm ∈ S+ Mω (K) = i=0 Proof C.f. [88], p. 19. Using this Lemma, we reformulate (2.7) as P dSDPω min α∈Λ(2ω) yα pα s.t. y ∈ R|Λ(2ω)| , y0 = 1 and Mω−ωi (y gi ) < 0, i = 1, . . . , m Mω (y) < 0, dSDP⋆ω max a s(ω) s(ω−ωi ) for i = 1, . . . , m and s.t. a P∈mR, G0 ∈ S+ , Gi ∈ S+ i=0 hMω−ωi (u(x, Λ(2ω)gi ), Gi i = p − a, (2.8) P α where p(x) = α∈Λ(2ω) pα x . We call dSDPω the dense Lasserre relaxation or the dense SDP relaxation of relaxation order ω for the polynomial optimization problem (2.1). By sorting the monomials in the moment and localizing matrix inequality constraints in (2.8), we can express Mω (u(x, Λ(2ω))) and Mω−ωi (u(x, Λ(2Ω) gi )) as Mω (u(x, Λ(2ω))) = X α∈Λ(2ω) Bα xα , Mω−ωi (u(x, Λ(2ω − 2ωi )) gi ) = X Cαi xα , α∈Λ(2ω) for some matrices Bα ∈ Ss(ω) and Cαi ∈ Ss(ω−ωi ) . Thus we can rewrite the primal-dual pair of SDP (2.8) as the primal-dual pair of equivalent SDP in standard form P (PωSDP ) min α∈Λ(2ω) pα yα |Λ(2ω)| s.t. y ∈ RP , y0 = 1, and B0 + α∈Λ(2ω)\{0} yα Bα < 0, P C0i + α∈Λ(2ω)\{0} yα Cαi < 0, i = 1, . . . , m (2.9) Pm (DωSDP ) max −G0 (1, 1) − i=1 hC0i , Gi i s(ω) s(ω−ωi ) s.t. a ∈ R, G0 ∈PS+ , Gi ∈ S+ for i ∈ {1, . . . , m} and m hBα , G0 i + i=1 hCαi , Gi i = pα , 0 6= α ∈ Λ(2ω) In general SDP can be solved in polynomial time. Efficient solvers for the SDP (2.9) in standard form are provided by the software packages SeDuMi [95] and SDPA [104]. Lasserre’s relaxation in the unconstrained case The procedure to derive a sequence of convergent SDP relaxations in the case of an unconstrained polynomial optimization problem minn p(x), (2.10) x∈R where p ∈ R [x] and p⋆ := minx p(x), is similar to the constrained case, which we discussed in the previous subsection. Let p be of even degree 2l, otherwise inf p = −∞. Moreover, we will exploit the characterization of sum of squares decompositions by semidefinite matrices and Putinar’s Positivstellensatz. In order to apply this theorem, it is necessary to construct an appropriate semialgebraic set. First, we derive the following relaxation, R p⋆ = inf n p dµ | µ ∈ MP (Rn ) o (2.11) s(l) ≥ inf L(p) | L : R [x] → R, L(1) = 1, Ml (L(x)) ∈ S+ . 22 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION We order the expression Ml (L(x)) and introduce symmetric matrices Bα ∈ Ss(l) such that Ml (L(x)) = P α α α∈Λ(2l) Bα L(x ). Finally we identify yα = L(x ) for α ∈ Λ(2l) \ {0} and y0 = 1 to obtain the following relaxation for (2.10) P (Pl ) min Pα pα yα (2.12) s.t. α6=0 yα Bα < −B0 . As in the constrained case we can apply a dual approach to (2.10), P p⋆ = sup n {a ∈ R | p(x) − a ≥ 0 ∀ x ∈ Rn } ≥ sup o a ∈ R | p(x) − a ∈ R[x]2 s(l) . = sup a | p(x) − a = hMl (x) , Gi, G ∈ S+ (2.13) Thus, we derive another relaxation to problem (2.10), (Dl ) max −G(1, 1) s.t. hBα , Gi = pα , G < 0. α 6= 0 (2.14) With the duality theory of convex optimization it can be shown easily, that the two convex programs (2.12) and (2.14) are dual to each other. In the case (2.14) has an interior feasible solution, strong duality holds, that is Pl⋆ = Dl⋆ . The idea of the following theorem was proposed by Shor [91] first. The presented version is due to Lasserre [51]. Theorem 2.11 (Shor) If the nonnegative polynomial p−p⋆ is a sum of squares of polynomials, then (2.10) is equivalent to (2.12). More precisely, p⋆ = ZP and, if x⋆ is a global minimizer of (2.10), then y ⋆ := x⋆1 , . . . , x⋆n , (x⋆1 )2 , x⋆1 x⋆2 , . . . , (x⋆1 )2m , . . . , (x⋆n )2m is a minimizer of (2.12). Next, we treat the general case, that is, when p − p⋆ is not sum of squares. As mentioned at the beginning we have to construct a semialgebraic set in order to be able to apply Putinar’s Positivstellensatz. Suppose we know that a global minimizer x⋆ of p(x) has norm less than a for some a > 0, that is, p(x⋆ ) = p⋆ and || x⋆ ||2 ≤ a. Then, with x → qa (x) = a2 − || x ||22 , we have p(x) − p⋆ ≥ 0 on Ka := {qa (x) ≥ 0}. Obviously, M (Ka ) is archimedian, as the condition (iii) in Theorem 2.8 is satisfied for N = a2 . Now, we can use that every polynomial f , strictly positive on the semialgebraic set Ka is contained in the quadratic module M (Ka ). For every ω ≥ l, consider the following semidefinite program P (Pωa ) min α pα y α , s.t. Mω (y) < 0, (2.15) Mω−1 (qa y) ≥ 0. Writing Mω−1 (qa y) = nite program P α yα Dα , for appropriate matrices Dα (| α |≤ 2ω), the dual of (Pωa ) is the semidefi(Dωa ) max s.t. −G(1, 1) − a2 H(1, 1), hG, Bα i + hH, Dα i = pα , α 6= 0. (2.16) Then, the following theorem is due to Lasserre [51]. Theorem 2.12 (Lasserre) Given (Pωa ) and (Dωa ) for some a > 0 such that || x⋆ ||2 ≤ a for some global minimizer x⋆ . Then 23 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION (a) as ω → ∞, one has inf(Pωa ) ↑ p⋆ . Moreover, for ω sufficiently large, there is no duality gap between (Pωa ) and its dual (Dωa ), and (Dωa ) is solvable. (b) min(Pωa ) = p⋆ if and only if p − p⋆ ∈ Mω (Ka ). In this case, the vector y ⋆ := x⋆1 , . . . , x⋆n , (x⋆1 )2 , x⋆1 x⋆2 , . . . , (x⋆1 )2ω , . . . , (x⋆n )2ω is a minimizer of (Pωa ). In addition, max(Pωa ) = min(Dωa ). Proof (a) From x⋆ ∈ Ka and with y ⋆ := x⋆1 , . . . , x⋆n , (x⋆1 )2 , x⋆1 x⋆2 , . . . , (x⋆1 )2ω , . . . , (x⋆n )2ω it follows that Mω (y ⋆ ), Mω−1 (qa y ⋆ ) < 0 so that y ⋆ is feasible for (Pωa ) and thus inf(Pωa ) ≤ p⋆ . Now, fix ǫ > 0 arbitrary. Then, p − p⋆ + ǫ > 0 and therefore, with Theorem 2.7 there is some N0 such that r2 r1 X X tj (x)2 qi (x)2 + q(x) p − p⋆ + ǫ = i=1 j=1 for some polynomials qi (x), i = 1, . . . , r1 , of degree at most N0 , and some polynomials tj (x), j = 1, . . . , r2 , of degree at most N0 − 1. Let qi ∈ Rs(N0 ) , tj ∈ Rs(N0 −1) be the corresponding vectors of coefficients, and let r2 r1 X X T tj tTj qi qi , Z := G := i=1 j=1 so that G, H < 0. It is immediate to check that (G, H) feasible for (Dωa ) with value −G(1, 1) − a2 H(1, 1) = (p⋆ − ǫ). From weak duality follows convergence as p⋆ − ǫ ≤ inf(Pωa ) ≤ p⋆ . For strong duality and for (b), c.f. [51]. We needed to add the constraint qa (x) ≥ 0, in order to show convergence of the SDP relaxation (Pωa ). For applications it is often sufficient to consider an SDP relaxation, which does not take into account this constraint. Thus, we denote the primal-dual pair of SDP dSDPω dSDP⋆ω P min α pα y α s.t. Mω (y) < 0, max −G(1, 1) s.t. hG, Bα i = pα ∀α 6= 0, as the dense SDP relaxation of relaxation order ω for polynomial optimization problem (2.10), which is consistent with the dense SDP relaxation for the constrained case. This sequence of SDP is not guaranteed to converge to the minimum of (2.10) for ω → ∞. However, it provides a non-decreasing sequence of lower bounds to p⋆ , min(dSDPωmax ) ≤ min(dSDPωmax +1 ) ≤ . . . ≤ p⋆ . 24 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Global minimizer Usually one is not only interested in finding the minimum value p⋆ of p on K, but also in obtaining a global minimizer x⋆ ∈ K ⋆ with p(x⋆ ) = p⋆ . It will be shown that in Lasserre’s procedure not only min(dSDPω ) converges to the infimum p⋆ , but also a convergence to the minimizer x⋆ of (2.2) in the case it is unique. Definition 2.4 Lω solves (Pω ) nearly to optimality (ω ∈ N ) if Lω is a feasible solution of (Pω ) (ω ∈ N ) such that limω→∞ Lω (p) = limω→∞ Pω⋆ . This notation is useful because (Pω ) might not possess an optimal solution, and even if it does, we might not be able to compute it exactly. For an example, c.f. [88], Example 22. Obviously Lω solves (Pω ) nearly to optimality (ω ∈ N ) if and only if limω→∞ Lω (f ) = p⋆ . The following theorem is the basis for the convergence to a minimizer in the case K ⋆ is a singleton. Theorem 2.13 Suppose K 6= ∅ and Lω solves (Pω ) nearly to optimality (ω ∈ N ). Then Z α α ⋆ ∀d ∈ N : ∀ǫ > 0 : ∃k0 ∈ N ∩ [d, ∞) : ∀k ≥ k0 : ∃µ ∈ M(K ) : Lω (x ) − x dµ < ǫ. α∈Λ(2d) Proof [88], p. 11. In the convenient case where K ⋆ is a singleton it is possible to guarantee convergence of the minimizer: Corollary 2.2 K ⋆ = {x⋆ } is a singleton and Lω solves (Pω ) nearly to optimality (ω ∈ N ). Then lim (Lω (x1 ), . . . , Lω (xn )) = x⋆ . ω→∞ Proof We set d = 1 in Theorem 2.13 and note that M(K ⋆ ) contains only the Dirac measure δx⋆ at the point x⋆ . It is possible to apply Corollary 2.2 to certify that p⋆ has almost been reached after successively solving the relaxations (Pω ). Corollary 2.3 Suppose M (K) is archimedian, p has a unique minimizer on the compact semialgebraic set K and Lω solves (Pω ) nearly to optimality for all ω ∈ N . Then holds for all ω ∈ N , Lω (p) ≤ p⋆ ≤ p(Lω (x1 ), . . . , Lω (xn )), and the lower and upper bounds for p⋆ converge to p⋆ for ω → ∞. Proof Lω (p) ≤ p⋆ follows from Theorem 2.10. The convergence of p(Lω (x1 ), . . . , Lω (xn )) is a consequence of Corollary 2.2. To see that p⋆ is a lower bound, observe that gi (Lω (x1 ), . . . , Lω (xn )) = Lω (gi ) ≥ 0, whence (Lω (x1 ), . . . , Lω (xn )) ∈ K for all k ∈ N . The case where several optimal solutions exist is more difficult to handle. In fact, as soon as there are two or more global minimizers, it often occurs that symmetry in the problem prevents the nearly optimal solutions of the SDP relaxations to converge to a particular minimizer. Henrion and Lasserre established a sufficient condition for the dense SDP relaxations to detect all optimal solutions [35]. Given the dense SDP relaxation dSDPω for some order ω ≥ ωmax and y ⋆ an optimal solution of this semidefinite program. In the case rankMω (y ⋆ ) = rankMω−ωmax (y ⋆ ) (2.17) ⋆ holds, the SDP relaxation dSDPω is exact. That is, min(dSDPω ) = p . Moreover, Henrion and Lasserre provided an algorithm for extracting all global optimal solutions of the POP (2.1), if (2.17) holds. See [35] for details. Note, (2.17) is not necessary, dSDPω may be exact already for some relaxation order ω with ω until rankMω (yω⋆ ) > rankMω−ωmax (yω⋆ ). For many POP it may not be practical to increase (2.17) holds, n+ω as the size of the moment and localizing matrix constraints in dSDPω given by grows rapidly n for increasing ω. 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION 2.1.4 25 Sparse SDP relaxations for polynomial optimization problems The dense SDP relaxation (2.8) by Lasserre is a powerful theoretical result since it allows to approximate the solutions of polynomial optimization problems (2.1) as closely as desired by solving a finite sequene of n+ω SDP relaxations. However, since the size of the SDP relaxation grows as , even for medium scale ω POPs the SDP relaxations become intractable for present SDP solvers in the case of small choices of the relaxation order ω. Therefore, it is crucial to reduce the size of the semidefinite programs to be solved, in order to be able to attempt large scale POPs. In this section we will review the approach [102] to exploit sparsity in a large scale POP by introducing a sequence of sparse SDP relaxations, which is of much smaller size than the dense SDP relaxations (2.8). A second method to exploit sparsity in a general optimization problem with linear and/ or nonlinear matrix inequality constraints is presented in [42]. In many problems of type (2.1), the involved polynomials p, g1 , . . . , gm are sparse. Waki, Kojima, Kim and Muramatsu constructed a sequence of SDP relaxations which exploits the sparsity of such polynomial optimization problems [102]. This method shows strong numerical efforts in comparision to Lasserre’s relaxations (2.8). The convergence of the sparse SDP relaxations to the optimum of the original problem (2.1) was shown by Lasserre [52] and Kojima and Muramatsu [49]. In the following, we give a review of the sparse SDP relaxations for POP with structured sparsity. Let the polynomial optimization problem be given as in (2.2), minx∈K p(x), where K is a compact semialgebraic set defined by the m inequality constraints g1 ≥ 0, . . . , gm ≥ 0. We characterize sparsity for a POP (2.2) with the following definition. Definition 2.5 Given a POP (2.2), we denote the n × n symbolic matrix R defined by ⋆, if xi xj occurs in some monomial of p, Ri,j = ⋆, if xi and xj occur in the same gl (l = 1, . . . , m), 0, else, as the correlative sparsity pattern matrix of the POP. The graph G = (V, N ) with vertex set V := {1, . . . , n} and edge set N := {i, j} ∈ V 2 | Ri,j = ⋆ , is called the corresponding correlative sparsity pattern graph. A POP is defined to be correlatively sparse, if R is sparse. We will construct a sequence of SDP relaxations to this polynomial optimization problem, which exploits the sparsity pattern characterized by the correlative sparsity pattern matrix R. Under a certain condition on the sparsity pattern of the problem, the optima of these SDP relaxations converge to the optimum of the polynomial optimization problem (2.2). First, let {1, . . . , n} be the union ∪qk=1 Ik of subsets Ik ⊂ {1, . . . , n}, such that every gj , j ∈ {1, . . . , m} is only concerned with variables {xi | i ∈ Ik } for some k. And it is required the objective p can be written as p = p1 + . . . + pq where each pk uses only variables {xi | i ∈ Ik }. A possible choice for the sets I1 , . . . , Ip are the maximal cliques of the correlative sparsity pattern graph G. In order to tackle the sparse SDP relaxations we need some further definitions. Definition 2.6 Given a subset I of {1, . . . , n} we define the sets AI AIω = {α ∈ Nn : αi = 0 if i ∈ / I} , P = α ∈ Nn : αi = 0 if i ∈ / I and i∈I αi ≤ ω . Then, we define R [x, G] := {f ∈ R [x] : supp(f ) ⊆ G}. Also, the restricted moment matrix Mr (y, I) and localizing matrices Mr (gy, I) are defined for I ⊆ {1, . . . , n} , r ∈ N and g ∈ R [x]. They are 26 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION obtained from Mr (y) and Mr (gy) by retaining only those rows (and columns) α ∈ Nn of Mr (y) and Mr (gy) with supp(α) ⊆ AIr . In doing so, Mr (y, I) and Mr (gy, I) can be interpreted as moment and localizing matrices with rows and columns indexed in the canonical basis u(x, AIr ) of R x, AIr . Finally, we denote the P P 2 2 set of sum of square polynomials in R [x, G] as R [x, G] . In analogy to Theorem 2.3, R [x, G] can be written as X 2 R [x, G] = u(x, G)T V u(x, G) : V < 0 . Let m′ be the number of inequality constraints which define the basic, closed, semialgebraic set K. In our initial setting m′ = m, but later on m′ > m may hold, in the case we add further inequality constraints to restrict the set K. We introduce a condition for the index sets I1 , . . . , Iq . Assumption 1: Let K ⊆ Rn as in (2.23). The index set J = {1, . . . , m′ } is partitioned into q disjoint sets Jk , k = 1, . . . , q, and the collections {Ik } and {Jk } satisfy: 1. For every j ∈ Jk , gj ∈ R x, AIk , that is, for every j ∈ Jk , the constraint gj (x) ≥ 0 is only concerned with the variables x(Ik ). Equivalently, viewing gj as a polynomial in R [x] , gjα 6= 0 ⇒ supp(α) ∈ AIk . 2. The objective function p ∈ R [x] can be written p= q X k=1 pk , with pk ∈ R x, AIk , k = 1, . . . , q. Equivalently, fα 6= 0 ⇒ supp(α) ∈ ∪qk=1 AIk . Example 2.4 For n = 6 and m = 6, let g1 (x) = x1 x2 − 1, g2 (x) = x21 + x2 x3 − 1, g3 (x) = x2 + x23 x4 , and g4 (x) = x3 + x5 , g5 (x) = x3 x6 , g6 (x) = x2 x3 . Then we can construct {Ik } and {Jk } for q = 4 with I1 = {1, 2, 3} , I2 = {2, 3, 4} , I3 = {3, 5} , I4 = {3, 6} , J1 = {1, 2, 6} , J2 = {3} , J3 = {4} , J4 = {5} . Now, we can construct sparse SDP relaxations in analogy to the dense SDP relaxations (2.8). For each deg g j = 1, . . . , m′ write ωj = ⌈ 2 j ⌉. Then, for some ω ∈ N define the following semidefinite program (sSDPω ) inf y s.t. P α pα y α Mω (y, Ik ) < 0, k = 1, . . . , q, Mω−ωj (gj y, Ik ) < 0, j ∈ Jk ; k = 1, . . . , q, y0 = 1. (2.18) Program (2.18) is well defined under Assumption 3, and it is easy to see, that it is an SDP relaxation of problem (2.2). In fact, it is also easy to see, that sSDPω is a weaker relaxation for (2.2) than dSDPω , as the partial moment and localizing matrices in the constraints of (2.18) are minors of the full moment and localizing matrices in the constraints of (2.8), i.e., min(sSDPω ) ≤ min(dSDPω ) ≤ min(POP) ∀ω ∈ N . We call (2.18) the sparse SDP relaxations relaxations for polynomials optimization Lasserre or sparse problems. There are symmetric matrices Bαk and Cαjk such that P Mω (y, Ik ) = Pα∈Nn yα Bαk , k = 1, . . . , q, Mω−ωj (gj y, Ik ) = α∈Nn yα Cαjk , k = 1, . . . , q, j ∈ Jk , (2.19) 2.1. POSITIVE POLYNOMIALS AND POLYNOMIAL OPTIMIZATION with Bαk = 0 and Cαjk = 0 whenever supp(α) ∈ / AIk . Then we can rewrite (2.18) as P inf y α pα y α X s.t. yα Bαk < −B0k , k = 1, . . . , q, 06X =α∈Nn yα Cαjk < 06=α∈Nn −C0jk , 27 (2.20) j ∈ Jk ; k = 1, . . . , q, and we derive the dual of this semidefinite program as (sSDP⋆ω ) sup X Yk ,Zjk ,λ k:α∈AIk λ i h P hYk , Bαk i + j∈Jk hZjk , Cαjk i + λδα0 = pα ∀ α ∈ Γω , (2.21) Yk , Zjk < 0, j ∈ Jk , k = 1, . . . , q, Sq Ik n where Γω := α ∈ N : α ∈ k=1 A ; | α |≤ 2ω . The main advantage of the sparse SDP relaxations is the reduction of the size of the matrix inequality constraints. In order to understand the improved efficiency, let us compare the computational complexity of theP dense relaxation (PωSDP ) and the sparse relaxation (Pωsp ). The number of variables in (Pωsp ) is bounded q nk +2ω by k=1 . Supposed nk ≈ nq for all k, the number of variables is bounded by O(q( nq )2ω ), a strong ω improvement compared with O(n2ω ), the number of variables in (PωSDP ). Also in (Pωsp ) there are p LMI constraints of size O(( nq )ω ) and m + q LMI constraints of size O(( nq )ω−ωmax ), to be compared with a single LMI constraint of size O(nω ) and m LMI constraints of size O(nω−ωmax ) in (PωSDP ). As pointed out, the sparse SDP relaxations are weaker than the dense ones. The question arises, whether we still have convergence to the minimum of the POP. This question was answered positively by Lasserre [52]. We need two further conditions to show convergence. Assumption 2: For all k = 1, . . . , q − 1, Ik+1 ∩ k [ j=1 Ij ⊆ Is for some s ≤ k. (2.22) The property of Assumption 2 is called the running intersection property. Note that (2.22) is always satsfied for q = 2. Since property (2.22) depends on the ordering, it can be satisfied possibly after some relabelling of the {Ik }. In the case of Example 2.4 it is easy to check Assumption 2 is satisfied, but in general it is not obvious. However, Waki et al. [102] presented a general procedure to guarantee Assumption 2 is satisfied. Given G = (V, E) the correlative sparsity pattern graph, we denote by G̃ = (V, Ẽ) its chordal extension. A graph is said to be chordal if every (simple) cycle of the graph with more than three edges has a chord. A graph G(V, E) is a chordal extension of G(V, E) if it is a chordal graph and E ⊆ E. See [4] for basic properties on choral graphs. Then, the maximal cliques C1 , . . . , Cq of G̃ satisfy the running intersection property, and the number q of maximal cliques in a chordal graph is bounded by n. Furthermore, there are efficient algorithms to determine the maximal cliques of a chordal graph, whereas it is NP-hard to determine the maximal cliques of an arbitrary graph. Assumption 3: Let K ⊆ Rn be a closed semialgebraic set. Then, there is M > 0 such that || x ||∞ < M for all x ∈ K. This assumption implies || x(Ik ) ||2∞ < nk M 2 , k = 1, . . . , q, where x(Ik ) := {xi | i ∈ Ik }, and therefore we add to K the q redundant quadratic constraints gm+k (x) := nk M 2 − || x(Ik ) ||≥ 0, k = 1, . . . , q, and set m′ = m + q, so that K is now defined by: K := {x ∈ Rn | gj (x) ≥ 0, j = 1, . . . , m′ } . (2.23) i h Notice that gm+k ∈ R x, AI2k for every k = 1, . . . , q. With Assumption 3, K is a compact semialgebraic set. Moreover, Assumption 3 is needed to guarantee the quadratic module M (K) is archimedian, the condition of Putinar’s Positivstellensatz. Finally, we obtain the following convergence result. 28 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Theorem 2.14 Let p⋆ denote the global minimum of (2.2) and let Assumption 1-3 hold. Then: (a) inf(sSDPω ) ↑ p⋆ as ω → ∞. (b) If K has nonempty interior, then strong duality holds and (sSDP⋆ω ) solvable for sufficiently large ω, i.e., inf(sSDPω ) = max(sSDP⋆ω ). (c) Let y ω be a nearly optimal solution of (sSDPω ), with e.g. X α pα yα ≤ inf(sSDPω ) + 1 , ω ∀ω ≥ ω0 , and let ŷ ω := {yαω : | α |= 1}. If (2.2) has a unique global minimizer x⋆ ∈ K, then ŷ ω → x⋆ ω → ∞. as Proof C.f. [52]. As in the dense case, it is also possible to extract global minimizers of the POP from the sparse SDP relaxations in certain cases where the minimizer of the POP is not unique. In fact, Lasserre derived the following sparse version of condition (2.17): Given y ⋆ is an optimal solution for the sparse SDP relaxation sSDPω for some order ω ≥ ωmax . If the rank conditions, rankMω (y ⋆ , Ih ) = rankMω−ah (y ⋆ , Ih ) ⋆ rankMω (y , Ih ∩ Ih′ ) = 1 ∀ h ∈ {1, . . . , q}, ∀h 6= h′ with Ih ∩ Ih′ 6= ∅, (2.24) with ah := maxj∈Jh ωj , hold, then sSDPω is exact and all global minimizers can be extracted. However, (2.24) are very restrictive sufficient conditions for the SDP relaxations to be exact, and it is not practical to apply them to large scale POP in most cases. The software SparsePOP [103] is an implementation of the sparse SDP relaxations. The running intersection property is guaranteed by choosing the maximal cliques of the chordal extension of the correlative sparsity pattern graph as the index sets I1 , . . . , Iq in (2.18). Instead of imposing the additional constraints of Assumption 3, in SparsePOP linear box constraints are imposed for each component of x ∈ Rn , lbdi ≤ xi ≤ ubdi ∀ i ∈ {1, . . . , n}. (2.25) Moreover, SparsePOP adds small linear perturbation terms to the objective function of the POP, in order to enforce the POP to have a unique global minimizer. 2.2 Exploiting sparsity in linear and nonlinear matrix inequalities Optimization problems with nonlinear matrix inequalities, including quadratic and polynomial matrix inequalities, are known as hard problems. They frequently belong to large-scale optimization problems. We present a basic framework for exploiting the sparsity characterized in terms of a chordal graph structure via positive semidefinite matrix completion [28]. Depending on where the sparsity is observed, two types of sparsities are studied: the domain-space sparsity (d-space sparsity) for a symmetric matrix X that appears as a variable in objective and/or constraint functions of a given optimization problem and is required to be positive semidefinite, and the range-space sparsity (r-space sparsity) for a matrix inequality involved in the constraint of the problem. The d-space sparsity is basically equivalent to the sparsity studied by Fukuda et. al [21, 68] for an equality standard form SDP. One of the two d-space conversion methods proposed in this section corresponds to an extension of their conversion method, and the other d-space conversion method is an extension of the method used for the sparse SDP relaxation of polynomial optimization problems in [102, 103] and for the sparse SDP relaxation of a sensor network localization problem in [43]. The r-space sparsity concerns with a matrix inequality M (y) < 0, (2.26) 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 29 involved in a general nonlinear optimization problem. Here M denotes a mapping from Rs into Sn . If M is linear, (2.26) is known as a linear matrix inequality (LMI), which appears in the constraint of a dual standard form of SDP. If each element of M (y) is a multivariate polynomial function in y ∈ Rs , (2.26) is called a polynomial matrix inequality and the SDP relaxation [36, 37, 46, 48, 49, 52], which is an extension of the SDP relaxation [51] for POP, can be applied to (2.26). We assume a similar chordal graph structured sparsity as the d-space sparsity on the set of row and column index pairs (i, j) of the mapping M such that Mij is not identically zero, i.e., Mij (y) 6= 0 for some y ∈ Rs . A representative example satisfying the r-space sparsity can be found with tridiagonal M . We do not impose any additional assumption on (2.26) to derive a r-space conversion method. When M is polynomial in y ∈ Rs , we can effectively combine it with the sparse SDP relaxation method [46, 49] for polynomial optimization problems over symmetric cones to solve (2.26). We propose two methods to exploit the r-space sparsity. One may be regarded as a dual of the d-space conversion method by Fukuda et. al [21]. More precisely, it exploits the sparsity of the mapping M in the range space via a dual of the positive semidefinite matrix completion to transform the matrix inequality (2.26) to a system of multiple matrix inequalities with smaller sizes and an auxiliary vector variable z ∈ Rq . The resulting matrix inequality system is of the form fk (y) − L e k (z) < 0 (k = 1, 2, . . . , p), M (2.27) fk denotes a mapping from and y ∈ Rs is a solution of (2.26) if and only if it satisfies (2.27) for some z. Here M s k e R into the space of symmetric matrices with some size and L a linear mapping from Rq into the space of fk (k = 1, 2, . . . , p) symmetric matrices with the same size. The sizes of symmetric matrix valued mappings M and the dimension q of the auxiliary variable vector z are determined by the r-space sparsity pattern of M . fk are all 2 × 2 and q = n − 2. The other r-space conversion For example, if M is tridiagonal, the sizes of M method corresponds to a dual of the second d-space conversion method mentioned previously. We discuss how the d-space and r-space conversion methods enhance the correlative sparsity for POP introduced in the previous section. Furthermore, we present numerical results to demonstrate how the size of problems involving large scale matrix inequalities is reduced under the four proposed conversion methods. 2.2.1 An SDP example A simple SDP example is shown to illustrate the two types of sparsities considered in this paper, the d-space sparsity and the r-space sparsity, and compare it to the correlative sparsity from 2.1.4 that characterizes the sparsity of the Schur complement matrix. Let A0 be a tridiagonal matrix in Sn such that A0ij = 0 if |i − j| > 1, and define a mapping M from Sn into Sn by 1 − X11 0 0 M (X) = ... 0 X21 0 1 − X22 0 ... 0 X32 0 0 .. . ... 0 X43 ... ... .. 0 0 X12 X23 0 X34 . ... 1 − Xn−1,n−1 ... Xn,n−1 for every X ∈ Sn . Consider an SDP minimize A0 • X subject to M (X) < 0, X < 0. ... Xn−1,n 1 − Xnn (2.28) Among the elements Xij (i = 1, 2, . . . , n, j = 1, 2, . . . , n) of the matrix variable X ∈ Sn , the elements Xij with |i − j| ≤ 1 are relevant and all other elements Xij with |i − j| > 1 are unnecessary in evaluating the objective function A0 • X and the matrix inequality M (X) < 0. Hence, we can describe the d-sparsity 30 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION pattern as a symbolic tridiagonal matrix with the nonzero symbol ⋆ ⋆ ⋆ 0 ... ⋆ ⋆ ⋆ ... .. 0 . ⋆ ⋆ . . .. . . . . . . .. . 0 0 . . . .. 0 0 ... ... On the other hand, the r-space sparsity pattern ⋆ 0 0 ⋆ ... ... 0 0 ⋆ ⋆ 0 0 0 .. . ⋆ ⋆ 0 0 0 . ... ⋆ ⋆ is described as ... 0 ⋆ ... 0 ⋆ .. . . ... ... ... ⋆ ⋆ ... ⋆ ⋆ Applying the d-space conversion method using basis representation described in 2.2.3, and the r-space conversion method using clique trees presented in 2.2.5, we can reduce the SDP (2.28) to n−1 X 0 0 0 Aii Xii + 2Ai,i+1 Xi,i+1 + Ann Xnn minimize i=1 1 0 X11 −X12 subject to − < 0, 0 0 −X −z 21 1 1 0 Xii −Xi,i+1 (2.29) < 0 (i = 2, 3, . . . , n − 2), − 0 0 −Xi+1,i zi−1 − zi 1 0 Xn−1,n−1 −Xn−1,n < 0, − −X X + z 0 1 n,n−1 n,n n−2 −Xii −Xi,i+1 0 0 < 0 (i = 1, 2, . . . , n − 1). − −Xi+1,i −Xi+1,i+1 0 0 This problem has (3n − 3) real variables Xii (i = 1, 2, . . . , n), Xi,i+1 (i = 1, 2, . . . , n − 1) and zi (i = 1, 2, . . . , n − 2), and (2n − 1) linear matrix inequalities with size 2 × 2. Since the original SDP (2.28) involves an n × n matrix variable X and an n × n matrix inequality M (X) < 0, we can expect to solve the SDP (2.29) much more efficiently than the SDP (2.28) as n becomes larger. We can formulate both SDPs in terms of a dual standard form for SeDuMi [95]: maximize bT y subject to c − AT y < 0, where b ∈ Rl , A ∈ Rl×m and c ∈ Rm for some positive integers l and m. Table 2.2.1 shows numerical results on the SDPs (2.28) and (2.29) solved by SeDuMi. We observe that the SDP (2.29) greatly reduces the size of the coefficient matrix A, the number of nonzeros in A and the maximum SDP block compared to the original SDP (2.28). In addition, it should be emphasized that the l × l Schur complement matrix is sparse in the SDP (2.29) while it is fully dense in the the original SDP (2.28). As shown in Figure 2.1, the Schur complement matrix in the SDP (2.29) allows a very sparse Cholesky factorization. The sparsity of the Schur complement matrix is characterized by the correlative sparsity from 2.1.4. Notice a hidden correlative sparsity in the SDP (2.28), that is, each element Xij of the matrix variable X appears at most once in the elements of M (X). This leads to the correlative sparsity when the SDP (2.28) is decomposed into the SDP (2.29). The sparsity of the Schur complement matrix and the reduction in the size of matrix variable from 10000 to 2 are the main reasons that SeDuMi can solve the largest SDP in Table 1 with a 29997 × 79992 coefficient matrix A in less than 100 seconds. 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 31 SeDuMi CPU time in seconds (sizeA, nnzA, maxBl, nnzSchur) the SDP (2.28) the SDP (2.29) 0.2 (55×200,128,10,3025) 0.1 (27×72, 80,2,161) 1091.4 (5050×20000,10298,100,25502500) 0.6 (297×792,890,2,1871) OOM 6.3 (2997×7992,8990,2,18971) OOM 99.2 (29997×79992,89990,2,189971) n 10 100 1000 10000 Table 2.1: Numerical results on the SDPs (2.28) and (2.29). Here sizeA denotes the size of the coefficient matrix A, nnzA the number of nonzero elements in A, maxBl the maximum SDP block size, and nnzSchur the number of nonzeros in the Schur complement matrix. OOM means out of memory error. 0 0 5 50 100 10 150 15 200 20 250 25 0 5 10 15 nz = 94 20 25 0 50 100 150 nz = 1084 200 250 Figure 2.1: The sparsity pattern of the Cholesky factor of the Schur complement matrix for the SDP (2.29) with n = 10 and n = 100. 2.2.2 Positive semidefinite matrix completion A problem of positive semidefinite matrix completion is: Given an n × n partial symmetric matrix X with entries specified in a proper subset F of N × N , where N = {1, . . . , n}, find an X ∈ Sn+ satisfying X ij = Xij ((i, j) ∈ F ) if it exists. If X is a solution of this problem, we say that X is completed to a positive semidefinite symmetric matrix X. For example, the following 3 × 3 partial symmetric matrix 3 3 X = 3 3 2 2 2 is completed to a 3 × 3 positive semidefinite symmetric 3 X = 3 2 matrix 3 2 3 2 . 2 2 For a class of problems of positive semidefinite matrix completion, we discuss the existence of a solution and its characterization in this section. This provides a theoretical basis for both d- and r-space conversion methods. 32 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Let us use a graph G(N, E) with the node set N and an edge set E ⊆ N × N to describe a class of n × n partial symmetric matrices. We assume that (i, i) 6∈ E, i.e., the graph G(N, E) has no loop. We also assume that if (i, j) ∈ E, then (j, i) ∈ E, and (i, j) and (j, i) are interchangeably identified. Define E• Sn (E, ?) Sn+ (E, ?) = E ∪ {(i, i) : i ∈ N }, = the set of n × n partial symmetric matrices with entries specified in E • , = {X ∈ Sn (E, ?) : ∃X ∈ Sn+ ; X ij = Xij if (i, j) ∈ E • } (the set of n × n partial symmetric matrices with entries specified in E • that can be completed to positive semidefinite symmetric matrices). For a graph G(N, E) shown in Figure 2.2 as X11 X22 X33 S6 (E, ?) = X43 X61 X62 X63 Let #C = C = = S SC + X(C) = J(C) = an illustrative example, we have X16 X26 X34 X36 : Xij ∈ R (i, j) ∈ E • . X44 X45 X54 X55 X56 X65 X66 (2.30) the number of elements in C for every C ⊆ N , {X ∈ Sn : Xij = 0 if (i, j) 6∈ C × C} for every C ⊆ N, {X ∈ SC : X < 0} for every C ⊆ N , e ∈ SC such that X eij = Xij ((i, j) ∈ C × C) X for every X ∈ Sn and every C ⊆ N , {(i, j) ∈ C × C : 1 ≤ i ≤ j ≤ n} for every C ⊆ N . ′ Note that X ∈ SC is an n × n matrix although Xij = 0 for every (i, j) 6∈ C × C. Thus, X ∈ SC and X ′ ∈ SC can be added even when C and C ′ are distinct subsets of N . When all matrices involved in an equality or a matrix inequality belong to SC , matrices in SC are frequently identified with the #C × #C matrix whose elements are indexed with (i, j) ∈ C × C. If N = {1, 2, 3} and C = {1, 3}, then a matrix variable X ∈ SC ⊂ Sn has full and compact representations as follows: X11 0 X13 X11 X13 0 0 0 . and X = X= X31 X33 X31 0 X33 It should be noted that X ∈ SC ⊂ Sn has elements Xij with (i, j) ∈ C ×C in the 2×2 compact representation on the right. Let Eij = the n × n symmetric matrix with 1 in (i, j)th and (j, i)th elements and 0 elsewhere for every (i, j) ∈ N × N . Then Eij (1 ≤ i ≤ j ≤ n) form a basis of Sn . Obviously, if i, j ∈ C ⊆ N , then Eij ∈ SC . We also observe the identity X(C) = X (i,j)∈J(C) Eij Xij for every C ⊆ N. (2.31) 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 33 This identity is utilized in 2.2.3. With these notations we can now state the result from matrix completion which forms the basis for our d-space and r-space conversion techniques. Let G(N, E) be a graph and Ck (k = 1, . . . , p) be its maximal k cliques. We assume that X ∈ Sn (E, ?). The condition X(Ck ) ∈ SC + (k = 1, 2, . . . , p) is necessary for n X ∈ S+ (E, ?). For the graph G(N, E) shown in Figure 2.2, the maximal cliques are C1 = {1, 6}, C2 = {2, 6}, C3 = {3, 4}, C4 = {3, 6}, C5 = {4, 5} and C6 = {5, 6}. Hence, the necessary condition for X ∈ S6 (E, ?) to be completed to a positive semidefinite matrix is that its 6 principal submatrices X(Ck ) (k = 1, 2, . . . , 6) are positive semidefinite. Although this condition is not sufficient in general, it is a sufficient condition for X ∈ Sn+ (E, ?) when G(N, E) is chordal. As stated in 2.1.4, in this case, the number of the maximal cliques is bounded by the number of nodes of G(N, E), i.e., p ≤ n. In general we have the following result. Lemma 2.2 Let Ck (k = 1, 2, . . . , p) be the maximal cliques of a chordal graph G(N, E). Suppose that k X ∈ Sn (E, ?). Then X ∈ Sn+ (E, ?) if and only if X(Ck ) ∈ SC + (k = 1, 2, . . . , p). Proof: C.f. [28]. Since the graph G(N, E) in Figure 2.2 is not a chordal graph, we can not apply Lemma 2.2 to determine whether X ∈ S6 (E, ?) of the form (2.30) belongs to S6+ (E, ?). In such a case, we need to introduce a chordal extension of the graph G(N, E) to use the lemma effectively. Figure 2.3 shows two chordal extensions. If we choose the left graph as a chordal extension G(N, E) of G(N, E), the maximal cliques are C1 = {3, 4, 6}, C2 = {4, 5, 6}, C3 = {1, 6} and C4 = {2, 6}, consequently, X ∈ S6+ (E, ?) is characterized by k X(Ck ) ∈ SC + (k = 1, 2, 3, 4). 2 3 4 1 6 5 Figure 2.2: A graph G(N, E) with N = {1, 2, 3, 4, 5, 6} 2 3 4 2 3 4 1 6 5 1 6 5 (a) (b) Figure 2.3: Chordal extensions of the graph G(N, E) given in Figure 2.2. (a) The maximal cliques are C1 = {3, 4, 6}, C2 = {4, 5, 6}, C3 = {1, 6} and C4 = {2, 6}. (b) The maximal cliques are C1 = {3, 4, 5}, C2 = {3, 5, 6}, C3 = {1, 6} and C4 = {2, 6}. Remark 2.2 To compute the positive definite matrix completion of a matrix, we can recursively apply Lemma 2.6 of [21]. A numerical example is shown on page 657 of [21]. 2.2.3 Exploiting the domain-space sparsity In this section, we consider a general nonlinear optimization problem involving a matrix variable X ∈ Sn : minimize f0 (x, X) subject to f (x, X) ∈ Ω and X ∈ Sn+ , (2.32) 34 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION where f0 : Rs × Sn → R, f : Rs × Sn → Rm and Ω ⊂ Rm . Let E denote the set of distinct row and column index pairs (i, j) such that a value of Xij is necessary to evaluate f0 (x, X) and/or f (x, X). More 1 2 precisely, for Xkl = Xij (k, l) 6= (i, j), f0 (x, X 1 ) 6= f0 (x, X 2 ) and/or f (x, X 1 ) 6= f (x, X 2 ) hold for some x ∈ Rs , X 1 ∈ Sn and X 2 ∈ Sn . Consider a graph G(N, E). We call E the d-space sparsity pattern and G(N, E) the d-space sparsity pattern graph. If G(N, E) is an extension of G(N, E), then we may replace the condition X ∈ Sn+ by X ∈ Sn+ (E, ?). To apply Lemma 2.2, we choose a chordal extension G(N, E) of G(N, E). Let C1 , C2 , . . . , Cp be its maximal cliques. Then we may regard f0 and f as functions in x ∈ Rs and X(Ck ) (k = 1, 2, . . . , p), i.e., there are functions f˜0 and f˜ in the variables x and X(Ck ) (k = 1, 2, . . . , p) such that f0 (x, X) = f˜0 (x, X(C1 ), X(C2 ), . . . , X(Cp )) for every (x, X) ∈ Rs × Sn , (2.33) f (x, X) = f˜(x, X(C1 ), X(C2 ), . . . , X(Cp )) for every (x, X) ∈ Rs × Sn . Therefore, the problem (2.32) is equivalent to minimize subject to f˜0 (x, X(C1 ), X(C2 ), . . . , X(Cp )) f˜(x, X(C1 ), X(C2 ), . . . , X(Cp )) ∈ Ω and k X(Ck ) ∈ SC + (k = 1, 2, . . . , p). (2.34) As an illustrative example, we consider the problem whose d-space sparsity pattern graph G(N, E) is shown in Figure 2.2: X Xij minimize − (i,j)∈E, i<j (2.35) 6 X (Xii − αi )2 ≤ 6, X ∈ S6+ , subject to i=1 where αi > 0 (i = 1, 2, . . . , 6). As a chordal extension, we choose the graph G(N, E) in (a) of Figure 2.3. Then, the problem (2.34) becomes 4 X ˜ minimize f0k (X(Ck )) k=1 (2.36) 4 X Ck ˜ subject to fk (X(Ck )) ≤ 6, X(Ck ) ∈ S+ (k = 1, 2, 3, 4), k=1 where f˜01 (X(C1 )) f˜03 (X(C3 )) f˜1 (X(C1 )) f˜2 (X(C2 )) f˜4 (X(C4 )) = = = = = −X34 − X36 , f˜02 (X(C2 )) = −X45 − X56 , −X16 , f˜04 (X(C4 )) = −X26 , (X33 − α3 )2 + (X44 − α4 )2 + (X66 − α6 )2 , (X55 − α5 )2 , f˜3 (X(C3 )) = (X11 − α1 )2 , (X22 − α2 )2 . (2.37) k The positive semidefinite condition X(Ck ) ∈ SC + (k = 1, 2, . . . , p) in the problem (2.34) is not an ordinary positive semidefinite condition in the sense that overlapping variables Xij ((i, j) ∈ Ck ∩ Cl ) exist Cl k in two distinct positive semidefinite constraints X(Ck ) ∈ SC + and X(Cl ) ∈ S+ if Ck ∩ Cl 6= ∅. We describe two methods to transform the condition into an ordinary positive semidefinite condition. The first one was given in the papers [21, 68] where a d-space conversion method was proposed, and the second one was originally used for the sparse SDP relaxation of polynomial optimization problems [102, 103] and also in the paper [43] where a d-space conversion method was applied to an SDP relaxation of a sensor network localization problem. We call the first one the d-space conversion method using clique trees and the second one the d-space conversion method using basis representation. 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 35 The d-space conversion method using clique trees We can replace X(Ck ) (k = 1, 2, . . . , p) by p independent matrix variables X k (k = 1, 2, . . . , p) if we add all k l equality constraints Xij = Xij for every (i, j) ∈ Ck ∩ Cl with i ≤ j and every pair of Ck and Cl such that Ck ∩ Cl 6= ∅. For the chordal graph G(N, E) given in (a) of Figure 2.3, those equalities turn out to be the 8 equalities k l 1 2 1 2 X66 − X66 = 0 (1 ≤ k < l ≤ 4), X44 = X44 , X46 = X46 These equalities are linearly dependent, and we can choose a maximal number of linearly independent equalities that are equivalent to the original equalities. For example, either of a set of 5 equalities 1 2 1 2 1 2 1 3 1 4 X44 − X44 = 0, X46 − X46 = 0, X66 − X66 = 0, X66 − X66 = 0, X66 − X66 = 0. (2.38) and a set of 5 equalities 1 2 1 2 1 2 2 3 3 4 X44 − X44 = 0, X46 − X46 = 0, X66 − X66 = 0, X66 − X66 = 0, X66 − X66 =0 (2.39) is equivalent to the set of 8 equalities above. In general, we use a clique tree T (K, E) with K = {C1 , C2 , . . . , Cp } and E ⊆ K × K to consistently choose a set of maximal number of linearly independent equalities. Here T (K, E) is called a clique tree if it satisfies the clique-intersection property, that is, for each pair of nodes Ck ∈ K and Cl ∈ K, the set Ck ∩ Cl is contained in every node on the (unique) path connecting Ck and Cl . See [4] for basic properties on clique trees. We fix one clique for a root node of the tree T (K, E), say C1 . For simplicity, we assume that the nodes C2 , . . . , Cp are indexed so that if a sequence of nodes C1 , Cl2 , . . . , Clk forms a path from the root node C1 to a leaf node Clk , then 1 < l2 < · · · < lk , and each edge is directed from the node with a smaller index to the other node with a larger index. Thus, the clique tree T (K, E) is directed from the root node C1 to its leaf nodes. Each edge (Ck , Cl ) of the clique tree T (K, E) induces a set of equalities k l Xij − Xij = 0 ((i, j) ∈ J(Ck ∩ Cl )), or equivalently, Eij • X k − Eij • X l = 0 ((i, j) ∈ J(Ck ∩ Cl )), where J(C) = {(i, j) ∈ C × C : i ≤ j} for every C ⊆ N . We add equalities of the form above for all (Ck , Cl ) ∈ E when we replace X(Ck ) (k = 1, 2, . . . , p) by p independent matrix variables X k (k = 1, 2, . . . , p). We thus obtain a problem minimize f˜0 (x, X 1 , X 2 , . . . , X p ) subject to f˜(x, X 1 , X 2 , . . . , X p ) ∈ Ω, (2.40) k l Eij • X − Eij • X = 0 ((i, j, k, l) ∈ Λ), C X k ∈ S+k (k = 1, 2, . . . , p), where Λ = {(g, h, k, l) : (g, h) ∈ J(Ck ∩ Cl ), (Ck , Cl ) ∈ E}. (2.41) This is equivalent to the problem (2.34). See Section 4 of [68] for more details. Now we illustrate the conversion process above by the simple example (2.35). Figure 2.4 shows two clique trees for the graph given in (a) of Figure 2.3. The left clique tree in Figure 2.4 leads to the 5 equalities in (2.38), while the right clique tree in Figure 2.4 induces the 5 equalities in (2.39). In both cases, the problem (2.40) has the following form minimize subject to 4 X k=1 4 X k=1 fˆ0k (X k ) fˆk (X k ) ≤ 6, the 5 equalities in (2.38) or (2.39), k X k ∈ SC + (k = 1, 2, 3, 4), 36 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION C ={3,4,6} 1 C ={2,6} 4 C ={1,6} 3 C ={4,5,6} 2 C ={2,6} 4 C ={3,4,6} 1 C ={1,6} 3 C ={4,5,6} 2 Figure 2.4: Two clique trees with K = {C1 = {1, 2}, C2 = {1, 4}, C3 = {1, 6}, C4 = {1, 3, 5}}. where 1 1 2 2 fˆ01 (X 1 ) = −X34 − X36 , fˆ02 (X 2 ) = −X45 − X56 , 4 4 3 3 ˆ ˆ f03 (X ) = −X16 , f04 (X ) = −X26 , fˆ1 (X 1 ) = (X 1 − α3 )2 + (X 1 − α4 ) + (X 1 − α6 )2 , 44 33 66 2 3 4 fˆ2 (X 2 ) = (X55 − α5 )2 , fˆ3 (X 3 ) = (X11 − α1 )2 , fˆ4 (X 4 ) = (X22 − α2 )2 . Remark 2.3 The d-space conversion method using clique trees can be implemented in many different ways. The fact that the chordal extension G(N, E) of G(N, E) is not unique offers flexibility in constructing an optimization problem of the form (2.40). More precisely, a choice of chordal extension G(N, E) of G(N, E) decides how “small” and “sparse” an optimization problem of the form (2.40) is, which is an important issue for solving the problem more efficiently. For the size of the problem (2.40), we need to consider the sizes of the matrix variables X k (k = 1, 2, . . . , p) and the number of equalities in (2.40). Note that the sizes of the matrix variables X k (k = 1, 2, . . . , p) are determined by the sizes of the maximal cliques Ck (k = 1, 2, . . . , p). This indicates that a chordal extension G(N, E) with smaller maximal cliques Ck (k = 1, 2, . . . , p) may be better theoretically. (In computation, however, this is not necessarily true because of overhead of processing too many small positive semidefinite matrix variables.) The number of equalities in (2.40) or the cardinality of Λ is also determined by the chordal extension G(N, E) of G(N, E). Choosing a chordal extension G(N, E) with smaller maximal cliques increases the number of equalities. Balancing these two contradicting targets, decreasing the sizes of the matrix variables and decreasing the number of equalities was studied in the paper [68] by combining some adjacent cliques along the clique tree T (K, E). See Section 4 of [68] for more details. In addition to the choice of a chordal extension G(N, E) of G(N, E), the representation of the functions and the choice of a clique tree add flexibilities in the construction of the problem (2.40). That is, the representation of the functions f0 : Rs × Sn → R and f : Rs × Sn → Rm in the vector variable x and the matrix variables X(Ck ) (k = 1, 2, . . . , p) as in (2.33); for example, we could move the term (X66 − α6 )2 from f˜1 (x, X(C1 )) to either of f˜k (x, X(Ck )) (k = 2, 3, 4). These choices of the functions f0 , f and a clique tree affect the sparse structure of the resulting problem (2.40), which is also important for efficient computation. The domain-space conversion method using basis representation Define J¯ = p [ J(Ck ), k=1 ¯ = (Xij : (i, j) ∈ J) ¯ f¯0 (x, (Xij : (i, j) ∈ J)) = ¯ ¯ f (x, (Xij : (i, j) ∈ J)) = ¯ the vector variable consisting of Xij ((i, j) ∈ J), s n f0 (x, X) for every (x, X) ∈ R × S , f (x, X) for every (x, X) ∈ Rs × Sn . 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 37 We represent each X(Ck ) in terms of a linear combination of the basis Eij ((i, j) ∈ J(Ck )) of the space SCk as in (2.31) with C = Ck (k = 1, 2, . . . , p). Substituting this basis representation into the problem (2.34), we obtain ¯ minimize f¯0 (x, (Xij : (i, j) ∈ J)) subject to f¯(x,X (Xij : (i, j) ∈ J¯) ∈ Ω, (2.42) Ck Eij Xij ∈ S+ (k = 1, 2, . . . , p). (i,j)∈J(Ck ) We observe that the illustrative example (2.35) is converted into the problem X Xij minimize − (i,j)∈E, i<j 6 X 2 (Xii − αi ) ≤ 6, subject to i=1X Ck Eij Xij ∈ S+ (k = 1, 2, 3, 4). (2.43) (i,j)∈J(Ck ) Remark 2.4 Compared to the d-space conversion method using clique trees, the d-space conversion method using basis representation described above provides limited flexibilities. To make the size of the problem (2.42) smaller, we need to select a chordal extension G(N, E) of G(N, E) with smaller maximal cliques Ck (k = 1, 2, . . . , p). As a result, the sizes of semidefinite constraints become smaller. As we mentioned in Remark 2.3, however, too many smaller positive semidefinite matrix variables may yield heavy overhead in computation. 2.2.4 Duality in positive semidefinite matrix completion In order to present the r-space conversion methods in the next section, we need to derive some results, which can be understood as a dual approach to the positive semidefinite matrix completion approach from the 2.2.2. Throughout this section, we assume that G(N, E) denotes a chordal graph. In Lemma 2.2, we have described a necessary and sufficient condition for a partial symmetric matrix X ∈ Sn (E, ?) to be completed to a positive semidefinite symmetric matrix. Let Sn (E, 0) = Sn+ (E, 0) = {A ∈ Sn : Aij = 0 if (i, j) 6∈ E • }, {A ∈ Sn (E, 0) : A < 0}. In this section, we derive a necessary and sufficient condition for a symmetric matrix A ∈ Sn (E, 0) to be positive semidefinite, i.e., A ∈ Sn+ (E, 0). This condition is used for the range-space conversion methods in Section 5. We note that these two issues have primal-dual relationship: X A ∈ Sn+ (E, 0) if and only if Aij Xij ≥ 0 for every X ∈ Sn+ (E, ?). (2.44) (i,j)∈E • Suppose A ∈ Sn (E, 0). Let C1 , C2 , . . . , Cp be the maximal cliques of G(N, E). Then, we can consistently p X ek ∈ SCk (k = 1, 2, . . . , p) such that A = ek . We know that A is positive decompose A ∈ Sn (E, 0) into A A k=1 semidefinite if and only if A • X ≥ 0 for every X ∈ Sn+ . This relation and Lemma 2.2 are used in the following. Since A ∈ Sn (E, 0), this condition can be relaxed to the condition (2.44). Therefore, A is positive semidefinite if and only if the following SDP has the optimal value 0. " p # X X ek A minimize Xij subject to X ∈ Sn+ (E, ?). (2.45) (i,j)∈E • k=1 ij 38 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION We can rewrite the objective function as X (i,j)∈E • " p X k=1 ek A # Xij = p X k=1 ij = (i,j)∈E • p X k=1 X ek Xij A ij ek • X(Ck ) for every X ∈ Sn (E, ?). A + ek ∈ SCk (k = 1, 2, . . . , p). Applying Lemma 2.2 to the constraint Note that the second equality follows from A n X ∈ S+ (E, ?) of the SDP (2.45), we obtain an SDP minimize p X ek • X(Ck ) A k=1 k subject to X(Ck ) ∈ SC + (k = 1, 2, . . . , p), (2.46) which is equivalent to the SDP (2.45). The SDP (2.46) involves multiple positive semidefinite matrix variables with overlapping elements. We have described two methods to convert such multiple matrix variables into independent ones with no overlapping elements in Sections 2.2.3 and 2.2.3, respectively. We apply the method given in Section 2.2.3 to the SDP (2.46). Let T (K, E) be a clique tree with K = {C1 , C2 , . . . , Cp } and E ⊆ K × K. Then, we obtain an SDP minimize p X k=1 subject to ek • X k A Eij • X k − Eij • X l = 0 ((i, j, k, l) ∈ Λ), k X k ∈ SC + (k = 1, 2, . . . , p), (2.47) which is equivalent to the SDP (2.46). Here Λ is given in (2.41). Theorem 2.15 A ∈ Sn (E, 0) is positive semidefinite if and only if the system of LMIs ek − L e k (z) < 0 (k = 1, 2, . . . , p). A (2.48) has a solution v = (vghkl : (g, h, k, l) ∈ Λ). Here z = (zghkl : (g, h, k, l) ∈ Λ) denotes a vector variable consisting of zghkl ((g, h, k, l) ∈ Λ), and X X e k (z) = − L Eij zijhk + Eij zijkl (i, j, h); (i, j, h, k) ∈ Λ (i, j, l); (i, j, k, l) ∈ Λ for every z = (zijkl : (i, j, k, l) ∈ Λ) (k = 1, 2, . . . , p). (2.49) Proof: In the previous discussions, we have shown that A ∈ Sn (E, 0) is positive semidefinite if and only if the SDP (2.47) has the optimal value 0. The dual of the SDP (2.47) is maximize 0 subject to (2.48). (2.50) The primal SDP (2.47) attains the objective value 0 at a trivial feasible solution (X1 , X2 , . . . , Xp ) = (0, 0, . . . , 0). If the dual SDP (2.50) is feasible or the system of LMIs (2.48) has a solution, then the primal SDP (2.47) has the optimal value 0 by the week duality theorem. Thus we have shown the “if part” of the theorem. Now suppose that the primal SDP (2.47) has the optimal value 0. The primal SDP (2.47) has an interior-feasible solution; for example, take X k to be the #Ck ×#Ck identity matrix in SCk (k = 1, 2, . . . , p). By the strong duality theorem (Theorem 4.2.1 of [69]), the optimal value of the dual SDP (2.50) is zero, which implies that (2.50) is feasible. As a corollary, we obtain the following (Theorem 2.3 of [1]). 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 39 k Theorem 2.16 A ∈ Sn (E, 0) is positive semidefinite if and only if there exist Y k ∈ SC + (k = 1, 2, . . . , p) p X Y k. which decompose A as A = k=1 Proof: Since the “if part” is straightforward, we prove the “only if” part. Assume that A is positive ek − L e k (z̃) (k = 1, 2, . . . , p). semidefinite. By Theorem 2.15, the LMI (2.48) has a solution z̃. Let Y k = A p X k e k (z̃) = 0 by construction, we obtain the desired result. L Then Y k ∈ SC + (k = 1, 2, . . . , p). Since k=1 Conversely, Theorem 2.15 can be derived from Theorem 2.16. In the paper [1], Theorem 2.16 was proved by Theorem 7 of Grone et al. [28] (Lemma 2.2 in this thesis). We conclude this section by applying Theorem 2.15 to the case of the chordal graph G(N, E) given in (a) of Figure 2.3. The maximal cliques are C1 = {3, 4, 6}, C2 = {4, 5, 6}, C3 = {1, 6} and C4 = {2, 6}, so that A ∈ S6 (E, 0) is decomposed into 4 matrices e1 A e2 A e3 A e4 A or, e1 A e3 A = = = = = = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 A33 A43 0 A63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 A54 0 0 A11 0 0 0 0 A61 0 0 A34 A44 0 A64 0 0 0 0 0 A36 0 A46 0 0 0 A66 0 0 0 A45 A55 A65 ∈ S{3,4,6} , 0 0 0 ∈ S{4,5,6} , 0 A56 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 A16 0 0 0 0 0 0 0 0 0 0 0 0 0 A22 0 0 0 0 0 0 0 A62 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 A26 0 0 0 0 0 0 0 0 ∈ S{1,6} , ∈ S{2,6} , A33 A34 A36 0 A45 0 e2 = A54 A55 A56 ∈ S{4,5,6} , A43 A44 A46 ∈ S{3,4,6} , A 0 A65 0 A63 A64 A66 A A A11 A16 22 26 e4 = ∈ S{2,6} ∈ S{1,6} , A A62 0 A61 0 (2.51) in the compact representation. We note that this decomposition is not unique. For example, we can move e1 to any other A ek . We showed two clique trees with K = {C1 , C2 , C3 , C4 } the (6, 6) element A66 from A in Figure 2.4. For the left clique tree, we have Λ = {(4, 4, 1, 2), (4, 6, 1, 2), (6, 6, 1, 2), (6, 6, 1, 3), (6, 6, 1, 4))}. 40 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Thus, the system of LMIs (2.48) becomes A33 A34 A43 A44 − z4412 A63 A64 − z4412 z4412 A45 z4612 A54 A55 A56 z4612 A65 z6612 A11 A16 < 0, A61 z6613 A36 < 0, A46 − z4612 A − z − z − z 6612 6613 6614 66 < 0, A22 A62 A26 z6614 < 0. For the right clique tree, we have Λ = {(4, 4, 1, 2), (4, 6, 1, 2), (6, 6, 1, 2), (6, 6, 2, 3), (6, 6, 3, 4)} and A33 A34 A36 A43 A44 − z4412 A46 − z4612 < 0, A A − z A − z 63 64 4412 66 6612 z4412 A45 z4612 A54 A55 A56 < 0, z A z − z 4612 65 6612 6623 A11 A16 A22 A26 < 0, < 0. A61 z6623 − z6634 A62 z6634 2.2.5 (2.52) (2.53) Exploiting the range-space sparsity In this section, we present two range-space conversion methods, the r-space conversion method using clique trees based on Theorem 2.15 and the r-space conversion method using matrix decomposition based on Theorem 2.16. The range-space conversion method using clique trees Let F = {(i, j) ∈ N × N : Mij (y) 6= 0 for some y ∈ Rs , i 6= j}. We call F the r-space sparsity pattern and G(N, F ) the r-space sparsity pattern graph of the mapping M : Rs → Sn . Apparently, M (y) ∈ Sn (F, 0) for every y ∈ Rs , but the graph G(N, F ) may not be chordal. Let G(N, E) be a chordal extension of G(N, F ). Then M (y) ∈ Sn (E, 0) for every y ∈ Rs . (2.54) Let C1 , C2 , . . . , Cp be the maximal cliques of G(N, E). fk (k = 1, 2, . . . , p) to decompose the mapping M : Rs → To apply Theorem 2.15, we choose mappings M n S such that M (y) = p X k=1 fk (y) for every y ∈ Rs , M fk : Rs → SCk (k = 1, 2, . . . , p). M (2.55) Let T (K, E) be a clique tree where K = {C1 , C2 , . . . , Cp } and E ⊂ K × K. By Theorem 2.15, y is a solution of (2.26) if and only if it is a solution of fk (y) − L e k (z) < 0 (k = 1, 2, . . . , p) M (2.56) e k in (2.49). for some z = (zghkl : (g, h, k, l) ∈ Λ), where Λ is given in (2.41) and L We may regard the r-space conversion method using clique trees described above as a dual of the d-space conversion method using clique trees applied to the SDP minimize M (y) • X subject to X < 0, (2.57) 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 41 where X ∈ Sn denotes a variable matrix and y ∈ Rs a fixed vector. We know that M (y) < 0 if and only if the optimal value of the SDP (2.57) is zero, so that (2.57) serves as a dual of the matrix inequality M (y) < 0. Each element zijkl of the vector variable z corresponds to a dual variable of the equality constraint Eij • X k − Eij • X l = 0 in the problem (2.40), while each matrix variable X k ∈ SCk in the fk (y) − L e k (z) < 0. problem (2.40) corresponds to a dual matrix variable of the kth matrix inequality M Remark 2.5 On the flexibilities in implementing the r-space conversion method using clique trees, the comments in Remark 2.3 are valid if we replace the sizes of the matrix variable X k by the size of the fk : Rs → SCk and the number of equalities by the number of elements zijkl of the vector variable mapping M z. The correlative sparsity of (2.56) depends on the choice of the clique tree and the decomposition (2.55). As an example, we consider the case where M is tridiagonal, i.e., the (i, j)th element Mij of M is zero if |i − j| ≥ 2, to illustrate the range space conversion of the matrix inequality (2.26) into the system of matrix inequalities (2.56). By letting E = {(i, j) : |i − j| = 1}, we have a simple chordal graph G(N, E) with no cycle satisfying (2.54), its maximal cliques Ck = {k, k + 1} (k = 1, 2, . . . , n − 1), and a clique tree T (K, E) with K = {C1 , C2 , . . . , Cn−1 } and E = {(Ck , Ck+1 ) ∈ K × K : k = 1, 2, . . . , n − 2}. For every y ∈ Rs , let fk M (y) = Mkk (y) Mk,k+1 (y) ∈ SCk Mk+1,k (y) 0 Mn−1,n−1 (y) Mn−1,n (y) ∈ SCk Mn,n−1 (y) Mnn (y) if 1 ≤ k ≤ n − 2, if k = n − 1. fk : Rs → SCk (k = 1, 2, . . . , n − 1) as in (2.55) with Then, we can decompose M : Rs → Sn (E, 0) into M p = n − 1. We also see that Λ = e k (z) = L {(k + 1, k + 1, k, k + 1) : k = 1, 2, . . . , n − 2}, E22 z2212 ∈ SC1 −Ek,k zk,k,k−1,k + Ek+1,k+1 zk+1,k+1,k,k+1 ∈ SCk −En−1,n−1 zn−1,n−1,n−2,n−1 ∈ SCn−1 if k = 1, if k = 2, 3, . . . , n − 2, if k = n − 1, Thus the resulting system of matrix inequalities (2.56) is M11 (y) M12 (y) < 0, M21 (y) −z2212 Mkk (y) + zk,k,k−1,k Mk,k+1 (y) < 0 (k = 2, 3, . . . , n − 2), −zk+1,k+1,k,k+1 Mk+1,k (y) Mn−1,n−1 (y) + zn−1,n−1,n−2,n−1 Mn−1,n (y) < 0. Mn,n−1 (y) Mnn (y) The range-space conversion method using matrix decomposition By Theorem 2.16, we obtain that y ∈ Rs is a solution of the matrix inequality (2.26) if and only if there exist Y k ∈ SCk (k = 1, 2, . . . , p) such that p X k=1 k Y k = M (y) and Y k ∈ SC + (k = 1, 2, . . . , p). Let J = ∪pk=1 J(Ck ) and Γ(i, j) = {k : i ∈ Ck , j ∈ Ck } ((i, j) ∈ J). Then we can rewrite the condition above as X k Eij • Y k − Eij • M (y) = 0 ((i, j) ∈ J) and Y k ∈ SC (2.58) + (k = 1, 2, . . . , p). k∈Γ(i,j) 42 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION We may regard the r-space conversion method using matrix decomposition as a dual of the d-space conversion method using basis representation applied to the SDP (2.57) with a fixed y ∈ Rs . Each variable Xij ((i, j) ∈ J) in the problem (2.42) corresponds to a dual real variable of the (i, j)th equality constraint of the problem (2.58), while X each matrix variable Y k in the problem (2.58) corresponds to a dual matrix k Eij Xij ∈ SC variable of the constraint + . (i,j)∈J(Ck ) Remark 2.6 On the flexibilities in implementing the r-space conversion method using matrix decomposition, the comments in Remsark 2.4 are valid if we replace the sizes of the semidefinite constraints by the sizes of the matrix variables Y k (k = 1, 2, . . . , p). We illustrate the r-space conversion method using matrix decomposition with the same example where M is tridiagonal as in Section 2.2.3 In this case, we see that p = n − 1, = {k, k + 1} (k = 1, 2, . . . , n − 1), = {(k, k), (k, k + 1), (k + 1, k + 1)} (k = 1, 2, . . . , n − 1), [ J = {(k, k) : k = 1, 2, . . . , n} {(k, k + 1) : k = 1, 2, . . . , n − 1}, {1} if i = j = 1, {k} if i = k, j = k + 1 and 1 ≤ k ≤ n − 1, Γ(i, j) = {k − 1, k} if i = j = k and 2 ≤ k ≤ n − 1, {n − 1} if i = j = n. Ck J(Ck ) Hence, the matrix inequality (2.26) with the tridiagonal M : Rs → Sn is converted into E11 • Y 1 − E11 • M (y) = 0, Ek,k+1 • Y k − Ek,k+1 M (y) = 0 (k = 1, 2, . . . , n − 1), Ekk • Y k−1 + Ekk • Y k − Ekk • M (y) = 0 (k = 2, . . . , n − 1), Enn • Y n−1 − Enn • M (y) = 0, k k Y Y kk k,k+1 k ∈ SC (k = 1, 2, . . . , n − 1). Yk = k k + Yk+1,k Yk+1,k+1 2.2.6 Enhancing the correlative sparsity When we are concerned with the SDP relaxation of polynomial SDPs (including ordinary polynomial optimization problems) and linear SDPs, another type of sparsity called the correlative sparsity plays an important role in solving the SDPs efficiently. The correlative sparsity was dealt with extensively in the paper [45], and was introduced in 2.1.4 . It is known that the sparse SDP relaxation [51, 102] for a correlatively sparse polynomial optimization problem leads to an SDP that can maintain the sparsity for primal-dual interior-point methods. See Section 6 of [45]. In this section, we focus on how the d-space and r-space conversion methods enhance the correlative sparsity. We consider a polynomial SDP of the form maximize f0 (y) subject to k Fk (y) ∈ Sm + (k = 1, . . . , p). (2.59) Here f0 ∈ R[y], Fk a mapping from Rn into Smk with all polynomial components in y ∈ Rn . For simplicity, we assume that f0 is a linear function of the form f0 (y) = bT y for some b ∈ Rn . In this case, with the definition from 2.1.4 the correlative sparsity pattern graph is given by the graph G(N, E) with the node set N = {1, 2, . . . , n} and the edge set i 6= j, both values yi and yj are necessary E = (i, j) ∈ N × N : . to evaluate the value of Fk (y) for some k When a chordal extension G(N, E) of the correlative sparsity pattern graph G(N, E) is sparse or all the maximal cliques of G(N, E) are small-sized, we can effectively apply the sparse SDP relaxation [51, 102] to 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 43 the polynomial SDP (2.59). As a result, we have a linear SDP satisfying a correlative sparsity characterized by the same chordal graph structure as G(N, E). More details can be found in Section 6 of [45]. Even when the correlative sparsity pattern graph G(N, E) or its chordal extension G(N, E) is not sparse, the polynomial SDP may have “a hidden correlative sparsity” that can be recognized by applying the d-space and/or rspace conversion methods to the problem to decompose a large size matrix variable (and/or inequality) into multiple smaller size matrix variables (and/or inequalities). To illustrate this, let us consider a polynomial SDP of the form minimize bT y subject to F (y) ∈ Sn+ , where F denotes a mapping from Rn into Sn defined by 1 − y14 0 0 4 0 1 − y 0 2 .. . 0 0 F (y) = ... ... ... 0 0 0 y1 y2 y2 y3 y3 y4 ... ... .. . 0 0 y1 y2 y2 y3 0 y3 y4 ... 4 1 − yn−1 . . . yn−1 yn ... yn−1 yn 1 − yn4 . This polynomial SDP is not correlatively sparse at all (i.e., G(N, E) becomes a complete graph) because all variables y1 , y2 , . . . , yn are involved in the single matrix inequality F (y) ∈ Sn+ . Hence, the sparse SDP relaxation (2.18) is not effective for this problem. Applying the r-space conversion method using clique trees to the polynomial SDP under consideration, we have a polynomial SDP minimize bT y 1 − y14 y1 y2 < 0, subject to y y z 1 2 1 4 (2.60) 1 − yi yi yi+1 < 0 (i = 2, 3, . . . , n − 2), y y −z + z i i+1 i−1 i 4 1 − yn−1 yn−1 yn < 0, 4 yn−1 yn 1 − yn − zn−2 which is equivalent to the original polynomial SDP. The resulting polynomial SDP now satisfies the correlative sparsity as shown in Figure 2.5. Thus the sparse SDP relaxation (2.18) is efficient for solving (2.60). The correlative sparsity is important in linear SDPs, too. We have seen such a case in Section 2.2.1. We can rewrite the SDP (2.28) as n−1 X 0 0 0 Aii Xii + 2Ai,i+1 Xi,i+1 − Ann Xnn maximize − i=1 n−1 n X X (2.61) Ein Xi,i+1 < 0, Eii Xii + subject to I − i=1 i=1 X Eij Xij < 0, 1≤i≤j≤n where I denotes the n×n identity matrix. Since the coefficient matrices of all real variables Xij (1 ≤ i ≤ j ≤ n) are nonzero in the last constraint, the correlative sparsity pattern graph G(N, E) forms a complete graph. Applying the d-space conversion method using basis representation and the r-space conversion method using clique trees to the original SDP (2.28), we have reduced it to the SDP (2.29) in Section 2.1. We rewrite the constraints of the SDP (2.29) as an ordinary LMI form: maximize bT y s X (2.62) k k subject to A0 − Ah yh < 0 (k = 1, 2, . . . , p). h=1 44 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION 0 0 5 5 10 10 15 15 20 20 25 25 30 30 35 35 0 5 10 15 20 nz = 218 25 30 35 0 5 10 15 20 nz = 128 25 30 35 Figure 2.5: The correlative sparsity pattern of the polynomial SDP (2.60) with n = 20, and its Cholesky factor with a symmetric minimum degree ordering of its rows and columns. Here p = 2n − 2, s = 3n − 3, each Akh is 2 × 2 matrix (k = 1, 2, . . . , p, h = 0, 1, . . . , 3n − 3), b ∈ R3n−3 , y ∈ R3n−3 , and each element yh of y corresponds to some Xij or some zi . Comparing the SDP (2.61) with the SDP (2.62), we notice that the number of variables is reduced from n(n + 1)/2 to 3n − 3, and the maximum size of the matrix inequality is reduced from n to 2. Furthermore, the correlative sparsity pattern graph becomes sparse. See Figure 2.6. 0 0 5 50 100 10 150 15 200 20 250 25 0 5 10 15 nz = 161 20 25 0 50 100 150 nz = 1871 200 250 Figure 2.6: The correlative sparsity pattern of the SDP (2.62) induced from (2.29) with n = 10 and n = 100, and its Cholesky factor with a symmetric minimum degree ordering of its rows and columns. Now we consider an SDP of the form (2.62) in general. The edge set E of the correlative sparsity pattern graph G(N, E) is written as E = (g, h) ∈ N × N : g 6= h, Akg 6= 0 and Akh 6= 0 for some k , where N = {1, 2, . . . , s}. It is known that the graph G(N, E) characterizes the sparsity pattern of the Schur 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 45 complement matrix of the SDP (2.62). More precisely, if R denotes the s × s sparsity pattern of the Schur complement matrix, then Rgh = 0 if (g, h) 6∈ E • . Furthermore, if the graph G(N, E) is chordal, then there exists a perfect elimination ordering, a simultaneous row and column ordering of the Schur complement matrix that allows a Cholesky factorization with no fill-in. For the SDP induced from (2.29), we have seen the correlative sparsity pattern with a symmetric minimum degree ordering of its rows and columns in Figure 2.6, which coincides with the sparsity pattern of the Schur complement matrix whose symbolic Cholesky factorization is shown in Figure 2.1. Remark 2.7 As mentioned in Remark 2.5, the application of r-space conversion method using clique trees to reduce the SDP (2.28) to the SDP (2.29) can be implemented in many different ways. In practice, it should be implemented to have a better correlative sparsity in the resulting problem. For example, we can reduce the SDP (2.28) to n−1 X 0 0 0 Aii Xii + 2Ai,i+1 Xi,i+1 + Ann Xnn minimize i=1 1 0 X11 −X12 subject to < 0, − 0 0 −X −z 21 1 1 0 Xii −Xi,i+1 (2.63) < 0 (i = 2, 3, . . . , n − 2), − −zi 0 0 −Xi+1,i Xn−1,n−1 −Xn−1,n 1 0 Pn−2 < 0, − 0 1 z −X X + i n,n−1 n,n i=1 0 0 −Xii −Xi,i+1 − < 0 (i = 1, 2, . . . , n − 1), 0 0 −Xi+1,i −Xi+1,i+1 which is different from the SDP (2.29). This is obtained by choosing a different clique tree in the rspace conversion method using clique trees for the SDP (2.28). In this case, all auxiliary variables zi (i = 1, 2, . . . , n−2) are contained in a single matrix inequality. This implies that the corresponding correlative sparsity pattern graph G(N, E) involves a clique of size n − 2. See Figure 2.7. Thus the correlative sparsity becomes worse than the previous conversion. Among various ways of implementing the d- and r-space conversion methods, determining which one is effective for a better correlative sparsity will be a subject which requires further study. 0 0 5 50 100 10 150 15 200 20 250 25 0 5 10 15 nz = 217 20 25 0 50 100 150 nz = 11377 200 250 Figure 2.7: The correlative sparsity pattern of the SDP (2.62) induced from (2.63) with n = 10 and n = 100 where the rows and columns are simultaneously reordered by the Matlab function symamd (a symmetric minimum degree ordering). 46 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION 2.2.7 Examples of d- and r-space sparsity in quadratic SDP We present how to take advantage of the d- and r-space conversion methods introduced in the previous sections for the class of quadratic SDP, and demonstrate the effectiveness of these methods on examples of quadratic SDP. In the following we first consider a quadratic SDP of the form minimize s X ci xi subject to M (x) < 0, (2.64) i=1 where ci ∈ [0, 1] (i = 1, 2, . . . , s), M : Rs → Sn , and each non-zero element Mij of the mapping M : Rs → Sn is a polynomial in x = (x1 , x2 , . . . , xs ) ∈ Rs with degree at most 2. We apply d- and r-space conversion methods to (2.64), get a new quadratic SDP with smaller size matrix inequality constraints and relax this quadratic SDP to obtain a linear SDP which can be solved by standard SDP solvers. The test problems of quadratic SDP we consider for numerical experiments are three max-cut problems, a Lovas theta problem, a box-constrained quadratic problem from SDPLIB [9], a sensor network localization problem and discretized partial differential equations (PDE) with Neumann and Dirichlet boundary conditions. In fact, a more detailed and systematic study of SDP relaxations exploiting d- and r-space sparsity in the case of quadratic optimization problems derived from PDEs compared to the hierarchy of sparse SDP relaxations (2.18) is presented in Chapter 3. SDP relaxations of a quadratic SDP In this subsection, we apply the d- and r-space conversion methods to the quadratic SDP (2.64), and derive four kinds of SDP relaxations: (a) a dense SDP relaxation without exploiting any sparsity. (b) a sparse SDP relaxation by applying the d-space conversion method using basis representation given in 2.2.3. (c) a sparse SDP relaxation by applying the r-space conversion method using clique trees in 2.2.5. (d) a sparse SDP relaxation by applying both, the d-space conversion method using basis representation and the r-space conversion method using clique trees. We write each non-zero element Mij (x) as Mij (x) = Qij • 1 xT x xxT for every x ∈ Rs . for some Qij ∈ S1+s . Assume that the rows and columns of each Qij are indexed from 0 to s. Let us cij : Rs × Ss → R of the quadratic function Mij : Rs → R: introduce a linearization (or lifting) M T cij (x, X) = Qij • 1 x M for every x ∈ Rs and X ∈ Ss , x X c : Rs × Ss → Sn of M : Rs → Sn whose (i, j)th element is M cij . which induces a linearization (or lifting) M Then we can describe the dense SDP relaxation (a) for (2.64) as minimize n X i=1 c(x, X) < 0 and ci xi subject to M 1 x xT X For simplicity, we rewrite the dense SDP relaxation above as n X c(W ) < 0, W00 = 1 and W < 0, ci W0i subject to M (a) minimize i=1 < 0. 2.2. EXPLOITING SPARSITY IN LINEAR AND NONLINEAR MATRIX INEQUALITIES 47 where T s (W01 , W02 , . . . , W0s ) = x ∈ R and W = 1 xT x X ∈ S1+s . Let G(N ′ , F ′ ) be the d-space sparsity pattern graph for the SDP (a) with N ′ = {0, 1, . . . , s}, and F ′ = the set ofPdistinct row and column index pairs (i, j) of Wij that is necessary to evaluate the objective n function i=1 ci W0i and/or the LMI M (W ) < 0. Let G(N ′ , E ′ ) be a chordal extension of G(N ′ , F ′ ), and ′ ′ C1 , C2 , . . . , Cr′ be the maximal cliques of G(N ′ , E ′ ). Applying the d-space conversion method using basis representation, we obtain the SDP relaxation n X ci W0i minimize i=1 c((Wij : (i, j) ∈ J)) < 0, W00 = 1, (b) subject to M X C′ Eij Wij ∈ S+k (k = 1, 2, . . . , r). (i,j)∈J(Ck′ ) Here J = ∪rk=1 J(Ck′ ), (Wij : (i, j) ∈ J) = the vector variable of the elements Wij ((i, j) ∈ J) and c((Wij : (i, j) ∈ J)) M = c(W ) for every W ∈ Ss (E ′ , 0). M To apply the r-space conversion method using clique trees to the quadratic SDP (2.64), we assume that M : Rs → Sn (E, 0) for some chordal graph G(N, E) where N = {1, 2, . . . , n} and E ⊆ N × N . Then, we convert the matrix inequality M (x) < 0 in (2.64) into an equivalent system of matrix inequalities (2.56). The application of the LMI relaxation described above to (2.56) leads to the SDP relaxation n X minimize ci W0i (c) i=1 k e k (z) < 0 (k = 1, 2, . . . , p), W00 = 1, W < 0, subject to M (W ) − L k fk : Rs → SCk . We may apply the where M : S1+s → SCk denotes a linearization (or lifting) of M linearization to (2.64) first to derive the dense SDP relaxation (a), and then apply the r-space conversion method using clique trees to (a). This results in the same sparse SDP relaxation (c) of (2.64). Note that both c take values from Sn (E, 0). Thus, they provide the same r-space sparsity pattern characterized M and M by the chordal graph G(N, E). Finally, the sparse SDP relaxation (d) is derived by applying the d-space conversion method using basis representation to the the sparse LMI relaxation (c). We note that the d-space sparsity pattern graph for the SDP (c) with respect to the matrix variable W ∈ S1+s is the same as the one for the SDP (a). Hence, the sparse SDP relaxation (d) is obtained in the same way as the SDP (b) is obtained from the SDP (a). Consequently, we have the sparse SDP relaxation n X ci W0i minimize i=1 k e k (z) < 0 (k = 1, 2, . . . , p), W00 = 1, (d) subject to M ((Wij : (i, j) ∈ J)) − L X Cj′ Eαβ Wαβ ∈ S+ (j = 1, 2, . . . , r). ′ (α,β)∈J(Ck ) Here J = ∪rk=1 J(Ck′ ), (Wij : (i, j) ∈ J) = the vector variable of the elements Wij ((i, j) ∈ J) and k M ((Wij : (i, j) ∈ J)) = k M (W ) for every W ∈ Ss (E ′ , 0). Quadratic SDPs with d- and r-sparsity from randomly generated sparse graphs Quadratic SDP problems were constructed by first generating two graphs G(Nd , Ed ) with Ns = {1, 2, . . . , 1+ s} and G(Nr , Er ) with Nr = {1, 2, . . . , n} using the Matlab program generateProblem.m [44], which was 48 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION developed for sensor network localization problems. Sparse chordal extensions G(Nd , E d ) and G(Nr , E r ) were then obtained by the Matlab functions symamd.m and chol.m. Next, we generated data matrices Qij ∈ S1+s (i = 1, 2, . . . , n, j = 1, 2, . . . , n) and a data vector c ∈ Rs so that the d- and r-space pattern graphs of the resulting quadratic SDP coincide with G(Nd , Ed ) and G(Nr , Er ), respectively. Some characteristics of the chordal extensions G(Nd , E d ) of G(Nr , Er ) and G(Nr , E r ) of G(Nr , Er ) used in the experiments are shown in Table 4. For the problem with s = 40 and n = 640, the d- and r-space sparsity pattern obtained from the symmetric approximate minimum degree permutation of rows and columns by the Matlab function symamd.m is displayed in Figure 2.8. s 80 320 40 40 n 80 320 160 640 Domain space sparsity #E d NoC Max Min 143 63 3 3 649 260 7 3 70 30 3 3 70 30 3 3 Range space sparsity #E d NoC Max Min 216 72 7 3 840 301 9 3 426 150 7 3 1732 616 13 3 Table 2.2: Some characteristics of d- and r-sparsities of the tested quadratic SDPs. #E d (or #E r ) denotes the number of edges of G(Nd , E d ) (or G(Nr , E r )), NoC the number of the maximal cliques of G(Nd , E d ) (or G(Nr , E r )), Max the maximum size of the maximal cliques of G(Nd , E d ) (or G(Nr , E r )), and Min the minimum size of the maximal cliques of G(Nd , E d ) (or G(Nr , E r )). 0 0 5 100 10 200 15 300 20 25 400 30 500 35 600 40 0 5 10 15 20 25 nz = 181 30 35 40 0 100 200 300 nz = 3432 400 500 600 Figure 2.8: The d-space sparsity pattern of the quadratic SDP with s = 40 and n = 640 on the left and the r-space sparsity pattern on the right. Table 5 shows numerical results on the quadratic SDPs whose d- and r- sparsity characteristics are given in Table 4. We observe that both the d-space conversion method using basis representation in (b) and the r-space conversion method using clique tree in (c) work effectively, and that their combination (d) results in the shortest CPU time among the four methods. Quadratic SDPs arising from applications For additional numerical experiments, we selected five SDP problems from SDPLIB [9], an quadratic SDP from a sensor network localization problem, and two quadratic SDP derived from PDE with Neumann and Dirichlet boundary condition. The test problems in Table 2.4 are 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP s 80 n 80 320 320 40 160 40 640 49 SeDuMi CPU time in seconds (the size of the Schur complement matrix, the max. size of matrix variables) (a) (b) (c) (d) 296.51 1.38 1.58 0.73 (3321, 81) (224, 80) ( 801, 81) (252, 19) OOM 74.19 80.09 35.20 (970, 320) (322, 321) (1216, 20) 6.70 4.22 2.91 0.74 (861, 160) (111, 160) (1626, 41) (207, 21) 158.95 151.20 120.86 5.71 (861, 640) (111, 640) (6776, 41) (772, 21) Table 2.3: Numerical results on the quadratic SDPs with d- and r- sparsity from randomly generated sparse graphs. OOM indicates out of memory error in Matlab. mcp500-1, maxG11, maxG32: An SDP relaxation of the max cut problem from SDPLIB. thetaG11: An SDP relaxation of the Lovasz theta problem from SDPLIB. qpG11: An SDP relaxation of the box constrained quadratic problem form SDPLIB. d2n01s1000a100FSDP: A full SDP relaxation [6] of the sensor network localization problem with 1000 sensors, 100 anchors distributed in [0, 1]2 , radio range = 0.1, and noise = 10%. The method (iii) in Table 2.4 for this problem is equivalent to the method used in SFSDP [43], a sparse version of full SDP relaxation. ginzOrNeum(11): An SDP relaxation of the discretized nonlinear, elliptic PDE (4.4) (Case II, Neumann boundary condition) of [61]. We choose a 11 × 11 grid for the domain [0, 1]2 of the PDE. pdeBifurcation(20): An SDP relaxation of the discretized nonlinear, elliptic PDE (4.5) (Dirichlet boundary condition) of [61]. We choose a 20 × 20 grid for the domain [0, 1]2 of the PDE. The SDP relaxations (i), (ii) and (iii) in Table 2.4 indicate (i) a dense SDP relaxation without exploiting any sparsity. (ii) a sparse SDP relaxation by applying the d-space conversion method using clique trees given in Section 3.1. (iii) a sparse SDP relaxation by applying the d-space conversion method using basis representation given in Section 3.2. Table 2.4 shows that CPU time spent by (ii) is shorter than that by (i) and (iii) for all tested problems except for mcp500-1 and pdeEllipticNeum11. Notice that it took shorter CPU time to solve (iii) than (i) except for maxG32 and thetaG11. We confirm that applying at least one of the d-space conversion methods greatly reduces CPU time for the test problems. The d-space sparsity patterns for the test problems are displayed in Figures 2.9 and 2.10. 2.3 Reduction techniques for SDP relaxations for large scale POP The global minimization of a multivariate polynomial over a semialgebraic set is a severely nonconvex, difficult optimization problem in general. In 2.1.3 a hierarchy of SDP relaxations has been proposed whose 50 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Problem mcp500-1 maxG11 maxG32 thetaG11 qpG11 d2n01s1000a100FSDP ginzOrNeum(11) pdeBifurcation(20) SeDuMi CPU time (size.SC.mat., Max.size.mat.var.) (i) (ii) (iii) 65.5 (500, 500) 94.5 (7222, 44) 15.9 (2878, 44) 220.5 (800, 800) 12.1 (2432, 80) 26.8 (8333, 24) 5373.8 (2000, 2000) 971.4 (13600, 210) OOM 345.9 (2401, 801) 23.9 (4237, 81) 458.5 (9134, 25) 2628.5 (800, 1600) 16.0 (2432, 80) 72.5 (9133, 24) 5193.5 (4949, 1002) 16.9 (7260, 45) 19.5 (15691, 17) 216.1 (1453, 485) 2.2 (1483, 17) 2.1 (1574, 4) 1120.4 (2401, 801) 4.3 (2451, 17) 5.3 (2001, 3) Table 2.4: Numerical results on SDPs from some applications. size.SC.mat. denotes the size of the Schur complement matrix and Max.size.mat.var. the maximum size of matrix variables. optima have been proven to converge to the optimum of a POP for increasing order of the relaxation. The practical use of this powerful theoretical result has been limited by the capacity of current SDP solvers, as the size of the SDP relaxations grows rapidly with increasing order. A first approach to attempt this problem has been the concept to exploit structured sparsity in a POP [47]. Whenever a POP satisfies a certain sparsity pattern, a convergent sequence of sparse SDP relaxations (2.18) of substantially smaller size can be constructed. Compared to the dense SDP relaxation (2.8), the sparse SDP relaxation (2.18) can be applied to POPs of larger scale. Still, the size of the sparse SDP relaxation remains the major obstacle in order to solve large scale POPs, which contain polynomials of higher degree. We propose a substitution procedure to transform an arbitrary POP into an equivalent quadratic optimization problem (QOP). It is based on replacing quadratic terms in higher degree monomials by new variables successively, and adding the substitution relations as constraints to the optimization problem. The idea to transform a POP into an equivalent QOP can be traced back to Shor [92], who exploited it to derive dual lower bounds for the minimum of a polynomial function. As the substitution procedure is not unique, we introduce different heuristics which aim at deriving a QOP with as few additional variables as possible. Moreover, we show that sparsity of a POP is maintained under the substitution procedure. The main advantage of deriving an equivalent QOP for a POP is that the sparse SDP relaxation of first order can be applied to solve it approximately. The substitution procedure and the considerations to minimize the number of additional variables while maintaining the sparsity are presented in 2.3.1. While a POP and the QOP derived from it are equivalent, we face the problem that the quality of the SDP relaxation for a QOP deteriorates in many cases. We discuss in 2.3.2 how to tighten the SDP relaxation for a QOP in order to achieve good approximations to the global minimum even for SDP relaxation of first or second order. For that purpose methods as choosing appropriate lower and upper bounds for the multivariate variables, Branch-and-Cut bounds to shrink the feasible region of the SDP relaxation and locally convergent optimization methods are proposed. Finally, the power of this technique is demonstrated in 2.3.3, where it is applied to solve various large scale POP of higher degree. 2.3.1 Transforming a POP into a QOP The aim of 2.3 is to propose a technique to reduce the size of SDP relaxations for general POPs, which enables us to attempt large scale polynomial optimization efficiently. This technique, which transforms a POP into an equivalent QOP, reduces the size of the SDP relaxation by decreasing the minimum relaxation order ωmax , whereas the technique due to [102] presented in 2.1.4 aims at reducing the SDP relaxation n+ω by replacing matrix inequality constraints of size through matrix inequality constraints of size ω 51 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP 0 200 0 0 100 200 200 400 300 600 400 800 500 1000 600 1200 700 1400 400 600 800 1000 1200 1400 1600 1800 2000 800 0 500 1000 nz = 10000 1500 2000 0 100 200 300 400 nz = 5601 500 600 700 800 1600 0 200 400 600 800 1000 nz = 4800 1200 1400 1600 Figure 2.9: The d-space sparsity pattern with the symmetric approximate minimum degree permutation of rows and columns provided by the Matlab function symamd.m for maxG32(left), thetaG11(middle) and qpG11(right). 0 0 100 50 200 100 300 150 400 200 0 100 200 300 500 400 250 600 300 700 500 350 600 800 400 700 900 450 1000 0 200 400 600 nz = 9138 800 1000 0 50 100 150 200 250 300 nz = 2421 350 400 450 800 0 100 200 300 400 nz = 3201 500 600 700 800 Figure 2.10: The d-space sparsity pattern with the symmetric approximate minimum degree permutation of rows and columns provided by the Matlab function symamd.m for d2n01s1000a100FSDP(left), pdeEllipticNeum11(middle) and pdeEllipticBifur20(right). ni + ω p . A general QOP is a special case of the POP (2.1), where the polynomials p, gi (i = 1, . . . , m) ω are at most of degree 2. With respect to the definition of ωk , the minimal relaxation order ωmax of the sparse SDP relaxation (2.18) equals one. As pointed out in [102], the sparse SDP relaxation sSDP1 and the dense SDP relaxation dSDP1 of order one are equivalent for any QOP. The equivalence of a QOP and its SDP relaxation has been shown for a few restrictive classes of QOPs. For instance, if in a QOP p and −gi (i = 1, . . . , m) are convex quadratic polynomials, the QOP is equivalent to the corresponding SDP relaxation [56]. Also, equivalence of QOPs and their SDP relaxations was shown for the class of uniformly OD-nonpositive QOPs [41]. As shown in [52], min(sSDPω ) → min(POP) for ω → ∞, but to the best of our knowledge there is no result for a rate of convergence or guaranteed approximation of the global minimum for a fixed relaxation order ω ≥ ωmax in the case of a general POP. To illustrate the idea of our transformation technique, consider the following example of a simple unconstrained POP, whose optimal value is −∞: min 10x31 − 102 x31 x2 + 103 x21 x22 − 104 x1 x32 + 105 x42 (2.65) It is straight forward that POP (2.65) is equivalent to min 10x1 x3 − 102 x3 x4 + 103 x24 − 104 x4 x5 + 105 x25 s.t. x3 = x21 , x4 = x1 x2 , x5 = x22 , (2.66) 52 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION where we introduced three additional variables x3 , x4 and x5 . Obviously QOP (2.66) is not the only QOP equivalent to POP (2.65): The QOP min s.t. 10x3 − 102 x2 x3 + 103 x5 x6 − 104 x1 x4 + 105 x2 x4 x3 = x1 x5 , x4 = x2 x6 , x5 = x21 , x6 = x22 , (2.67) is equivalent to (2.65) as well. We notice the number of additional variables in QOP (2.66) equals three, whereas it equals four in QOP (2.67). Thus, there are numerous ways to transform a higher degree POP into a QOP in general. For the transformation procedures we are proposing, we consider 1) the number of additional variables should be as small as possible, in order to obtain a SDP relaxation of smaller size, 2) sparsity of a POP should be maintained under the transformation and 3) the quality of the SDP relaxation for the derived QOP should be as good as possible. How to deal with 3) is discussed in 2.3.2, 1) and 2) are discussed in the following. Maintaining sparsity The transformation proposed in the previous subsection raises the question, whether the correlative sparsity of a POP is preserved under the transformation, i.e., whether the resulting QOP is correlatively sparse as well. Let POP⋆ be a correlatively sparse POP of dimension n, G(N, E ′ ) the chordal extension of its csp graph, (C1 , . . . , Cp ) the maximal cliques of G(N, E ′ ) and nmax = maxi=1,...,p | Ci |. Let xn+1 = xi xj be ˜ denote the POP derived after substituting the substitution variable for some i, j ∈ {1, . . . , n}. Let POP ⋆ ′ xn+1 = xi xj in POP . Given the chordal extension G(N, E ) of the csp graph of POP⋆ , a chordal extension ˜ over the vertex set Ñ = N ∪ {n + 1} can be obtained by the extension: For a clique of the csp graph of POP Cl with {i, j} ⊂ Cl add the edges {v, n + 1} for all v ∈ Cl and obtain the clique C̃l . For each clique Ck not containing {i, j}, set C̃k = Ck . In the end we obtain the graph G(Ñ , Ẽ ′ ) which is a chordal extension of ˜ the csp graph G(Ñ , Ẽ) of POP. Note, (C̃1 , . . . , C̃p ) are maximal cliques for G(Ñ , Ẽ ′ ) and for all C̃l holds | C̃l |≤| Cl | +1, i.e. ñmax ≤ nmax + 1. Moreover, the number of maximal cliques p remains unchanged under the transformation. As pointed out, G(Ñ , Ẽ ′ ) is one possible chordal extension of G(Ñ , Ẽ). It seems reasonable to expect that the heuristics we are using for the chordal extension, such as the reverse CuthillMcKee and the symmetric minimum degree ordering, add less edges to G(Ñ , Ẽ) than we did in constructing G(Ñ , Ẽ ′ ). Thus, we are able to apply the sparse SDP relaxations efficiently to the POPs derived after each iteration of the transformation algorithm. For illustration we consider Figure 2.11 and Figure 2.12, where the csp matrices of two POPs and their QOPs are pictured. 0 0 5 10 10 20 15 30 20 40 25 50 30 60 35 70 0 5 10 15 20 nz = 598 25 30 35 0 10 20 30 40 nz = 940 50 60 70 Figure 2.11: CSP matrix of the chordal extension of POP pdeBifurcation(7) (left) and its QOP (right) derived under strategy BI. 53 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP 0 0 5 10 10 20 15 20 30 25 40 30 50 35 40 60 45 70 0 5 10 15 20 25 nz = 412 30 35 40 45 0 10 20 30 40 nz = 624 50 60 70 Figure 2.12: CSP matrix of the chordal extension of POP Mimura(25) (left) and its QOP (right) derived under strategy BI. We observe that the sparsity pattern of the chordal extension of the csp graph is maintained under the substitution procedure. However, if the number of substitutions, which is required to transform a higher degree POP into a QOP, is far greater than the number of variables of the original POP, it may occur that we obtain a dense QOP under the transformation procedure. To illustrate this effect, consider the chordal extension of csp matrix of the QOP derived for the POP randomEQ(7,3,5,8,0) which is pictured in Figure 2.13. In that example, the number n of variables of the original POP equals seven, the number of additional variables equals 108. 0 0 1 20 2 40 3 4 60 5 80 6 100 7 8 0 1 2 3 4 nz = 43 5 6 7 8 0 20 40 60 nz = 2235 80 100 Figure 2.13: CSP matrix of the chordal extension of POP randomWithEQ(7,3,5,8,0) (left) and its QOP (right). Minimizing the number of additional variables Let n denote the number of variables involved in a POP and ñ the number of variables in the corresponding QOP. The first question we are facing is, how to transform a POP into a QOP such that the number k0 := ñ − n of additional variables is as small as possible. Each additional variable xn+k corresponds to 54 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION the substitution of a certain quadratic monomial xi xj by xn+k . Given an arbitrary POP, the question to find a substitution procedure minimizing ñ is a difficult problem. We propose four different heuristics for transforming a POP into a QOP, which aim at reducing the number k0 of additional variables. At the end of this section we give some motivation, why it is more important to find a strategy optimizing the quality of the SDP relaxation than one that minimizes the number k0 of additional variables. Our transformation algorithm iterates substitutions of pairs of quadratic monomials xi xj in the higher degree monomials in objective function and constraints by a new variable xn+k , and adding the substitution relation xn+k = xi xj as constraints to the POP. Let POP0 denote the original POP, and POPk the POP obtained after the k-th iteration, i.e. after substituting xn+k = xi xj and adding it as constraint to POPk−1 . The algorithm terminates as soon as POPk0 is a QOP for some k0 ∈ N. In each iteration of the transformation algorithm we distinguish two steps. The first one is to choose which pair of variables (xi , xj ) (1 ≤ i, j ≤ n + k) is substituted by the additional variable xn+k+1 . The second one is to choose to which extent xi xj is substituted by xn+k+1 in each higher degree monomial. Step 1: Choosing the substitution variables k Definition 2.7 Let POPk be a POP of dimension ñ with m̃ constraints (g1k , . . . , gm ). The higher monok k mial set MS of POP is given by MkS = α ∈ Nñ | ∃i ∈ {0, . . . , m̃} s.t. α ∈ supp(gik ) and | α |≥ 3 , where g0 := p, gi0 := gi , and the higher monomial list Mk of POPk by Mk = (α, wα ) | α ∈ MkS and wα := # i | α ∈ supp(gik ) . By Definition 2.7, the higher monomial list of a QOP is empty. Definition 2.8 Given α ∈ Nn and a pair (i, j) where 1 ≤ i, jα ≤ n, we define the dividing coefficient x xα α ∈ R[x] and / R[x]. ki,j ∈ N0 as the integer that satisfies α k kα +1 ∈ i,j i,j (xi xj ) k (xi xj ) 0 Given POP the k-th iterate of POP and its higher monomial list Mk , determine the symmetric matrix C(POPk ) ∈ R(n+k)×(n+k) given by X α ki,j wα . C(POPk )i,j = C(POPk )j,i = (α,wα )∈Mk We consider two alternatives to choose a pair (xi , xj )(1 ≤ i, j ≤ n + k) to be substituted by xn+k+1 : A. Naive criterion: xα xi xj ∈ R[x]. Choose a pair (xi , xj ) such that there exists a α ∈ MS (POPk ) which satisfies B. Maximum criterion: Choose a pair (xi , xj ) such that C(POPk )i,j ≥ C(POPk )u,v ∀1 ≤ u, v ≤ n+k. Step 2: Choose the substitution strategy Next we have to decide to what extent we substitute xn+k+1 = xi xj in each monomial of MS (POPk ). We will distinguish full and partial substitution. Let us demonstrate the importance of considering that question on the following two examples. Example 2.5 Consider two different substitution strategies for transforming the problem to minimize x41 into a QOP: min x41 (1) ւ ց (2) min x22 min x21 x2 s.t. x2 = x21 s.t. x2 = x21 (2.68) ↓ min x1 x3 s.t. x2 = x21 x3 = x1 x2 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP 55 In both substitution strategies, we choose x21 for substitution in the first step. In (1) we fully substituted x21 by x2 , whereas in (2) we substituted x21 partially. By choosing full substitution in the first iteration in (1), we need one additional variable to obtain a QOP, partial substitution requires two additional variables to yield a QOP. Example 2.6 (1) min s.t. ↓ min s.t. ↓ min s.t. ւ x33 x1 x2 x3 ≥ 0 x3 = x21 min x61 s.t. x31 x2 ≥ 0 ց min s.t. ↓ min s.t. x3 x4 x1 x2 x3 ≥ 0 x3 = x21 x4 = x23 (2) x21 x23 x1 x2 x3 ≥ 0 x3 = x21 x24 x2 x4 ≥ 0 x3 = x21 x4 = x1 x2 (2.69) x3 x4 x2 x5 ≥ 0 x3 = x21 x4 = x23 x5 = x1 x3 In this example full substitution (1) of x21 requires three, and partial substitution (2) only two additional variables to yield a QOP. The examples illustrate it depends on the structure of the higher monomial set, whether partial or full substitution requires less additional variables and results in a smaller size of the SDP relaxation. In general partial and full substitution are given as follows. I. Full substitution: Let tf ri,j : R[x] → R[z], where x ∈ Rr and z ∈ Rr+1 for a r ∈ N and i, j ∈ {1, . . . , r}, be a linear operator defined by its mappings for each monomial xα , ( min(α ,α ) αj−1 αj −min(αi ,αj ) αj+1 αi−1 αi −min(αi ,αj ) αi+1 zj zj+1 . . . zrαr zr+1 i j , if i 6= j, z1α1 . . . zi−1 zi zi+1 . . . zj−1 r α αi tf i,j (x ) = ⌊ 2 ⌋ αi−1 mod(αi ,2) αi+1 zi+1 . . . zrαr zr+1 zi z1α1 . . . zi−1 , if i = j. P substitutes xi xj Thus, tf ri,j (g(x)) = α∈supp(g) cα (g)tf ri,j (xα ) for any g ∈ R[x]. The operator tf n+k i,j by xn+k+1 in each monomial to the maximal possible extent. II. Partial substitution: Let tp ri,j : R[x] → R[z], where x ∈ Rr and z ∈ Rr+1 for a r ∈ N and i, j ∈ {1, . . . , r}, be a linear operator defined by its mappings for each monomial xα , if i 6= j, tf ri,j (xα ), t r (xα ), if i = j and αi odd, f i,j tp ri,j (xα ) = r α if i = j and log2 (αi ) ∈ N0 , tf i,j (x ), z α1 . . . z αi−1 z gi z αi+1 . . . z αr z 21 (αi −gi ) , else, 1 i−1 i i+1 r where gi := gcd(2⌊log2 (αi )⌋ , αi ). Thus, tp ri,j (g(x)) = r+1 P r α α∈supp(g) cα (g)tp i,j (x ) for any g ∈ R[x]. We notice that full and partial substitution only differ in the case i = j, αi even and log2 (αi ) ∈ / N0 holds. By pairwise combining the choice of A or B in Step 1 and the choice of I or II in Step 2, we obtain four different procedures to transform POPk−1 into POPk that we denote as AI, AII, BI and BII. We do not expect AI or AII to result in a POP with a small number of substitutions, as A does not take into account 56 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION the structure of the higher degree monomial list MSk−1 , but we use AI and AII to evaluate the potential of BI and BII. The numerical performance of these four procedures is demonstrated on some example POPs in Table 2.6, where n denotes the number of variables in the original POP, deg the degree of the highest order polynomial in the POP, and k0 the number of additional variables required to transform the POP into a QOP under the respective substitution strategy. The POPs pdeBifurcation(n) are derived from discretizing differential equations, which is the topic of Chapter 3, the other POPs are test problems from [102]. As expected, strategy B is superior to A for all but one example class of POP, when reducing the number of variables is concerned. The entire algorithm to transform a POP into a QOP can be summarized by the scheme in Table 2.5. As mentioned before the QOP of dimension n + k derived by AI, AII, BI or BII is equivalent to the original POP of dimension n. In fact it is easy to see, if x̃ ∈ Rn+k an optimal solution of the QOP, the vector (x̃1 , . . . , x̃n ) of the first n components of x̃ is an optimizer of the original POP. INPUT WHILE 1. 2. 3. OUTPUT POP0 with M0S MkS 6= ∅ Determine the pair (xi , xj ) for substitution by A or B. Apply tf ki,j or tp ki,j to each polynomial in POPk and derive POPk+1 . Update k → k + 1, POPk → POPk+1 , MkS → Mk+1 S . QOP = POPk0 Table 2.5: Scheme for transforming a POP into a QOP. POP BroydenBand(20) BroydenBand(60) nondquar(32) nondquar(8) optControl(10) randINEQ(8,4,6,8,0) randEQ(7,3,5,8,0) pdeBifurcation(5) pdeBifurcation(10) randINEQ(3,1,3,16,0) randUnconst(3,2,3,14,0) n 20 60 32 8 60 8 7 25 100 3 3 deg 6 6 4 4 4 8 8 3 3 16 14 k0 (AI) 229 749 93 21 60 253 135 25 100 145 86 k0 (AII) 211 691 93 21 60 307 146 25 100 192 107 k0 (BI) 60 180 94 22 60 248 116 25 100 105 63 k0 (BII) 40 120 94 22 60 238 115 25 100 117 69 Table 2.6: Number of required variables for strategies AI, AII, BI and BII. Computational complexity Finally, let us consider how the size of the sparse SDP relaxation of order ω = 1 for a QOP depends on the number k0 of additional variables. Let a sparse POP of dimension n be given by the polynomials (p, g1 , . . . , gm ) and the maximal cliques (C1 , . . . , Cp ) of the chordal extension. With the construction in Maintaining sparsity above, the corresponding QOP of dimension ñ = n + k0 has the maximal cliques (C̃1 , . . . , C̃p ) such that Ci ⊆ C̃i and ñi ≤ ni + k0 for all (i = 1, . . . , p), where ni =| Ci | and ñi =| C̃i |. All partial localizing matrices M0 (gk y, F̂k ) are scalars in sSDP1 (QOP ). The size of the partial moment matrices M1 (y, C̃i ) is d(1, ñi ) = ñi + 1 ≤ ni + k0 + 1 = O(k0 ). (2.70) 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP 57 Thus, the size of the linear matrix inequality is bounded by m+k X0 j=1 1+ p X i=1 d(1, ñi ) ≤ m + k0 + p (nmax + k0 + 1) ≤ m + k0 + n (nmax + k0 + 1). (2.71) The length of the vector variable y in sSDP1 (QOP ) is bounded by Pp Pp | y | ≤ i=1 | y(C̃p ) |= i=1 d(2, 2ñi ) ≤ 12 p (2nmax + 2k0 + 2) (2nmax + 2k0 + 1) ≤ 2p (nmax + k0 + 1)2 ≤ 2n (nmax + k0 + 1) = O(k02 ). (2.72) Thus, the size of the linear matrix inequalities of the sparse SDP relaxation is linear and the length of the moment vector y quadratic in the number k0 of additional variables. For this reason the computational cost does not grow too fast, even if k0 is not minimal. Heuristics BI and BII are sufficient in order to derive QOP with a small number k0 of additional variables. Moreover, the bounds (2.71) and (2.72) for the size of the primal and dual variables of the SDP relaxation for the QOP are to be compared to the respective bounds for the SDP relaxation of the POP. If we assume ωmax = ωi for all i ∈ {1, . . . , m}, the size of the linear matrix inequality in the SDP relaxation of order ωmax for the original POP can be bounded by m X j=1 d(nj , ωmax − ωj ) + p X i=1 d(ni , ωmax ) ≤ m + n nmax + ωmax ωmax , (2.73) and the length of the moment vector by p X i=1 d(2ni , 2ωmax ) ≤ n 2nmax + 2ωmax 2nmax . (2.74) Already for ωmax = 2 the bounds (2.73) and (2.74) are of second and fourth degree in nmax , whereas (2.71) and (2.72) are linear and quadratic in nmax + k0 , respectively. Therefore we can expect a substantial reduction of the SDP relaxation under the transformation procedure. Note, we did not exploit any sparsity in the SDP relaxation or any intersection of the maximal cliques (C1 , . . . , Cp ) and (C̃1 , . . . , C̃p ) when deriving these bounds. Thus, the actual size of SDP relaxations in numerical experiments may be far smaller than the one suggested by these bounds. 2.3.2 Quality of SDP relaxations for QOP A polynomial optimization problem (POP) and the quadratic optimization problem (QOP) derived from it under one of the transformation strategies AI, AII, BI or BII are equivalent. However, the same statement does not hold for the SDP relaxations of both problems. In fact, the SDP relaxation for QOP is weaker than the SDP relaxation for the original POP. Before stating this negative result, we consider an example to illustrate it. Example 2.7 Let a POP and its equivalent QOP be given by POP min s.t. ⇔ QOP min x̃23 x21 x22 s.t. x̃1 x̃3 ≥ 0 x21 x2 ≥ 0 x̃1 x̃2 = x̃3 . 58 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION The dense SDP relaxations of minimal relaxation order dSDP2 (POP) and dSDP1 (QOP) are given by min y22 s.t. y21 ≥ 0 M2 (y) = y00 y10 y01 y20 y11 y02 y10 y20 y11 y30 y21 y12 y01 y11 y02 y21 y12 y03 y20 y30 y21 y40 y31 y22 y11 y21 y13 y31 y22 y13 y02 y12 y03 y22 y13 y04 min ỹ002 s.t. ỹ101 ≥ 0 ỹ110 = ỹ001 <0 ỹ000 ỹ100 M1 (ỹ) = ỹ010 ỹ001 ỹ100 ỹ200 ỹ110 ỹ101 ỹ010 ỹ110 ỹ020 ỹ011 ỹ001 ỹ101 < 0. ỹ011 ỹ002 The equivalence of POP and QOP holds with the relation = (x̃1 , x̃2 , x̃3 , x̃21 , x̃1 x̃2 , x̃1 x̃3 , x̃22 , x̃2 x̃3 , x̃23 ) (x1 , x2 , x1 x2 , x21 , x1 x2 , x21 x2 , x22 , x1 x22 , x21 x22 ). (2.75) Given a feasible solution y ∈ Rd(2,4) = R15 for dSDP2 (POP), we exploit the relations (2.75) to define a vector ỹ = (ỹ000 , ỹ100 , ỹ010 , ỹ001 , ỹ200 , . . . , ỹ002 ) ∈ Rd(3,1) = R10 as ỹ := (y00 , y10 , y01 , y11 , y20 , y11 , y21 , y02 , y12 , y22 ). Then ỹ110 = y11 = ỹ001 holds by definition of ỹ, and ỹ101 dSDP2 (P OP ). Furthermore, for the moment matrix, we have y00 y10 y01 y10 y20 y11 M1 (ỹ) = y01 y11 y02 y11 y21 y12 = y21 ≥ 0 as y is a feasible solution of y11 y21 . y12 y22 Thus M1 (ỹ) < 0, as M1 (ỹ) is a principal submatrix of M2 (y) < 0. It follows that ỹ is feasible for dSDP1 (QOP) and that min(dSDP1 (QOP)) ≤ min(dSDP2 (POP)) holds. A generalization of the observation in Example 2.7 is given by the following proposition. Proposition 2.1 Let a POP of dimension n with ωmax > 1 of form (2.1) be given by the set of polynomials (p, g1 , . . . , gm ) and the corresponding QOP of dimension n + k derived via AI, AII, BI or BII by (p̃, g̃1 , . . . , g̃m ). Then, for each feasible solution of dSDPωmax (POP), there exists a feasible solution of dSDP1 (QOP) with the same objective value. Thus, min(dSDP1 (QOP))≤ min(dSDPωmax (POP)). Proof: Let y ∈ Rd(n,2ωmax) be a feasible solution of SDPωmax (POP). Each yα corresponds to a monomial xα for all α with | α |≤ 2ωmax , x ∈ Rn . Moreover, with respect to the substitution relation for all monomials x̃α ∈ Rn+k with | α |≤ 2 there exists a monomial xβ(α) ∈ Rn , such that x̃α = xβ(α) , | β(α) |≤ 2ωmax . (2.76) As β(·) in (2.76) is constructed via the substitution relations, β(α1 ) = β(α2 ) (2.77) holds for α1 , α2 ∈ Nn+k with | α1 |=| α2 |≤ 2, whenever QOP has a substitution constraint x̃α1 = x̃α2 . Now, define ỹ ∈ Rd(n+k,2) where ỹα := yβ(α) for all | α |≤ 2. Then, ỹ is feasible for dSDP1 (QOP), as all equality constraints derived from substitutions are satisfied due to (2.77), and as the principal submatrices of moment matrix M1 (ỹ) and of the localizing matrices M0 (ỹg̃k ) (k = 1, . . . , m), which are obtained by simultaneously deleting rows/columns linear dependent on the remaining rows/columns, are principal 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP 59 submatrices of Mωmax (y) and Mωmax −ωk (y gk ) (k=1,. . . ,m), respectively. Finally, the objective values for y and ỹ coincide. This result for the dense SDP relaxation can be extended to the sparse SDP relaxation of minimal relaxation order in an analog manner, if the maximal cliques (C̃1 , . . . , C̃m ) of the chordal extended csp graph of the QOP are chosen appropriately with respect to the maximal cliques of the chordal extended csp graph of the POP. Therefore it seems reasonable to expect that in general sSDP1 for the QOP provides an approximation to the global minimum of the POP which is far weaker than the one provided by sSDPωmax for the original POP. One possibility to strengthen the SDP relaxation for QOPs is to increase the relaxation order to some ω > 1. But, as in the case of the SDP relaxation for POP we can not guarantee to find the global minimum of a general QOP for any fixed ω ∈ N. Moreover each of the additional equality constraints results in 21 (d(n, ω − 1) + 1)d(n, ω − 1) equality constraints in sSDPω (QOP) for ω > 1. Therefore it seems more promising to consider additional techniques to improve the quality of the sSDP1 for QOPs. Local optimization methods As pointed out before, the minimum of the sparse SDP relaxation converges to the minimum of the QOP for ω → ∞. Moreover, an accurate approximation can be obtained by the sparse SDP relaxation of order ω ∈ {ωmax , . . . , ωmax + 3} for many POPs [102]. However, the quality of the sparse SDP relaxation for the QOP is weaker than the one for the original POP. Therefore, the solution provided by the sparse SDP relaxation for the QOP can be understood as a first approximation to the global minimizer of the original POP, and it may serve as initial point for a locally convergent optimization technique applied to the original POP. For instance sequential quadratic programming (SQP) [8] can be applied to POP where the sparse SDP solution for the corresponding QOP is taken as starting point. In the case a POP has equality constraints only, the number of constraints coincides with the number of variables and the feasible set is finite, we may succeed in finding the global optimizer of the POP by applying Newton’s method for nonlinear systems [77] to the polynomial system given by the feasible set of the POP, again starting from the solution provided by the sparse SDP relaxation for the QOP. Higher accuracy via Branch-and-Cut bounds The sparse SDP relaxations (2.18) incorporate lower and upper bounds for each component of the ndimensional variable, lbdi ≤ xi ≤ ubdi , ∀i ∈ {1, . . . , n}, (2.78) in order to establish the compactness of the feasible set of a POP. Compactness is a necessary condition to guarantee the convergence of the sequence of sparse SDP relaxations towards the global optimum of the POP. Moreover, the numerical performance for solving the sparse SDP relaxations depends heavily on the bounds (2.78). The tighter we choose these bounds, the better approximates the solution of the SDP the minimizer of the POP. Prior to solving the sparse SDP relaxations for the QOP derived from a POP, we fix the bounds (2.78) for the components of the POP and determine lower and upper bounds for the additional variables according to the substitution relation. For instance for xn+1 = x2i the bounds are defined as ( 0, if lbdi ≤ 0 ≤ ubdi lbdn+1 = , 2 2 (2.79) min(lbdi , ubdi ), else, 2 2 ubdn+1 = max(lbdi , ubdi ). In 2.3.3 we will discuss the sensitivity of the choice of the lower and upper bounds on the accuracy of the SDP solution for some example POPs. A more sophisticated technique to increase the quality of the SDP relaxation of the QOP is inspired by a Branch-and-Cut algorithm for bilinear matrix inequalities due to Fukuda and Kojima [22]. As nonconvex quadratic constraints can be reduced to bilinear ones, we are able to adapt this technique for a QOP derived from a higher degree POP. The technique is based on cutting the feasible region of the SDP, such that every feasible solution of the QOP remains feasible for the SDP. We distinguish two sets of constraints which 60 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION resemble the convex relaxations (5) proposed in [22]. Let (p, g1 , . . . , gm ) be a QOP with lower and upper bounds lbdi and ubdi for all components xi (i = 1, . . . , n). The first set of constraints we consider is the following. For each constraint gi (i = 1, . . . , m) of form xk = xi xj with i 6= j we add the following constraints to the QOP xk ≤ ubdj xi + lbdi xj − lbdi ubdj (2.80) xk ≤ lbdj xi + ubdi xj − ubdi lbdj . For each constraint of the form xk = x2i we add the following constraint to the QOP xk ≤ (ubdi + lbdi ) xi − lbdi ubdi . (2.81) The second set of constraints shrinks the feasible set of the SDP relaxation even further than the constraints (2.80) and (2.81). For each monomial xi xj of degree 2 which occurs in the objective p or one of the constraints gi (i = 1, . . . , m) of the QOP, we add constraints as follows. If the QOP contains a constraint gi (i = 1, . . . , m) of the form xk = xi xj , we add the constraints (2.80) for i 6= j and (2.81) for i = j. If the QOP does not contain a constraint xk = xi xj , we add the quadratic constraints xi xj xi xj for i 6= j and the constraint x2i ≤ ubdj xi + lbdi xj − lbdi ubdj ≤ lbdj xi + ubdi xj − ubdi lbdj ≤ (ubdi + lbdi ) xi − lbdi ubdi (2.82) (2.83) for i = j. When linearized, both, the linear constraints (2.80) and (2.81) and the quadratic constraints (2.82) and (2.83) result in a smaller feasible region of the SDP relaxation which still contains the feasible region of the QOP. The efficiency of these sets of additional constraints is demonstrated in Section 2.3.3 as well. Remark 2.8 A general QOP given by (p, g1 , . . . , gm ) of dimension n can be transformed into a QOP of dimension n+1 with linear objective function by adding the equality constraint h(x, xn+1 ) := xn+1 −p(x) = 0 and choosing xn+1 as objective. A QOP with linear objective is a special case of a quadratic SDP (2.64), which we can apply the SDP relaxation (a) - (d) from 2.2.7 to. Thus, an arbitrary POP can be attempted by three different SDP relaxations. 1) The sparse SDP relaxations (2.18) exploiting correlative sparsity applied directly to the POP, 2) the sparse SDP relaxations (2.18) applied to an equivalent QOP, and 3) the SDP relaxations (a)-(d) from 2.2.7 exploiting d- and/or - r-space sparsity applied to an equivalent QOP. Remark 2.9 The constraints (2.82) are a particular case of reformulation-linearization-techniques (RLT) [89, 90]. For the SDP relaxation (b) from 2.2.7, instead of the constraints (2.80) - (2.83) we impose RLT constraints in the following way: Add the constraints Wi,j Wi,j Wi,j Wi,j − lbdi W1,j − lbdj W1,i − ubdi W1,j − ubdj W1,i − lbdi W1,j − ubdj W1,i − lbdj W1,i − ubdi W1,j ≥ −lbdi lbdj , ≥ −ubdi ubdj , ≤ −lbdi ubdj , ≤ −lbdj ubdi , (2.84) to the SDP relaxation (b), if {i, j} subset of some clique of the chordal extension of the d-space sparsity pattern graph. The constraints (2.84) strengthen the SDP relaxation and preserve the d-space sparsity structure at the same time. In the latter, whenever we apply the SDP relaxations (b) from 2.2.7 to a QOP, we impose the constraints (2.84). 2.3.3 Numerical examples The substitution procedure and the sparse SDP relaxations are applied to a number of test problems. These test problems encompass medium and large scale POPs of higher degree. The numerical performance of the sparse SDP relaxations of these POPs under the transformation algorithm is evaluated. In the following the 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP Substitution AI AII BI BII ñ 138 153 115 119 size(Aq ) [777, 6934] [828, 6922] [753, 5785] [788, 6497] 61 nnz(Aq ) 7106 7116 5934 6653 Table 2.7: Size of sSDP1 for QOPs from POP randEQ(7,3,5,8,0) with n = 7. Branch-and-Cut bounds (2.80) and (2.81) are denoted as linear BC-bounds, (2.82) and (2.83) as quadratic BC-bounds. The optional application of sequential quadratic programming starting from the solution of the SDP relaxation is abbreviated as SQP. Given a numerical solution x of an equality and inequality constrained POP, its scaled feasibility error is given by ǫsc = min {− | hi (x)/σi (x) |, min {gj (x)/σ̂j (x), 0} ∀ i, j} , where hi (i = 1, . . . , k) denote the equality constraints, gj (j = 1, . . . , l) the inequality constraints, and σi and σ̂j are the maxima of the monomials in the corresponding polynomials hi and gj at x, respectively. Note, an equality and inequality constrained POP is a special case of the POP (2.1), if we define fi := gi (i = 1, . . . , l), gi := hi (i = l + 1, . . . , l + k) and gi := −hi (i = k + l + 1, . . . , 2k + l). The value of the objective function at x is given by f0 (x). Let NC denote the number of constraints of a POP. ’OOM’ as entry for the scaled feasibility error denotes the size of the SDP is too large to be solved by SeDuMi [95] and results in a memory error (’Out of memory’). A two-component entry for lbd or ubd indicates that the first component is used as a bound for the first n2 variables and the second component for the remaining n2 variables of the POP. All numerical experiments are conducted on a LINUX OS with CPU 2.4 GHz and 8 Gb memory. The total processing time in seconds is denoted as tC . Randomly generated POPs As a first class of test problems, consider randomly generated POPs with inequality or equality constraints. We are interested in the numerical performance of the sparse SDP relaxation for the corresponding QOPs for different substitution strategies and different choices of lower, upper and Branch-and-Cut bounds. We will focus on comparing strategies BI and BII as they yield POPs with a small number of additional variables. For the random equality constrained POP randEQ(7,3,5,8,0) [102] of degree 8 with 7 variables, the size of the SDP relaxation sSDP4 is described by the matrix Ap of size [2870, 95628] with 124034 non-zero entries. This size is reduced substantially under each of the four substitution strategies, as can be seen in Table 2.7. In this table the matrix Aq in SeDuMi input format [95] and its number of nonzero entries nnz(Aq ) describe the size of the sparse SDP relaxation. The reduction of the size of the SDP relaxation results in reducing the total processing time tC by two magnitudes, as can be seen in Table 2.8. Moreover, as reported in Table 2.8, the performance of AI, AII, BI and BII is similar - with the one of BI being slightly better than the others. In this example with few equality constraints, it is easy to obtain a feasible solution, but it requires additional techniques as SQP to obtain an optimal solution. We know, min(sSDP1 (QOP )) and min(sSDP4 (P OP )) are lower bounds for min(POP). Moreover, min(sSDP1 (QOP )) ≤ min(sSDP4 (P OP )) holds with Proposition 2.1. As reported in Table 2.8 the bound provided by sSDP1 (QOP ) is much weaker than the one provided by sSDP4 (P OP ). Note, the objective value f0 (x) and min(sSDP1 (QOP )) improve significantly if the lower and upper bounds are chosen tighter. When chosen sufficiently tight, an accurate optimal solution can be achieved without applying SQP. The main advantage of the transformation is the reduction of the total processing time by two magnitudes. The results for the inequality constrained POP randINEQ(8,4,6,8,0) [102] with ωmax = 4 and 8 variables are given in Table 2.9. In the column for (lbd, ubd) the entries (−0.5, 0.5)⋆ and (−0.5, 0.5)⋆⋆ denote 62 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Substitution AI AII BI BII BI BI BI SQP no no no no no yes no no BC-bounds none none none none none none none none (lbd, ubd) (−∞, ∞) (-1,1) (-1,1) (-1,1) (-1,1) (-1,1) (-0.5,0.5) (-0.3,0.3) ω 4 1 1 1 1 1 1 1 n or ñ 7 138 153 115 119 115 115 115 NC 4 135 150 112 116 112 112 112 ǫsc 6e-11 1e-13 1e-13 1e-13 1e-13 7e-18 9e-14 1e-13 min(sSDPω ) -0.708 -247.50 -254.92 -299.11 -284.98 -299.11 -6.55 -1.28 f0 (x) -0.708 -0.508 -0.517 -0.567 -0.455 -0.708 -0.706 -0.708 tC 333 3 3 2 3 3 3 2 Table 2.8: Results for SDP relaxation of randEQ(7,3,5,8,0). ubd2 = 0.75 6= 0.5 and (ubd2 , ubd5 ) = (0.75, 0) 6= (0.5, 0.5), respectively. By imposing linear Branch-andCut bounds we obtain a feasible solution, and tightening lbd and ubd improves the objective value of the approximative solution. Though we did not achieve the optimal value attained by sSDP(POP), it seems reasonable to expect that successively tightening the bounds further yields a feasible solution with optimal objective value. As in the previous example tC could be reduced by two magnitudes. Substitution BI BI BI BI SQP no no no no no BC-bounds none none linear linear linear (lbd, ubd) (−∞, ∞) (−0.75, 0.75) (−0.75, 0.75) (−0.5, 0.5)⋆ (−0.5, 0.5)⋆⋆ ω 4 1 1 1 1 n or ñ 8 239 239 239 239 NC 3 234 680 680 680 ǫsc 0 -1.3 0 0 0 f0 (x) -1.5 -0.9 -0.6 -0.8 -1.2 tC 1071 14 17 17 16 Table 2.9: Results for SDP relaxation of randINEQ(8,4,6,8,0). BroydenBand Another test problem is the BroydenBand(n) problem [66]. It is an unconstrained POP of degree 6 and dimension n, and its global minimum is 0. Numerical results are given in Table 2.10. The performance of the sparse SDP relaxation for the QOP with initial bounds and without applying SQP is poor, the optimal value of the approximate solution and the lower bounds min(sSDP1 (QOP )) are far from the global optimum. Also, SQP does not succeed in detecting the global optimum if started from an arbitrary starting point. As reported in Table 2.10, SQP detects a local minimizer with objective 3, if the initial point is an SDP solution with loose bounds for the QOP. It is interesting to observe that tight lower and upper bounds, and Branch-and-Cut bounds in combination with applying SQP are crucial to obtain the global minimum by solving the sparse SDP relaxation for the QOP. In fact, when we apply substitution strategy BI imposing quadratic Branch-and-Cut bounds yields the global minimum, whereas in the case of applying BII Branch-and-Cut bounds are not necessary to obtain the global minimum. Note, the total processing time is reduced from around 1300 seconds to less than 5 seconds. POPs derived from partial differential equations An important class of large scale polynomial optimization problems of higher degree is derived from discretizing systems of partial differential equations (PDE). How to derive POPs from PDEs and how to interpret their solutions is the topic of Chapter 3 and discussed in detail there. In this section we show the 63 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP Substitution BII BII BII BI BI BI BI BI BI BII BII BII BII BII BII SQP no yes yes yes no no no yes yes yes no no no yes yes yes BC-bounds none none linear quadratic none linear quadratic none linear quadratic none linear quadratic none linear quadratic (lbd, ubd) (−∞, +∞) (-1, 1) (-1, 1) (-1, 1) (-0.75,0) (-0.75,0) (-0.75,0) (-0.75, 0) (-0.75, 0) (-0.75, 0) (-0.75, 0) (-0.75, 0) (-0.75, 0) (-0.75, 0) (-0.75, 0) (-0.75, 0) ω 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 n or ñ 20 60 60 60 80 80 80 80 80 80 60 60 60 60 60 60 NC 0 40 100 1244 60 60 60 60 140 1284 40 100 1244 40 100 1244 min(sSDPω ) -3e-7 -128 -128 -106 -611 -611 -132 -1396 -611 -611 -26 -24 -8 -26 -24 -8 f0 (x) 5e-9 3 3 3 47 47 28 3 3 6e-8 33 24 9 1e-10 1e-6 2e-7 tC 1328 4 4 4 3 4 4 4 5 5 3 3 4 5 4 5 Table 2.10: Results for SDP relaxation for BroydenBand(20). transformation procedure from POP to QOP to be a very efficient technique for this class of POPs. Many POPs derived from PDE are of degree 3, but as the number of their constraints is in the same order as the number of variables, transformation into QOPs yields SDP relaxations of vastly reduced size. Due to the structure of the higher degree monomial set of these POPs, there is an unique way to transform them into QOPs. Therefore, we examine the impact of lower, upper and Branch-and-Cut bounds and not the choice of the substitution strategy. POP pdeBifurcation(6) pdeBifurcation(10) pdeBifurcation(14) Mimura(50) Mimura(50) Mimura(100) Mimura(100) StiffDiff(6,12) ginzOrDiri(9) ginzOrNeum(11) n 36 100 196 100 100 200 200 144 162 242 ñ 72 200 392 150 150 300 300 216 324 484 ωp 2 2 2 2 3 3 2 2 2 2 size(Ap ) [2186, 17605] [16592, 139245] [454497, 3822961] [3780, 31258] [19300, 280007] [39100, 565357] [7630, 63158] [18569, 163162] [74628, 666987] [166092, 1451752] nnz(Ap ) 23801 189737 5208475 39068 354067 713767 78818 219020 906558 2504418 ωq 1 1 1 1 2 2 2 1 1 1 size(Aq ) [422, 4039] [1643, 18646] [4126, 45189] [690, 5728] [7223, 76383] [14623, 155183] [1390, 11628] [878, 6700] [4567, 49305] [8063, 96367] nnz(Aq ) 4174 19039 46000 6078 91755 186155 12328 7402 50233 97776 Table 2.11: Size of the SDP relaxation for POP and QOP, respectively. Consider the POPs in Table 2.11, where ωp and ωq the relaxation order of sSDPω for POP and QOP, respectively, to demonstrate the reduction of the size of the SDP relaxation described by the size of the matrix A in SeDuMi input format [95] and its number of nonzero entries nnz(A). Thus, the SDP relaxations for the QOPs can be solved in vastly shorter time than the one for the original POPs. The computational results of the original SDP relaxation and the SDP relaxation of the QOPs for different lower, upper and Branch-and-Cut bounds are reported in Table 2.12 for the POP pdeBifurcation(·). In this example the accuracy of the sparse SDP relaxation for the QOP is improved by tightening the upper bounds for the 64 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION components of the variable x̃ in the QOP. Also, the additional application of SQP improves the accuracy a lot. Additional Branch-and-Cut bounds seem to have no impact on the quality of the solution. The total processing time is reduced substantially under the transformation. The original sparse SDP relaxation for pdeBifurcation(14) of dimension 200 cannot be solved in SeDuMi due to a memory error, but the SDP relaxation for the corresponding QOP with tight upper bounds can be solved accurately in 100 seconds. POP pdeBifurcation(6) pdeBifurcation(6) pdeBifurcation(6) pdeBifurcation(6) pdeBifurcation(6) pdeBifurcation(6) pdeBifurcation(6) pdeBifurcation(10) pdeBifurcation(10) pdeBifurcation(10) pdeBifurcation(10) pdeBifurcation(10) pdeBifurcation(14) pdeBifurcation(14) pdeBifurcation(14) pdeBifurcation(14) pdeBifurcation(14) Substitution AI AI AI AI AI AI AI AI AI AI AI AI AI AI SQP no no no no yes no yes no no yes no yes no no yes no yes BC-bounds none none linear quadratic none none none none none none none none none none none none none ubd 0.99 0.99 0.99 0.99 0.99 0.45 0.45 0.99 0.99 0.99 0.45 0.45 0.99 0.99 0.99 0.45 0.45 ω 2 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 n or ñ 36 72 72 72 72 72 72 100 200 200 200 200 196 392 392 392 392 ǫsc 8e-11 9.6e-2 9.6e-2 9.6e-2 7.3e-9 1.5e-2 1.4e-11 3.1e-10 4.7e-2 2.7e-13 6.4e-3 1e-11 OOM 2.4e-2 7.9e-14 3.6e-3 5.2e-11 f0 (x) -9.0 -22.1 -22.1 -22.1 -9.0 -9.5 -9.0 -21.6 -56.0 -21.6 -23.2 -21.6 -103.1 -39.9 -43.1 -39.9 tC 14 2 2 2 5 1 2 2159 20 66 13 22 90 418 85 107 Table 2.12: Results for SDP relaxation for POP pdeBifurcation with lbd=0. In the case of POP Mimura(50), c.f. Table 2.13, quadratic Branch-and-Cut bounds are necessary in addition to applying SQP, in order to obtain an accurate approximate solution of the global minimizer. In the POPs in Table 2.14 it is sufficient to apply SQP starting from the solution of the sparse SDP relaxation for the QOP. For these problems tC can be reduced by up to two magnitudes. Furthermore, the original SDP relaxation for ginzOrDiri(9) and ginzOrDiri(13) is too large to be solved, whereas the SDP relaxations for the QOPs are tractable. POP Mimura(50) Mimura(50) Mimura(50) Mimura(50) Mimura(50) Mimura(50) Mimura(100) Mimura(100) Substitution AI AI AI AI - SQP no yes no yes no yes no yes BC-bounds none none none none quadratic quadratic none none ubd [11, 14] [11, 14] [11, 14] [11, 14] [11, 14] [11, 14] [11, 14] [11, 14] ω 2 2 1 1 1 1 3 3 n or ñ 100 100 150 150 150 150 200 200 ǫsc 1.8e-1 4.1e-9 6.1e-1 5.1e-3 3.3e-1 1.0e-13 4.5e-2 2.0e-11 f0 (x) -899 -701 -1067 -731 -1017 -719 -733 -712 tC 20 31 2 163 2 16 532 557 Table 2.13: Results for SDP relaxation for POP Mimura with lbd = [0, 0]. The POP ginzOrNeum(·) in Table 2.15 is another example where the global optimizer can be found in a processing time reduced by a factor 100, if the lower bounds lbd and upper bounds ubd are chosen sufficiently tight and SQP is applied. In Table 2.13 and Table 2.15 the first components of lbd and ubd correspond to the lower and upper bounds for (x1 , . . . , x n2 ), respectively, whereas the second components correspond to the lower and upper bounds for (x n2 +1 , . . . , xn ). 65 2.3. REDUCTION TECHNIQUES FOR SDP RELAXATIONS FOR LARGE SCALE POP POP ginzOrDiri(5) ginzOrDiri(5) ginzOrDiri(5) ginzOrDiri(5) ginzOrDiri(9) ginzOrDiri(9) ginzOrDiri(9) ginzOrDiri(13) ginzOrDiri(13) StiffDiff(4,8) StiffDiff(4,8) StiffDiff(6,12) StiffDiff(6,12) Substitution AI AI AI AI AI AI AI SQP no yes no yes no no yes no yes yes yes yes yes ubd 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 5 5 5 5 ω 2 2 1 1 2 1 1 2 1 2 2 1 1 n or ñ 50 50 100 100 162 324 324 338 676 64 96 144 216 ǫsc 6e-6 4e-15 3e-1 4e-11 OOM 1e-1 6e-12 OOM 7e-9 2e-11 7e-10 4e-9 8e-10 f0 (x) -25 -25 -100 -22 -324 -72 -158 -32 -32 -71 -71 tC 598 598 7 10 144 185 1992 54 4 1008 48 Table 2.14: Results for SDP relaxation for POP ginzOrDiri with lbd =0 and StiffDiff with lbd=0. POP ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(5) ginzOrNeum(11) ginzOrNeum(11) ginzOrNeum(11) Substitution AI AI AI AI AI AI SQP no yes no yes no yes no yes no no yes lbd [0, 0] [0, 0] [0, 0] [0, 0] [1, 0.5] [1, 0.5] [1, 0.5] [1, 0.5] [1, 0.5] [1, 0.5] [1, 0.5] ubd [4, 2] [4, 2] [4, 2] [4, 2] [4, 1.5] [4, 1.5] [4, 1.5] [4, 1.5] [4, 1.5] [4, 1.5] [4, 1.5] ω 2 2 1 1 2 2 1 1 2 1 1 n 50 50 100 100 50 50 100 100 242 484 484 ǫsc 2.6 2e-13 24 8e-10 1e-1 2e-13 6e-2 4e-10 OOM 4e-2 5e-11 Table 2.15: Results for SDP relaxation for POP ginzOrNeum. f0 (x) -47 -45 -100 -45 -45 -45 -57 -45 tC 448 449 9 10 582 583 6 7 -263 -207 740 748 66 CHAPTER 2. SEMIDEFINITE PROGRAMMING AND POLYNOMIAL OPTIMIZATION Chapter 3 SDP Relaxations for Solving Differential Equations 3.1 Numerical analysis of differential equations Differential equations arise in models of many problems in engineerings, physics, chemistry, biology or economics. Only the simplest differential equations allow to find solutions given by explicit formulas. In most problems involving differential equations self-contained formulas for the solutions are not available. Therefore, one is interested in finding approximations to their solutions by applying numerical methods. In general we distinguish ordinary differential equations (ODE), which are differential equations where the unknown function is a function of a single variable, and partial differential equations (PDE), which are differential equations where the unknown function is a function of multiple independent variables and the equation involves its partial derivatives. Moreover, we distinguish linear and nonlinear differential equations. A differential equations is linear if the unknown function and its derivatives appear to the power one and nonlinear otherwise. The beginning of numerical analysis of ODE dates back to 1850 when Adams formulas were proposed, which are based on polynomial interpolation in equally spaced points. The idea is, given some initial value problem with ODE u′ = f (t, u) for t > t0 and u(t0 ) = u0 , we choose a time step ∆t > 0 and consider a finite set of time values tn = t0 + n∆t, n ≥ 0. We then replace the ODE by an algebraic expression that enables us to calculate a succession of approximate values vn ≈ u(tn ), n ≥ 0, where the simplest such approximate formula dates back to Euler: vn+1 = vn + ∆t f (tn , vn ) = vn + ∆tfn , fn := f (tn , vn ). The Adams formulas are higher order generalizations of Euler’s formula that are far more efficient at generating accurate approximate solutions. For instance, the fourth-order Adams-Bashworth formula is vn+1 = vn + 1 ∆t (55fn − 59fn−1 + 37fn−2 − 9fn−3 ) . 24 (3.1) The formula (3.1) is fourth order in the sense that it will normally converge at the rate O (∆t)4 . The second important class of ODE algorithms are the Runge-Kutta methods, which were developed at the beginning of the twentieth century [50, 86]. The most commonly used member of the family of RungeKutta methods is the fourth-order Runge-Kutta method, which advances a numerical solution from time 67 68 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS step tn to tn+1 with the aid of four evaluations of the function f : a b c d vn+1 = ∆t f (tn , vn ), = ∆t f (tn + 21 ∆t, vn + 21 a), = ∆t f (tn + 12 ∆t, vn + 21 b), = ∆t f (tn + ∆t, vn + c), = vn + 16 (a + 2b + 2c + d). (3.2) Another seminal step in the numerical analysis of ODE is the concept of stability due to Dahlquist [17]. He introduced what might be called the fundamental theorem of numerical analysis: consistency + stability = convergence. This theory is based on precise definitions of these three notions. Consistency is the property that the discrete formula has locally positive order of accuracy and thus models the right ODE. Stability is the property that discretization errors introduced at one time step cannot grow unboundedly at later time steps. Convergence is the property that the numerical solution converges to the correct result as ∆t → 0. When it comes to the numerical analysis of PDEs, we distinguish three main classes of methods: Finite Difference Methods (FDM), Finite Element Methods (FEM) and Finite Volume Methods (FVM). The origin of the Finite Difference Method dates back to the paper [13] of Courant, Friedrichs and Lewy. A finite difference scheme is applied to formulate PDE problems as polynomial optimization problems in 3.2. We give an introduction to the FDM in 3.1.1. The FEM dates back to the 1960s and will be briefly introduced in 3.1.2. As in the case of ODEs, stability is a crucial issue in the numerical analysis of PDEs. The group around von Neumann and Lax discovered that some finite difference methods for PDEs were subject to catastrophic instabilities. The fundamental result linking convergence and stability of a finite difference scheme is the Lax equivalence theorem [57]. Numerical methods for finding approximate solutions to PDEs have successfully offered insights for important and difficult examples of PDE problems: The Schrödinger equation in chemistry, elasticity equations in structural mechanics, the Navier-Stokes equations in fluid dynamics, Maxwell’s equations in telecommunications, Einstein’s equations in cosmology, nonlinear wave equations in optical fibers, Black-Scholes equations in option pricing, reaction-diffusion equations in biological systems. Because such a variety of nonlinear PDE problems arises in many disciplines in science and engineering, which requires to solve largescale nonlinear algebraic systems, the numerical analysis of partial differential equations remains a very challenging field. 3.1.1 The finite difference method All discretization based methods for solving differential equations aim at algebraizing the differential equations. In the finite difference method (FDM) [19, 65, 96] the most important step to algebraize the equation is achieved via replacing differentials by finite differences. In a first step the domain of the differential equations needs to be discretized. Note, the FDM requires the geometry of the domain to be simple. We will restrict ourselves to intervals [xmin , xmax ] in the one-dimensional case, and to rectangular domains [xmin , xmax ] × [ymin , ymax ] in the two-dimensional case. We choose a discretization Nx or (Nx , Ny ), respec−ymin −xmin , hy = ymax tively and define hx = xmax Nx −1 Ny −1 , xi := xmin + (i − 1) hx , yj := ymin + (j − 1) hy , ui := u(xi ), ui,j := u(xi , yj ), (i = 1, . . . , Nx ; j = 1, . . . , Ny ), (i = 1, . . . , Nx ; j = 1, . . . , Ny ), (3.3) where u denotes the unknown function in a differential equation. There are three choices to approximate the first derivate ux at some point xi : u −u i i−1 (forward difference) hx , ui+1 −ui (3.4) ux (xi ) ≈ , (backward difference) h ui+1x−ui−1 , (central difference) 2hx 69 3.1. NUMERICAL ANALYSIS OF DIFFERENTIAL EQUATIONS An approximation to uxx is derived by successively forming (ux )x : uxx (xi ) ≈ ui+1 − 2ui + ui−1 . h2x (3.5) Choosing these approximations the question arises what errors are inherent in substituting differentials by finite differences. The following proposition provides a simple accuracy analysis. Proposition 3.1 Let u be a three times continuously differentiable function on Ω = [xmin , xmax ], with 0 ∈ Ω. It holds, a) | ui+1 −ui−1 2hx b) | ui+1 −ui hx − ux (xi ) |≤ 12 hx maxx∈Ω | uxx (x) |, c) | ui −ui−1 hx − ux (xi ) |≤ 12 hx maxx∈Ω | uxx (x) |, d) | ui+1 −2ui +ui−1 h2x − ux (xi ) |≤ 16 h2x maxx∈Ω | uxxx (x) |, − uxx (xi ) |≤ 1 2 12 hx maxx∈Ω | yxxxx (x) |. Proof: Without loss of generality set xi−1 := −h, xi := 0, xi+1 := h. Using Taylor’s theorem we expand ui−1 and ui+1 around 0 and obtain ui−1 ui+1 = ui − hx ux (xi ) + = ui + hx ux (xi ) + 1 2 2! hx uxx (xi ) 1 2 2! hx uxx (xi ) − + 1 3 3! hx uxxx (ξ1 ), 1 3 3! hx uxxx (ξ2 ), for some ξ1 ∈ [xi−1 , xi ], for some ξ2 ∈ [xi , xi+1 ]. Subtracting these two equations yields 1 1 2 (ui+1 − ui−1 ) = ux (xi ) + h [uxxx(ξ1 ) + uxxx(ξ2 )], 2hx 2 · 3! x and implies a). Now, expand ui+1 around 0 with second order remainder: ui+1 = ui + hx ux (xi ) + 1 2 h uxx (ξ1 ), for some ξ1 ∈ [xi , xi+1 ] 2! x This equation implies b): | 1 1 (ui+1 − ui ) − ux (xi ) |≤ hx max | uxx (x) | . x∈Ω hx 2 (3.6) c) is shown analogously to b). As for d), expand ui+1 and ui−1 around xi = 0 with fourth order remainder: ui−1 ui+1 = ui − hx ux (xi ) + = ui + hx ux (xi ) + 1 2 2! hx uxx (xi ) 1 2 2! hx uxx (xi ) − + 1 3 3! hx uxxx (xi ) 1 3 3! hx uxxx (xi ) + + 1 4 4! hx uxxxx (ξ1 ), 1 4 4! hx uxxxx (ξ2 ), for some ξ1 ∈ [xi−1 , xi ], for some ξ2 ∈ [xi , xi+1 ]. Addition of these two equations yields 1 ui+1 − 2ui + ui−1 − uxx (xi ) = h2x (uxxxx (ξ1 ) + uxxxx(ξ2 )) , h2x 4! which implies d). Proposition 3.1 implies, if uxxx(x) is bounded between xi−1 and xi+1 , the error replacing ux (xi ) by the central difference scheme is of order h2x , whereas the error of using a forward or a backward difference scheme is only O(hx ). In addition to discretizing the domain of a differential equation and approximating its differentials by finite difference schemes, one needs to take into account conditions for the unknown function on the boundary 70 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS ∂Ω of the domain. The most common types of boundary conditions are Dirichlet and Neumann conditions. Given a boundary point x ∈ ∂Ω, its boundary condition is called Dirichlet, if u(x) is fixed on ∂Ω, and it orthogonal to ∂Ω is fixed on ∂Ω. We consider periodic is called Neumann if the partial derivative ∂u(x) ∂n boundary conditions as a third type, which is given, if values of u at the lower and upper end of its domain are identified. For example, in the one-dimensional case with Ω = [xmin , xmax ], periodic boundary conditions are given by u(xmin ) = u(xmax ). A differential equation equipped with boundary conditions on the entire ∂Ω is called a boundary value problem. In the one-dimensional case we have Nx and in the two-dimensional case we have Nx Ny unknown variables ui and ui,j , respectively. Replacing the differential equation at each interior grid point by its finite difference discretization generates Nx − 2 and (Nx − 2)(Ny − 2) equations, respectively. By exploiting relations between the boundary and the interior variables given by Dirichlet, Neumann or periodic boundary conditions and substituting them into the equations, we can reduce the number of variables to Nx − 2 and (Nx − 2)(Ny − 2), respectively. Thus, the number of variables coincides with the number of equations. In the case of a linear differential equation, finding the Nx Ny variables ui,j is therefore equivalent to solving a system of linear equations. In the case of a nonlinear differential equation, the far more challenging problem of a system of nonlinear algebraic equations needs to be solved. As mentioned in the introduction of this section, in order to establish the convergence ui,j → u(xi , yj ) (1 ≤ i ≤ Nx , 1 ≤ j ≤ Ny ) for (Nx , Ny ) → ∞, the notions of consistency and stability are crucial. Let D(u(x, y)) = f (x, y) ∀ (x, y) ∈ Ω, B(u(x, y)) = g(x, y) ∀ (x, y) ∈ ∂Ω, (3.7) a differential equation, where D(·) and B(·) are differential operators and f, g functions on Ω. Applying a finite difference discretization to (3.7) yields the system of equations Di,j ((uk,l )k,l ) = fi,j Bi,j ((uk,l )k,l ) = gi,j ∀ (i, j) ∈ {2, . . . , Nx − 1} × {2, . . . , Ny − 1}, ∀ (i, j) ∈ {1, Nx } × {1, . . . , Ny } ∪ {1, . . . , Nx } × {1, Ny }, (3.8) where fi,j := f (xi , yj ), gi,j := g(xi , yj ), Di,j and Bi,j finite difference approximations for the operators D and B, respectively. Definition 3.1 For the finite difference discretization (3.8) of the PDE problem (3.7) we define the local truncation error ri,j as ri,j := Di,j ((u(xk , yl ))k,l ) − fi,j , where (u(xk , yl ))k,l the vector of values of the exact solution u of (3.7) at the grid points (xk , yl ), which is approximated by the solution (uk,l )k,l of (3.8). The finite difference equation (3.8) is consistent with the original equation (3.7), if ri,j → 0 as (hx , hy ) → (0, 0). Consistency is a prerequisite for ui,j to converge to u(xi , yj ) as (hx , hy ) → 0, but it is not sufficient. We need to introduce the notion of stability of a difference scheme for that purpose: Definition 3.2 A finite difference scheme Di,j ((uk,l )k,l ) = fi,j for a first order PDE is stable if there is a J ∈ N and positive numbers hx0 and hy0 such that there exists a constant C for which hx Ny X l=1 2 | uk,l | ≤ Chx Ny J X X j=1 l=1 | uj,l |2 for k ∈ {1, . . . , Nx }, 0 < hx ≤ hx0 , and 0 < hy ≤ hy0 . A finite difference scheme Di,j ((uk,l )k,l ) = fi,j for a PDE which is second order in x is stable if there is a J ∈ N and positive numbers hx0 and hy0 such that there exists a constant C for which hx Ny X l=1 | uk,l |2 ≤ (1 + k 2 ) Chx for k ∈ {1, . . . , Nx }, 0 < hx ≤ hx0 , and 0 < hy ≤ hy0 . Ny J X X j=1 l=1 | uj,l |2 3.1. NUMERICAL ANALYSIS OF DIFFERENTIAL EQUATIONS 71 For characterizing stability the following notion is useful. Definition 3.3 An explicit finite difference scheme is any scheme that can be written as uk+1,l = a finite sum of ur,s with r ≤ k. A nonexplicit scheme is called implicit. In general, it is stability of a finite difference scheme which requires some effort to show, whereas consistency is straightforward. The oldest and most famous criterion for stability is the Courant-Friedrichs-Lewy condition: Theorem 3.1 For an explicit scheme for the hyperbolic PDE defined by ux + a uy = 0 (3.9) of the form uk+1,l = αuk,l−1 +βuk,l +γuk,l+1 , a necessary condition for stability is the Courant-FriedrichsLewy (CFL) condition, hx |≤ 1. |a hy Proof: [13] Moreover, Courant, Friedrichs and Lewy derived the general result: Theorem 3.2 There are no explicit, unconditionally stable, consitent finite difference schemes for hyperbolic systems (3.9) of partial differential equations. There is no general negative result like Theorem 3.2 for implicit schemes. Thus, from a stability point of view, it is advisable to choose central or backward difference approximations for the first order derivates. The result which links consistency, stability and convergence is the Lax-Richtmyer Equivalence Theorem: Theorem 3.3 A consistent finite difference scheme for a linear partial differential equation of first or second order for which the initial value problem is well-posed is convergent if and only if it is stable. Proof: [96] Thus, convergence of a finite difference scheme is usually proven by showing stability. There is no general convergence result for finite difference schemes of nonlinear partial differential equations. However, for certain classes of PDEs convergence of the FDM has been shown: Theorem 3.4 Let a parabolic PDE problem be given by a(x, y) uxx + d(x, y) ux − uy + f (y, u) = 0 ux (0, y) = ux (1, y) =0 u(x, 0) = u0 (x) ∀ (x, y) ∈ (0, 1) × (0, T ), ∀ y ∈ (0, T ), ∀ x ∈ (0, 1). If a, ax , ay , b, bx and by continuous in [0, 1] × [0, T ], there exists an a0 s.t. a ≥ a0 in [0, 1] × [0, T ], f , fu and fuu continuous in [0, T ] × R, there exists M0 ∈ R s.t. ∂f /∂u ≤ M0 in [0, T ] × R, some technical conditions hold and the solution u of the PDE problem is smooth, then (ui,j (Nx , Ny ))i,j converges uniformly to u in [0, 1] × [0, T ] as (Nx , Ny ) → ∞. Proof: Theorem 2.1 in [97]. Note, Theorem 3.4 can be extended to the case of arbitrary rectangular domains and arbitrary Dirichlet and Neumann conditions at xmin and xmax . To proof convergence of classes of elliptic or hyperbolic PDE problems is far more difficult and no result for a broad class of problems like Theorem 3.4 has been found in those cases. Some simple accuracy analysis of finite difference approximations was provided in Proposition 3.1, where we have seen that the accuracy of central differences is better than the one of forward and backward 72 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS differences. On the other hand, it is a well known phenomenon that central difference approximations for the first derivatives may cause oscillations in the numerical solution of a boundary value problem [65]. These oscillations do not occur under forward or backward difference schemes. However, we have also discussed that a forward finite difference scheme may not be stable, whereas implicit finite difference schemes are unconditionally stable for many classes of PDE problems. Thus, when choosing a difference scheme we have to consider accuracy, stability and how to avoid oscillations. It depends on the PDE problem which finite difference scheme is ”the best” one. Therefore, numerous difference schemes have been proposed, which may use different difference approximations on certain segments of the domain, may use different difference approximations in the different dimensions, may depend of the type of the PDE or may depend of the type of boundary conditions. Detailed analysis of these issues for a variety of finite difference schemes for different classes of PDE problems is discussed in detail in [96]. To summarize, when solving a linear or nonlinear ODE or PDE problem with the FDM, the three main tasks are 1) to choose a finite difference scheme whose solutions are accurate approximations for solutions of the PDE problem, 2) to show convergence of the difference scheme, and 3) to solve the (nonlinear) algebraic system of equations (3.8). To solve a system of nonlinear equations is a very hard problem in general. As mentioned in 2.1 solving a system of nonconvex polynomial equations is NP-hard. A standard method to solve the system of algebraic equations resulting from a finite difference approximation of a nonlinear PDE is to solve the system of equations corresponding to the linear part of the PDE, and to take the solution of the linear system as a starting point for gradient type methods, Newton’s method or other iterative methods applied to the nonlinear system or to a system where the nonlinear part is successively increased. The eventual success of such a continuation type method is base on the assumption that the solution of nonlinear system does not change much if the nonlinear part is increased by a small factor. In this thesis we attempt problem 3) by reformulating (3.8) as a polynomial optimization problem (POP) and solving this POP by sparse semidefinite programming relaxation techniques introduced in Chapter 2. One of the main advantages of this approach is the fact, that we do not require any initial guess and the nonlinearity of the scheme is taken into account directly. This is presented in detail in 3.2. 3.1.2 The finite element method and other numerical solvers The finite element method In addition to the finite difference method which our technique to be proposed in 3.2 is based on, we briefly introduce alternative methods for solving a PDE problem numerically. The most important one is the finite element method (FEM). The FEM is a very active field of research and there is exhaustive literature on it. For example, see [11, 59, 106] for detailed introductions and more advanced studies. The FEM is based on the idea to approximate a solution u of a PDE problem by a function ũ which is an element of a finitedimensional subspace of the function space u belongs to. I.e., the FEM can be understood as a method which discretizes the space we are searching for solutions of a PDE problem, whereas in the FDM the PDE is discretized. The origins of the FEM date back to works of Rayleigh [83], Ritz [84] and Galerkin [23] at the beginning of the 20th century. The FEM in its modern formulation is due to Courant [16] and Turner, Clough, Martin and Topp [99], among others. Typically, a FEM approach to solve a PDE consists of the following steps. First of all one is looking for a solution u for a PDE problem defined on a domain Ω in a certain function space. The most common function space to this end is the Sobolov space H0s (Ω) ⊂ Ls (Ω). Given that function space, one replaces the PDE problem by a weak, variational formulation where the test functions are elements of the same space H0s (Ω). In a second step known as meshing the domain Ω is partitioned into a finite number of subdomains of simple geometry, which are called elements. We denote such a partition by T . In the one-dimensional case intervals, in the two-dimensional case triangles, and in the three-dimensional case tetrahedra are a common choice for the elements. These elements define a mesh for Ω with nd nodes. Then, in a third step H0s (Ω) is approximated by the nd -dimensional subspace which is spanned by functions f1 , . . . , fnd . A common choice for this basis are for instance piece-wise linear functions fi which equal one at the node i and are 0 at all other nodes. The larger nd , i.e., the finer we choose the mesh, the better approximates span(f1 , . . . , fnd ) the space H0s (Ω). When replacingP H0s (Ω) by nd span(f1 , . . . , fnd ) in the weak formulation of the PDE problem and approximating u by ũ = i=1 di fi , 73 3.1. NUMERICAL ANALYSIS OF DIFFERENTIAL EQUATIONS ones obtains a finite number of equations in the unknowns d1 , . . . , dnd . Solving this system of equations yields the numerical approximation ũ for a solution of the original PDE problem. Finally, as for the Finite Difference Method, convergence of a finite element discretization needs to be shown, i.e., ones has to show that span(f1 , . . . , fnd ) converges to a subspace dense in H0s (Ω) if the number nd of nodes in the mesh and the corresponding number of basis functions goes to infinity. One of the biggest advantages of the FEM is its sound mathematical basis. As the PDE is formulated as a variational problem one has lots of powerful tools from functional analysis at hand to proof convergence of a finite element discretization. Let us demonstrate the outlined procedure on the following, simple ODE problem. Find u ∈ H02 (Ω) such that u′′ (x) u(0) = g(x) = u(1) = 0. on x ∈ Ω := [0, 1], (3.10) Its weak formulation is given by Z 0 1 u′′ (x) φ(x)dx = Z 0 1 g(x) φ(x)dx ∀ φ ∈ H02 (Ω), (3.11) a partition is given by T h := {xi := i h | i ∈ {0, . . . , h1 }} for any h > 0 with h1 ∈ N. With this partition the nodes of the mesh are given by x1 , . . . , x h1 −1 , i.e., nd = h1 − 1. We define the finite dimensional subspace Vh = span(f1 , . . . , fnd ) of H02 (Ω) via the basis Pnd functions fi : Ω → R with fi (xj ) := δi,j and fi linear on each interval (xj , xj+1 ). Then, let ũ := i=1 di fi satisfy the finite-dimensional relaxation of the weak formulation (3.11): R 1 Pnd R1 ( i=1 di f ′′ i (x)) φ(x) dx = 0 g(x)φ(x) dx 0 R R Pnd 1 ′′ 1 ⇔ fj (x) dx = 0 g(x)fj (x) dx i=1 di 0 f i (x) R 1 dj−1 −2dj +dj+1 ⇔ = 0 g(x)fj (x) dx h ∀ φ ∈ Vh ∀ j ∈ {1, . . . , nd } ∀ j ∈ {1, . . . , nd } (3.12) which is a system of linear equations in d1 , . . . , dnd . Solving it provides a numerical approximation ũ to a weak solution u of ODE (3.10). If we moreover assume g(x) = g constant on [0, 1], we obtain the system of equations dj−1 − 2dj + dj+1 = g ∀ j ∈ {1, . . . , nd }. (3.13) h2 With the definition of the basis functions fi it is clear, in this example holds di = f (xi ) for all i ∈ {1, . . . , nd }. Thus, (3.13) is actually identical to the system of equations we obtain when approximating (3.10) by a finite difference scheme. However, this connection with finite difference methods does not hold in general. The Finite Element Method provides the user with a great deal of freedom, such as how to choose the finite dimensional subspace to approximate H0s (Ω) or the mesh for Ω, and for most other finite element discretizations there is no equivalent finite difference scheme. When comparing the FEM to the FDM, we already mentioned its sound basis from functional analysis as one main advantage. Another one is, it is highly flexible for different domains of PDE problems: complicated geometries can be dealt with easily, whereas the FDM is restricted to relatively simple geometries. On the other hand, the FDM is far easier to implement than the FEM for many PDE problems arising in applications. However, in both methods one of the greatest challenges is to solve a system of nonlinear algebraic equations, in the case they are applied to a system of nonlinear differential equations. In general it is highly dependent on the PDE problem to be solved numerically which method, FDM or FEM, provides a better approximation to the continuous world when choosing a similar discretization. Other numerical solvers Beside the finite difference method and the finite element method another important class of methods to solve differential equations numerically is the finite volume method (FVM). Similar to the FDM, values are calculated at discrete points on a mesh. In the FVM these values are derived from calculating volume integrals over small volumes around the node points of the mesh. The FVM applies the divergence theorem 74 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS to convert volume into surface integrals, and it exploits, that a flux entering a given, small volume is equal to that leaving the volume. Each integration over a volume results in an algebraic equation. Thus, as in the FDM and FEM a system of algebraic equations needs to be solved to obtain a numerical solution. The FVM is popular for solving hyperbolic PDE problem, in particular in computational fluid dynamics. For a detailed introduction, see [58]. The Spectral method [27] is a class of techniques involving the use of the Fast Fourier Transform. It is suitable for PDE problems with very smooth solutions and provides highly accurate approximations for these problems. It is based on replacing the unknown function in the PDE by its Fourier series to get a system of ODEs in the time-dependent coefficients of the Fourier series. The spectral method and the FEM share the idea to approximate the solution of a PDE by a linear combination of basic functions. The difference being these basic functions in the Spectral method are nonzero over the entire domain, whereas the basic functions of the FEM are nonzero on small subdomains only. For this reason, the spectral method can be understood as a global approximation approach, whereas the FEM constitutes a local approximation approach. There are numerous other methods, such as multigrid methods, domain decomposition methods, level-set methods or meshfree methods. As PDE problems arise from very different settings and applications, it is highly dependent on a particular PDE which numerical method is most suitable for providing an accurate approximation. In all numerical methods a potentially hard system of algebraic equations need to be solved. In the next section we will attempt this problem for the FDM by sparse semidefinite programming relaxation and polynomial optimization techniques. Remark 3.1 We have seen, in each discretization-based method for nonlinear PDEs a system of algebraic equations needs to be solved. The classical tool for a system of nonlinear equations is Newton’s method, which converges locally quadratic. However, Newton’s method requires a starting point close to a solution of the system, in order to converge. For difficult nonlinear problems it may be a very challenging problem to find a good initial guess. There are various techniques to find a good initial guess for Newton’s method or other locally fast convergent techniques. One is to apply gradient methods or other first order techniques to find a rough approximation of a solution. Another one is to apply some homotopy-like continuation method, where the nonlinear problem is linearized and a solution of the linear problem is taken as initial point for a problem where the weight of the nonlinear part is increased incrementally. Finally, in many problems partial information of the solution, which may be obtained by numerical simulation, is utilized to get a sufficiently close initial guess. We will show in the following that SDP relaxation for polynomial problems are very useful to find a good initial guess for locally fast convergent methods. 3.2 3.2.1 Differential equations and the SDPR method Transforming a differential equation into a sparse POP In the previous section we gave an introduction to the numerical analysis of differential equations. In Chapter 2 we introduced polynomial optimization problems and their semidefinite programming (SDP) relaxations. In this section we will see how to transform a problem involving differential equations into a polynomial optimization problem (POP), in order to apply SDP relaxations to solve these differential equations numerically. For discretizing a differential equation we choose the Finite Difference Method (FDM). The FDM has the advantage of being easy to implement. Moreover, applying the FDM to a differential equation yields a sparse POP, as we will show in this section. In the following we will restrict ourselves to discussing the two-dimensional case. However, the procedures for differential equations with domains of different dimension are derived analogously. Recall, a general differential equation is given by D(u(x, y)) = f (x, y) ∀ (x, y) ∈ Ω, B(u(x, y)) = g(x, y) ∀ (x, y) ∈ ∂Ω, (3.14) 75 3.2. DIFFERENTIAL EQUATIONS AND THE SDPR METHOD where D(·) and B(·) are differential operators and f, g functions on Ω := [xmin , xmax ]×[ymin, ymax ]. Applying the FDM yields the system of equations Di,j ((uk,l )k,l ) = fi,j Bi,j ((uk,l )k,l ) = gi,j ∀ (i, j) ∈ {2, . . . , Nx − 1} × {2, . . . , Ny − 1}, ∀ (i, j) ∈ {1, Nx } × {1, . . . , Ny } ∪ {1, . . . , Nx } × {1, Ny }, (3.15) where fi,j := f (xi , yj ), gi,j := g(xi , yj ), Di,j and Bi,j finite difference approximations for the operators D and B, respectively. We assume the operators D and B are both polynomial in u(·, ·) and its derivates, which implies (3.15) is a system of polynomial equations. Note, we can reduce the dimension of this system of equations by exploiting the boundary conditions. In their most basic form Dirichlet, Neumann and periodic boundary conditons at xmin are given by u1,j = g1,j , u1,j = g1,j hx + u2,j and u1,j = uNx ,j , (3.16) respectively. If we substitute the ui,j corresponding to the boundary grid points in (3.15) by the terms given in (3.16), the number of variables in each direction is reduced by 2 in the case of Dirichlet and Neumann boundary conditions, or by 1 in the case of periodic boundary conditions. Thus, (3.15) is reduced to a system of n equations in n variables, D̂i,j ((uk,l )k,l ) − fˆi,j = 0 ∀ (i, j) ∈ {1, . . . , N̂x } × {1, . . . , N̂y }, (3.17) where n := N̂x N̂y . For instance under Neumann condition in x direction and periodic condition in y direction, n is given by n = N̂x N̂y = (Nx − 2)(Ny − 1). For simplicity of notation we will denote a solution of the original PDE problem (3.14) as u(·, ·) and a solution (uk,l )k,l of (3.17) as the vector u ∈ RN̂x N̂y , with u = (u1,1 , . . . , u1,N̂y , u2,1 , . . . , uN̂x ,N̂y ). As (3.15) is polynomial in the variable u, so is (3.17). We attempt to solve this system of equations by transforming it into a POP of type, min p(u) s.t. gi (u) ≥ 0 ∀i = 1, . . . , m, hj (u) = 0 ∀j = 1, . . . , k. (3.18) Note, (3.18) is a special case of (2.1), as an equality constraint h(x) = 0 is equivalent to the pair of inequality constraints h(x) ≥ 0 and −h(x) ≥ 0. Given a PDE problem (3.14), we take (3.17) derived from it as system of equality constraints for an optimization problem. Moreover, we choose lower and upper bounds of type (2.25), lbdi,j ≤ ui,j ≤ ubdi,j ∀ (i, j) ∈ {1, . . . , N̂x } × {1, . . . , N̂y }. (3.19) Choosing an objective function To derive a POP, it remains to choose an objective function F that is polynomial in u as well. The choice of an appropriate objective function is dependent on the PDE problem we are aiming to solve. In case there is at most one solution of the PDE problem, we are interested in the feasibility of the POP we construct. Thus any objective is a priori acceptable for that purpose. However, the accuracy of obtained solutions may depend on the particular objective function. In the case the solution of the PDE problem is not unique, the choice of the objective function determines the particular solution to be found. The objective function may correspond to a physical quantity a solution needs to optimize. For instance, in problems in fluid dynamics one is often interested in finding a solution of minimal kinetic energy. For such a problem we obtain the objective function by discretizing the kinetic energy function. A large class of PDEs which occur in many applications can be written as Euler-Lagrange equations. A typical case is a stable state equation of reaction-diffusion type. In this case, a canonical choice is a discretization of the corresponding energy integral as in 3.3.1. Another case is optimal control: A finite difference discretization of state and control constraints yields the feasible set and a discretization of the optimal value function yields the objective. We discuss numerical examples for optimal control problem in 3.3.6. 76 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS Additional polynomial constraints We mentioned above, it is crucial to add lower and upper bounds (3.19) for each ui,j when constructing a POP from a differential equation. When choosing these bounds care has to be taken. A choice of lbd and ubd which is too tight may exclude solutions of (3.17) from the feasible set of the POP, while a choice which is too loose may cause inaccurate results. In addition to those constraints we may impose further inequality or equality constraints gl (u) ≥ 0 and hj (u) = 0, (3.20) respectively, with gl , hj ∈ R[u]. Constraints (3.20) can be understood as restrictions for the admissible space of functions we search for solutions of a differential equation. One possibility to obtain such bounds is derived by constraining the partial derivatives. We call bounds of this type variation bounds. For the derivative in x−direction they are given by | ∂u(xi , yj ) |≤ M ∂x ∀i ∈ {2, . . . , Nx − 1}, ∀j ∈ {2, . . . , Ny − 1}. (3.21) Expression (3.21) can be transformed into polynomial constraints easily. Another possibility is to impose bounds in the spirit of [22] like (2.80), (2.81), (2.82) and (2.83) introduced in 2.3. Add ≤ ubdk,l ui.j + lbdi,j uk,l − lbdi,j ubdk,l ≤ lbdk,l ui,j + ubdi,j uk,l − ubdi,j lbdk,l . us,t us,t (3.22) for each constraint us,t = ui,j uk,l in the POP, and for each constraint us,t = u2i,j we add us,t ≤ (ubdi,j + lbdi,j ) ui,j − lbdi,j ubdi,j . (3.23) If there is no quadratic constraint of this type for (i, j, k, l) or (i, j), respectively, we may add the quadratic constraints ui,j uk,l ≤ ubdk,l ui,j + lbdi,j uk,l − lbdi,j ubdk,l (3.24) ui,j uk,l ≤ lbdk,l ui,j + ubdi,j uk,l − ubdi,j lbdk,l for (i, j) 6= (k, l) and the constraint u2i,j ≤ (ubdi,j + lbdi,j ) ui,j − lbdi,j ubdi,j (3.25) for (i, j) = (k, l). Note, by the construction of the constraints (3.22) - (3.25), they shrink the feasible set of an SDP relaxation for a POP, but they do not change the feasible set of the POP. I.e., they may be added to improve the numerical accuracy of SDP relaxations for solving the POP, but they have no impact on the space of functions we are searching for discrete approximations to a solution for a differential equation. A POP derived from a differential equation If we take together all constraints and the chosen objective function, we obtain the following POP: min s.t. F (u) Di,j (u) gl (u) hk (u) lbdi,j = fi,j ≥0 =0 ≤ ui,j ≤ ubdi,j ∀ (i, j) ∈ {1, . . . , N̂x } × {1, . . . , N̂y }, ∀ l ∈ {1, . . . , s}, ∀ k ∈ {1, . . . , m}, ∀ (i, j) ∈ {1, . . . , N̂x } × {1, . . . , N̂y }. (3.26) Every feasible solution u of (3.26) is a solution of the finite difference scheme for the PDE problem (3.14). Let demonstrate how to derive (3.26) for an example. Example 3.1 Consider the nonlinear elliptic PDE uxx (x, y) + uyy (x, y) + λu(x, y) 1 − u(x, y)2 =0 u(x, y) = 0 0 ≤ u(x, y) ≤ 1 ∀ (x, y) ∈ [0, 1]2 , ∀ (x, y) ∈ ∂[0, 1]2 , ∀ (x, y) ∈ [0, 1]2 , (3.27) 3.2. DIFFERENTIAL EQUATIONS AND THE SDPR METHOD 77 where the parameter λP is set to λ = 22. We apply the standard finite difference discretization for N̂ = N̂x = N̂y , choose F (u) = − 1≤i,j≤N̂ ui,j as objective function, and obtain the POP min s.t. − P ui,j = 0 ∀ (i, j) ∈ {1, . . . , N̂ }2 , (ui+1,j + ui,j+1 + ui,j−1 + ui−1,j − 4ui,j ) + 22ui,j 1 − u2i,j 0 ≤ ui,j ≤ 1 ∀ (i, j) ∈ {1, . . . , N̂ }2 , (3.28) where u0,k = uk,0 = uN̂+1,k = uk,N̂+1 = 0 for all k ∈ {1, . . . , N̂ }. The choice for F is motivated by the fact, that (3.27) is known to have one strictly positive and the trivial solution. The optimal solution of (3.28) is a discrete approximation to the strictly positive solution of (3.27). 1 h2x 1≤i,j≤N̂ Correlative sparsity In 3.2.2 we will introduce a method to solve (3.26) by sparse SDP relaxations. In order to apply this method efficiently, we need to show (3.26) satisfies a structured sparsity pattern. To show correlative sparsity is straight-forward: Proposition 3.2 Let two differential operators D1 and D2 be given by D1 (u) := a(u)uxx + c(u)uyy + d(u)ux + e(u)uz + f˜(u) and D2 (u) := a(u)uxx + b(u)uxy + c(u)uyy + d(u)ux + e(u)uz + f˜(u), where a(·), . . . , f˜(·) polynomial in the function u(·). Let N̂ = N̂x = N̂y and n := N̂ 2 Let F be linear function in u. Let nz (R) denote the number of nonzero entries in the CSP matrix R of the POP (3.26). Then, nz (R) ≤ 13n, if (3.26) derived from (3.15) with D := D1 , and nz (R) ≤ 25n, if (3.26) derived from (3.15) with D := D2 , for any choice of B and f . This implies, (3.26) is correlatively sparse in both cases. Proof: As F linear, the objective function does not cause any nonzero entries in R by Definition 2.5. Due to the finite difference discretization at most 12 unknown uk,l can occur in some equality constraint with a particular unknown ui,j for D = D1 , as pictured in Figure 3.1. Hence the maximum number of nonzero elements in the row of R corresponding to ui,j is 13, which implies nz (R) ≤ 13n. With the same argument holds nz (R) ≤ 25n for D = D2 ; see Figure 3.1. These bounds are tight; they are attained in the case of periodic conditions for x and y. Let R′ denote the n × n matrix corresponding to the graph G(N, E ′ ), which is a chordal extension of CSP graph G(N, E). For the computational efficiency it is also useful to know whether R′ is sparse or not. nz (R′ ) depends on the employed ordering method P for R, which is used to avoid fill-ins in the symbolic sparse Cholesky factorization LLT of the ordered matrix P RP T . R′ is constructed as R′ = L + LT . We examine two different methods of ordering R, the symmetric minimum degree (SMD) ordering and reverse Cuthill-McKee (RCM) ordering. See [24] for details about these orderings. 78 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS ui-1,j+1 ui+1,j+1 ui-2,j ui,j ui,j Figure 3.1: uk,l involved in some constraint with ui,j for D = D1 (left) and D = D2 (right). 300 600 250 500 200 400 n (R’)/ n 150 nz(R’)/ n nz(R’)/ n z n (R’)/ n z 300 100 200 50 100 0 0 2 4 6 8 n Figure 3.2: 10 12 14 0 16 4 x 10 nz (R′ ) n 0 2 4 6 n 8 10 12 4 x 10 for SMD (left) and RCM (right) ordering if D = D1 . We conduct some numerical experiments, in order to estimate the behavior of nz (R′ ). Figure 3.3 shows ′ ) obtained by the SMD and RCM examples of R′ after SMD and RCM ordering, and Figure 3.2 shows nz (R n orderings for the n × n-matrix R, respectively, for D = D1 and Dirichlet or Neumann condition in x and ′ ′ ) ) periodic condition in y. For n ∈ [100, 160000] it holds nz (R ≤ 300 for SMD ordering and nz (R ≤ 600 for n n ′ ) RCM ordering, respectively. The behavior of nz (R may suggest nz (R′ ) = O(n) for both ordering methods. n Hence we expect the sparse SDP relaxations to be efficient for solving (3.26) in numerical experiments. However, since the constants 300 and 600 are large, we can not always expect a quick solution of the sparse SDP relaxation. Domain-space sparsity In 2.2 we introduced the concept of domain-space and range-space sparsity of an optimization problem with matrix variables. Moreover, in 2.2.7 we constructed some linear SDP relaxations for quadratic SDP exploiting this sparsity. A QOP is a special case of a quadratic SDP. As all constraints in a QOP are scalar, it does not satisfy a range-space sparsity pattern. Thus, if we transform (3.26) into a QOP, we can apply the relaxations (a) or (b) from 2.2.7 to find approximate solutions for (3.26). In order to apply these relaxations efficiently, the question arises, whether QOPs derived from (3.26) satisfy domain-space sparsity. For certain classes of PDE problems, we obtain the following sparsity results. 79 3.2. DIFFERENTIAL EQUATIONS AND THE SDPR METHOD 0 0 0 50 50 50 100 100 100 150 150 150 200 200 200 250 250 250 300 300 300 350 350 350 400 400 0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400 400 0 50 100 150 200 250 nz = 27344 300 350 400 Figure 3.3: R (left), R′ obtained by SMD (center) and RCM (right) orderings for D = D1 with n = 400. Example 3.2 Given a rectangular domain Ω, f : Ω → R and differential operator B(·). Define the operators D1 (u) := auxx + cuyy + dux + euy + gu + hu2 , D2 (u) := auxx + cuyy + dux + euy + gu + hu3 , with a, c, d, e, g, h : Ω → R. Moreover, choose F linear in u when constructing (3.26) for a discretization n = N̂x N̂y . In the case D = D1 (3.26) is a quadratic SDP, in the case D = D2 we need to apply the method from 2.3 to (3.26) to transform it into a quadratic SDP. In fact, for the example with D = D2 there is an unique way to transform the POP into an QOP by defining n variables vi,j := u2i,j . Then, it is easy to see that the domain-space sparsity patterns of the quadratic SDP corresponding to (3.26) for D = D1 and D = D2 , respectively, is given by Figure 3.4. 0 0 0 5 100 100 10 200 200 300 300 400 400 500 500 600 600 15 20 25 30 35 700 700 40 800 0 5 10 15 20 25 nz = 121 30 35 40 0 100 200 300 400 nz = 3201 500 600 700 800 800 0 100 200 300 400 nz = 3201 500 600 700 800 Figure 3.4: d-space sparsity pattern of (3.26) for D = D1 (lef t), and D = D2 before (center) and after (right) reordering of rows and columns. Note, for the two cases in Example 3.2 the number of nonzero entries in every row but the first and last one of the domain-space sparsity pattern matrix is less or equal to two and three, respectively. This is considerably smaller than the upper bound 13 for the number of nonzero entries in each row of the correlative sparsity pattern matrix provided by Proposition 3.2. Likewise the size of an average maximal clique of the chordal extension of the domain-space sparsity pattern graph is far smaller than the one of the chordal extension of the correlative sparsity pattern graph. Thus, the primal SDP relaxation (b) from 2.2.7 for an QOP derived from a PDE of one of the two classes in Example 3.2 is far smaller than the sparse SDP relaxation (2.18) of relaxation order ω = 1 for the same QOP, and can be solved for much finer discretizations. However, in the primal SDP relaxation (b) there is no relaxation order we can increase, 80 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS and there is no general result how well the primal SDP relaxation approximates a QOP. The sparse SDP relaxations (2.18) provide a sequence of SDPs whose minima converge to the optimum of a QOP. In fact, their approximation accuracy is improving monotonously for increasing order ω. Therefore, as we will see in numerical examples in 3.3, the primal SDP relaxation (b) is only useful for QOPs, where the solution of the primal SDP relaxation is a good approximation of an optimal solution of the QOP. 3.2.2 The SDPR method In this section we introduce the method to solve the POP (3.26) derived from a PDE problem (3.14) to obtain discrete approximations to solutions of (3.14). In order to derive (3.26) from a PDE problem, we need to choose a discretization (Nx , Ny ), the bounds lbd and ubd, the objective F , if not given by the PDE problem, and possibly additional polynomial constraints gl and hk . If (3.26) is a POP of degree three or larger, we can either apply the sparse SDP relaxations (2.18) for some relaxation order ω, or we apply one of the heuristics AI, AII, BI, BII from 2.3.1 to transform (3.26) into a QOP. To the QOP we apply either the sparse SDP relaxations (2.18) with relaxation order ω = 1 or the primal SDP relaxation (b) exploiting domain-space sparsity from 2.2.7. Solving the SDP relaxations for the POP or the QOP, we obtain a first approximation û to an optimal solution of (3.26). The solution û can be used as an initial guess for locally fast convergent methods. One possibility is to apply sequential quadratic programming (SQP) [8] to (3.26), another one is to apply Newton’s method for nonlinear systems [77] to (3.17), both starting from û, in order to obtain a more accurate discrete approximation u to a solution of the PDE problem (3.14). This procedure is called the semidefinite programming relaxation (SDPR) method for solving a PDE problem of type (3.14), it is summarized in the following chart: Method 3.1 (SDPR method) I. Given a PDE problem (3.14), choose (Nx , Ny ), lbd, ubd, F , gl and hk to derive (3.26). II. If POP (3.26) of degree three or larger, we may apply AI, AII, BI or BII to transform it into a QOP. Then, apply sSDP1 (2.18) or relaxation (b) from 2.2.7 to this QOP. Denote the first n components of the solution vector of the applied SDP relaxation as û. III. If POP (3.26) of degree three or larger which has not been transformed into a QOP, choose relaxation order ω ≥ ωmax , apply sSDPω (2.18) to that POP and obtain its solution û. IV. Apply Newton’s method to (3.17) or SQP to (3.26), both with initial guess û, and obtain u as an approximate to an optimal solution of (3.26) and as a discrete approximation for a solution of the PDE problem (3.14). Recall, when applying the SDP relaxation (b) from 2.2.7 to a QOP, we impose the additional constraints (2.84) as explained in Remark 2.9. For locally fast convergent methods we consider SQP and Newton’s method, which we describe briefly in the following. However, we are by no means restricted to these two for choosing an iterative method which is fast convergent towards an highly accurate discrete approximation of a solution to (3.17) when starting from a guess close to the accurate approximation. Newton’s method The discretized PDE (3.17) is a special case of the problem r(x) = 0, where r : Rn → Rn , r(x) = [r1 (x), . . . , rn (x)]T and ri : Rn → R are smooth functions for all i ∈ {1, . . . , n}. The functions ri may be nonlinear in x. The basic form of Newton’s method for solving nonlinear equations 3.2. DIFFERENTIAL EQUATIONS AND THE SDPR METHOD 81 is given by NEWTON Choose x0 ; for k = 0, 1, 2, . . . Calculate a solution pk to the Newton equations J(xk )pk = −r(xk ); xk+1 = xk + pk ; end (for) This algorithm is motivated by the multidimensional Taylor’s theorem. Theorem 3.5 Suppose that r : Rn → Rn is continuously differentiable in some convex open set D and that x and x + p are vectors in D. We then have that Z 1 J(x + tp)pdt. r(x + p) = r(x) + 0 J(x + tp) is the Jacobian of r at x + tp, it is defined as J(x) = ∂rj ∂xi i,j=1,...,n ∇r1 (x)T .. = . . T ∇rn (x) We define a linear model Mk (p) of r(xk + p) given in Theorem 3.5, i.e., we approximate the second term on the right-hand-side by J(x)p, and write Mk (p) = r(xk ) + J(xk )p. The vector pk = −J(xk )−1 r(xk ) satisfies Mk (pk ) = 0. It is equivalent to the solution pk of the Newton equations in the NEWTON algorithm. As shown in [77], if x0 close to a nondegenerate root x⋆ and r continuously differentiable, then the sequence (xk )k of Newton’s method converges superlinearly to x⋆ . If r furthermore Lipschitz continuously differentiable, the convergence is quadratic. Sequential Quadratic Programming The POP (3.26) is a special case of the nonlinear programming problem min f (x) s.t. h(x) = 0, g(x) ≤ 0, (3.29) where f : Rn → R, h : Rn → Rm and g : Rn → Rp three times continuously differentiable. The basic idea of SQP is to model (3.29) at a given approximate solution xk by a quadratic program, and to use the solution to this subproblem to construct a better approximation xk+1 to the solution of (3.29). This method can be viewed as the natural extension of Newton’s method to the constrained optimization setting. It shares with Newton’s method the property of rapid convergence, when the iterates are close to the solution, and possible erratic behavior, when the iterates are far from the solution. SQP has two key features: First, it is not a feasible point method, its iterates xk+1 do not need to be feasible for (3.29). Second, in each iteration of an SQP approach a quadratic program is to be solved, which is not too demanding since highly efficient procedures for quadratic programs, i.e., programs with quadratic objective and linear constraints, exist. Let the Lagrange function of (3.29) be given by L(x, u, v) and a quadratic subproblem by min s.t. ∇f (xk )T (x − xk ) + 21 (x − xk )T Bk (x − xk ) ∇h(xk )T (x − xk ) + h(xk ) = 0, ∇g(xk )T (x − xk ) + g(xk ) ≤ 0, (3.30) 82 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS where Bk an approximation of the Hessian of the Lagrangian function at the iterate (xk , uk , vk ). Then, an SQP approach in its most basic form is given by the algorithm SQP Choose (x0 , u0 , v0 ), B0 , and a merit function φ for k = 0, 1, 2, . . . Form and solve (3.30) to obtain its optimal solution (x⋆ , u⋆ , v ⋆ ). Choose step length α so that φ(xk + α(x⋆ − xk )) < φ(xk ). xk+1 = xk + α(x⋆ − xk ), Set uk+1 = uk + α(u⋆ − uk ), vk+1 = vk + α(v ⋆ − vk ). Stop if (xk , uk , vk ) satisfies some convergence criterion. Compute Bk+1 from (xk , uk , vk ). end (for) A merit function is a function whose reduction implies progress towards the global optimum of problem (3.29). For more details about SQP see [8]. Grid-refining method In order to guarantee, that a discretized PDE problem (3.15) is a good approximation of (3.14), i.e., its solutions are good discrete approximations of continuous functions u(·, ·), it is necessary to choose fine grid discretizations (Nx , Ny ). However, a fine grid discretization results in a large scale POP (3.26). Even when exploiting correlative or domain-space sparsity, transforming it into a QOP and imposing tight lower and upper bounds, the SDP relaxation, which needs to be solved, is often computationally demanding - in particular in the cases where we need to choose a high relaxation order to obtain an accurate approximation to an optimal solution of (3.26). Thus, for many difficult PDE problems the SDP relaxations resulting from fine grid discretizations are intractable for current SDP solvers. To overcome this problem, we consider a grid-refining method. In our grid-refining method a solution obtained by applying the SDPR method to (3.14) for a coarse grid discretization in a first step is extended stepwise to finer and finer grids by subsequently interpolating coarse grid solutions and applying the SDPR method or locally convergent methods. This method is described by the following scheme: Step 1 - Initialize Step 2 - Extend Step 3a Step 3b Iterate Apply SDPR method with Nx (1), Ny (1), F1 , lbd1 , ubd1 , g(1), h(1), ω1 . obtain u1 Nx (k) = 2 Nx (k − 1) − 1 or Ny (k) = 2 Ny (k − 1) − 1 ⋆ Interpolation of uk−1 obtain uk−1 . Apply SDPR method with , Nx (k), Ny (k), Fk , lbdk , ubdk , g(k), h(k), ωk . Apply Newton’s method or SQP. obtain uk Step 2 and Step 3 Step 1 - SDPR method: Choose an objective function F1 (u), a discretization grid size (Nx (1), Ny (1)), lower bounds lbd1 , upper bounds ubd1 and an initial relaxation order ω1 . Apply SDPR with these parameters to (3.14) and obtain a solution u1 . Step 2 - Extension: Extend the (k-1)th iteration’s solution uk−1 to a finer grid. Choose either x- or y-direction as the direction of refinement, i.e. choose either Nx (k) = 2 Nx (k − 1) − 1 and Ny (k) = Ny (k − 1), or Nx (k) = Nx (k −1) and Ny (k) = 2 Ny (k −1)−1. In order to extend uk−1 to the new grid with the doubled number of grid points, assume without loss of generality the direction of extension is x. The interpolation ⋆ of the solution uk−1 to uk−1 is given by the scheme 3.2. DIFFERENTIAL EQUATIONS AND THE SDPR METHOD k−1 u2i−1,j k−1 ⋆ u2i,j ⋆ = = 83 k−1 ui,j ∀i ∈ {1, . . . , Nx (k − 1)}, ∀j ∈ {1, . . . , Ny (k)}, k−1 k−1 1 ∀i ∈ {1, . . . , Nx (k − 1) − 1}, ∀j ∈ {1, . . . , Ny (k)}. 2 ui+1,j + ui,j ⋆ The interpolated solution uk−1 is a first approximation to a solution of POP (3.26) for the Nx (k) × Ny (k)grid. Step 3a - Apply SDPR method: We choose new parameters Fk , lbdk , ubdk , ωk , g(k) and h(k). It is based on the idea, that we may be able to choose ωk < ωk−1 , if we exploit information given by the ⋆ interpolated solution uk−1 . One possibility to do this, is to choose the new objective function Fk = FM , k where FM is defined by X k−1 ⋆ 2 k (ui,j − ui,j ) . (3.31) FM (u) = i,j We may choose this objective function as we are interested in finding a feasible solution of (3.26) with ⋆ minimal Euclidean distance to the interpolated solution uk−1 . Another possibility to utilize the information k−1 ⋆ given by u is to tighten the lower and upper bounds by n o k−1 k−1 ⋆ lbdki,j = max lbdi,j , ui,j −δ ∀ i, j, n o k−1 k k−1 ⋆ ubdi,j = min ubdi,j , ui,j + δ ∀ i, j, for some δ > 0. Apply the SDPR method to obtain uk . Step 3b - Apply Newton’s method or SQP: It may occur the SDP relaxations of (3.26) for the finer grid become intractable, even if ωk < ωk−1 . Therefore, we may just apply Newton’s method to (3.17) or ⋆ SQP to (3.26) for the finer discretization (Nx (k), Ny (k)), both starting with the interpolated solution uk−1 as initial guess, to obtain a better approximate solution u. The steps 2 and 3 are repeated until an accurate solution for a high resolution grid is obtained. The SDPR method with all its options and the grid-refining method are demonstrated on a variety of numerical examples in 3.3. 3.2.3 Enumeration algorithm The SDPR method aims at finding a discrete approximation to a solution of a PDE problem. The freedom to choose an objective function for detecting particular solutions of a PDE problem is a feature of the SDPR method which is particularly interesting for a PDE problem with many solutions. Beside finding a particular solution of a PDE problem, another challenging problem is to find all solutions of a PDE problem. The problem of finding discrete approximations for all solutions of a PDE problem (3.14) is the problem of finding all real solutions of the system of polynomial equations (3.17). Classical methods to solve this problem are Gröbner basis or polyhedral homotopy method, which we describe briefly at the end of this section. They compute all complex solutions to (3.17) and it remains to choose the real solutions among them. A recent method which avoids computing all complex solutions and directly computes all real solutions is given by [55] for the case the solution set of (3.17) is finite. Another method is the extraction algorithm presented in [36] for finding all optimal solutions of (3.26). However, both methods, [55] and [36] do not exploit sparsity in (3.17) and (3.26), respectively, which restricts their applicability to small- and medium-scale systems. In the following we present an algorithm to enumerate all solutions of a system of polynomial equations first proposed in [63] for the cavity flow problem, but which can be extended to a more general case of (3.17). For this method we need to assume the number of solutions of (3.17) is finite, i.e. the feasible set of (3.26) is finite. We also assume that all feasible solutions of (3.26) are distinct with respect to the objective function F , i.e., there is no pair of feasible solutions with identical objective value. The SDPR method enables us to approximate the global minimal solution u⋆ =: u(1)⋆ of (3.26). Beside the minimal solution, we are also interested in finding the solution u(2)⋆ with the second smallest objective value, the solution u(3)⋆ with the third smallest objective value or in general the solution u(k)⋆ with the kth smallest objective value. Based 84 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS on the SDPR method we propose an algorithm that enumerates the solutions of (3.17) with the k smallest objective values. Our algorithm shares the idea of separating the feasible set by additional constraints with Branch-and-Bound and cutting plane methods that are used for solving mixed integer linear programs and general concave optimization problems [38]. In contrast to the linear constraints of those methods we impose quadratic constraints to separate the feasible set. Algorithm 3.1 Find the approximations to the solutions of (3.17) with the k smallest objective values: Given u(k−1) , the approximation to the solution with the (k − 1)th smallest objective value obtained by applying the SDPR method with relaxation order ω to POPk−1 from the (k − 1)th iteration of the algorithm. 1. Choose ǫk > 0. 2. Choose a vector bk ∈ {0, 1}n. 3. Add the following quadratic constraints to POPk−1 and denote the resulting POP with smaller feasible set as POPk . (k−1) 2 (3.32) ) ≥ ǫk for all j with bj = 1. (uj − uj 4. Apply the SDPR method with relaxation order ω to POPk . Obtain an approximation u(k) for u(k)⋆ . 5. Iterate steps 1–4. The idea of Algorithm 3.1 is to impose an additional polynomial inequality constraint (3.32) to the POP (3.26) in iteration k, that excludes the previous iteration’s solution u(k−1) from the feasible set of (3.26). In the case the feasible set of (3.26) is finite and u(k−1) is sufficiently close to u(k−1)⋆ , the new constraint excludes u(k−1)⋆ from the feasible set of (3.26) and u(k)⋆ is the new global minimizer of (3.26). Of course, there are various alternatives to step 3 in Algorithm 3.1, in order to exclude u(k−1)⋆ from the feasible set of the POP. One alternative constraint is (k−1)⋆ un+i − ǫi = 0 for all i with bi = 1, (3.33) ui − ui n where b ∈ {0, 1} , ǫi > 0 and un+i an additional slack variable bounded by −1 and 1. It is easy to see that (3.33) is violated, if u = u(k−1)⋆ . However, it turned out that the numerical performance of (3.33) is inferior to the one of (3.32) for problems of type (3.26) as the tuning parameters ǫi and b is far more difficult for (3.33) compared to (3.32). A second alternative to exclude u(k−1)⋆ are lp -norm constraints such as k u − u(k−1)⋆ kp = n p X (k−1)⋆ ui − ui i=1 ! p1 ≥ ǫ, (3.34) for p ≥ 1. The disadvantage of the constraints (3.34) is, they destroy the correlative sparsity of (3.26), as all ui (i = 1, . . . , n) occur in the same constraint. Therefore the advantage of the sparse SDP relaxations is lost and the POP can not be solved efficiently anymore. These observations justify to impose (3.32) as additional constraints in Algorithm 3.1. We obtain the following results for Algorithm 3.1. Proposition 3.3 Let (u(1) , . . . , u(k−1) ) be the output of the first (k − 1) iterations of Algorithm (3.1). If this output is a sufficiently close approximation of the vector (u(1)⋆ , . . . , u(k−1)⋆ ) of the (k − 1) solutions with smallest objective value, and if the feasible set of POP (3.26) is finite and distinct in terms of the n objective, i.e. F (u(1)⋆ ) < F (u(2)⋆ ) < . . ., then there exist b ∈ {0, 1} and ǫ ∈ Rn such that the output u(k) of Algorithm 3.1 (for kth iteration) satisfies u(k) (ω) → u(k)⋆ when ω → ∞. (3.35) 3.2. DIFFERENTIAL EQUATIONS AND THE SDPR METHOD 85 n Proof: As each u(j) is in a neighborhood of u(j)⋆ for all j ∈ {1, . . . , k − 1}, we can choose b ∈ {0, 1} and a vector ǫ ∈ Rn , s.t. 2 (j)⋆ (j) < ǫi , ∀j ∈ {1, . . . , k − 1} ∃i with bi = 1 s.t. ui − ui and for each j ∈ {1, . . . , k − 1} holds 2 (l)⋆ (j) ui − ui ≥ ǫi ∀l ≥ k ∀i with bi = 1. Let POP(k) denote (3.26) with the k systems of additional constraints given by step 3 in Algorithm 3.1, where the kth constraints are given by (3.32) for the constructed b and ǫ. Then it holds n o feas POP(k) = feas (3.26) \ u(1)⋆ , . . . , u(k−1)⋆ . Thus, u(k)⋆ is the global minimizer of POP(k) and the global minimum is F (u(k)⋆ ). As the bounds (3.19) guarantee the compactness of the feasible set, it holds with the convergence theorem for the sparse SDP relaxations [52], if ω → ∞, (3.36) u(k) (ω) → u(k)⋆ . Although we have proven convergence, the capacity of current SDP solvers restricts the choice of the relaxation order ω to small integers, typically ω = ωmax + 1 or ω = ωmax + 2. Moreover, we need to choose the parameters ǫ and b appropriately, to obtain good approximations of the k feasible solutions with the smallest objective value. In the numerical experiments in 3.3.5, we see the Gröbner basis method is an useful tool to tune the two parameters ǫ and b, as it allows to confirm whether we derive the k solutions of smallest objective value successfully in case (Nx , Ny ) is small. In the following we briefly describe Gröbner basis method and Polyhedral Homotopy Continuation as methods to test the SDPR method and to tune the parameters in Algorithm 3.1. Gröbner basis method The Gröbner basis method to find all complex solutions of a given system of zero dimensional polynomial equations is an useful tool for tuning the parameters of the SDPR method and Algorithm 3.1, and for validating its numerical results. In order to do this, we study (3.17) by the rational univariate representation [85], [78], which is a variation of the Gröbner basis method, for coarse discretizations (Nx , Ny ). For a mesh with N := Nx = Ny small (for instance N = 5 in the example in 3.3.5), (3.17) is solvable with this method (Groebner(Fgb) in Maple 11, nd gr trace and tolex gsl in Risa/Asir). Applying Gröbner basis method to solve (3.17) for a problem satisfying the assumptions of Proposition 3.3, and enumerating all solutions by their objective value allows us to confirm whether the solutions of the SDPR method are indeed the minimal solutions of (3.17) and to determine which relaxation order ω is sufficient to derive this global minimizer. The result is also used to tune parameters ǫki in Algorithm 3.1. We have no theorem which states that the tuning based on the coarse mesh case is good for the fine mesh case. However, we believe this tuning provides a better approximation for the fine mesh case, too. Note, whereas the Gröbner basis method finds all complex solutions of (3.17), the SDPR method finds the real solution of (3.17) that minimizes F . Polyhedral Homotopy Continuation Method Another recent approach for solving (3.17) is the Polyhedral Homotopy Continuation Method for polynomial systems [39]. Consider the problem to find all isolated zeros of a system of n polynomials f (x) = (f1 (x), . . . , fn (x)) = 0 in a n−dimensional complex vector variable x = (x1 , . . . , xn ) ∈ Cn . The idea of homotopy continuation methods is to define a smooth homotopy system with a continuation parameter t ∈ [0, 1], h(x, t) = (h1 (x), . . . , hn (x)) = 0, 86 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS using the algebraic structure of the polynomial system. The homotopy system is constructed, such that the solutions of the starting polynomial system h(x, 0) = 0 can be computed easily and that the target polynomial system h(x, 1) = 0 coincides with the system f (x) = 0 to be solved. Furthermore, every connected component of the solutions (x, t) ∈ Cn × [0, 1) of h(x, t) = 0 forms a smooth curve. The number of homotopy curves that are necessary to connect the isolated zeros of the target system to isolated zeros of the starting system determines the computational work involved in tracing homotopy curves. A recent software package to determine all complex isolated solutions of f (x) = 0 is PHoM by Gunji et al. [31]. Thus, we may apply PHoM to find the complex solutions of a discretized PDE problem pi,j (u) = 0 for all (i, j). Then, we select the real among all complex solutions of (3.17) and compare those solutions to the solutions obtained by the SDPR method. Beside its property of finding all complex solutions, PHoM has the drawback, that the computation time grows exponentially in the dimension n of the system. This is due to the fact, the number of isolated solutions increases exponentially in n. Therefore we are restricted to very coarse meshes with n ≤ 10 when applying PHoM to (3.17). As the Gröbner base method, the polyhedral homotopy method can be used for tuning the parameters in Algorithm 3.1, too. 3.2.4 Discrete approximations to solutions of differential equations In 3.1.1 we discussed, if a finite difference scheme is convergent and the discretization (Nx , Ny ) is chosen sufficiently fine, then a solution of (3.17) is a discrete approximation to a solution of the differential equation (3.14). However, there is no theorem proving the convergence of finite difference schemes for general classes of nonlinear differential equations. In the case of many nonlinear PDE we can not guarantee that a solution of (3.17) is indeed a discrete approximation of a solution of (3.14). Definition 3.4 A solution of (3.17) that is not a discrete approximation of a solution of the PDE problem (3.14) is called a fake solution. In numerical experiments our main indicator for a solution u of (3.17) not being a fake solution is if we succeed to extend u from a coarse grid to finer and finer grid via the grid refining method. Another property of solution to (3.17) is the notion of stability. Definition 3.5 Let J(·) denote the Jabobian of (3.17) and me (·) its maximal eigenvalue. A solution u to (3.17) is called stable, if all eigenvalues of J(u) are non-positive, i.e., if me (u) ≤ 0. If not, it is called unstable. Distinguishing stable and unstable solutions is of interest for certain classes of nonlinear PDE problems. In 3.3.3 we will discuss Reaction-Diffusion equations as an example of such a class. 3.3 Numerical experiments In 3.2.2 we introduced the SDPR method for computing discrete approximations to solutions of differential equations. In this section we demonstrate the broad scope of this method by applying it to problems involving nonlinear differential equations which arise from a wide range of fields. It is indeed possible to find highly accurate approximations to many nonlinear differential equations by techniques based on sparse SDP relaxations. For our numerical experiments we apply the software SparsePOP [103] as an implementation of the sparse SDP relaxations (2.18). For applying domain-space sparsity conversion methods we use SparseCoLO [20], and as an SQP based solver we utilize the MATLAB Optimization toolbox routine fmincon. As SDP solver for SparsePOP or the primal SDP relaxation from (2.2.7) we apply SeDuMi [95]. For an eventual transformation from POP into a QOP we use one of the heuristics AI, AII, BI or BII from 2.3. When applying the SDPR method to obtain an approximate solution for (3.26), the most important measure of accuracy of its solution u is the scaled feasibility error ǫsc defined by hk (u) gl (u) Di,j (u) − fi,j |, − | |, min ,0 ∀ i, j, k, l , ǫsc = min − | σi,j (u) σk (u) σ̂l (u) 87 3.3. NUMERICAL EXPERIMENTS where σi,j (u), σk (u) and σ̂l (u) the maxima over all monomials in the corresponding enumerator polynomials. Note, ǫsc (u) does measure how accurate u approximates a feasible solution of (3.26), but it does not measure how accurate the finite difference scheme approximates the continuous problem (3.14). Another question is, how well F (u) approximates the minimum of (3.26). In the case we choose the primal relaxations (a) and (b) from 2.2.7 or the dual relaxations (2.18) for some linear objective F , F (u) approximates the minimum of (3.26) very accurately, if the feasibility error of u is small. But in case F is nonlinear and we choose the dual relaxations, we have to consider the optimality error ǫobj defined by ǫobj = | min(sSDPω ) − F (u) | max {1, | F (u) |} as a measure for the optimality of u. Recall, when applying the SDPR method to solve a differential equation, the most important choices are: The objective function F , bounds lbd and ubd, relaxation order ω in the case of dual relaxations (2.18), Newton’s method or SQP as locally fast convergent method, possibly additional constraintss hk and gl . Finally, ones needs to decide whether to apply the grid-refining method from 3.2.2 with extension strategy 3a or 3b. To evaluate the results of the SDPR method for approximating the solutions of differential equations, we consider (a) to apply the SDPR method to PDE problems where an analytical solution is known, and (b) to compare the performance of the SDPR method with the performance of MATLAB Optimization Toolbox, a general purpose solver based on the Finite Element Method. Finally, we may apply Gröbner basis computation or the Polyhedral Homotopy Method to verify that the SDPR method provides accurate approximations to feasible solutions of (3.26), as mentioned in 3.2.3. Moreover, J(u) denotes the Jacobian of (3.17) at u and me (u) its largest eigenvalue. All numerical experiments are conducted on a LINUX OS with CPU 2.4 GHz and 8 Gb memory. The total processing time in seconds is denoted as tC . 3.3.1 A nonlinear elliptic equation with bifurcation A well known yet interesting nonlinear elliptic PDE problem, which we have already seen in Example 3.27, is given by uxx (x, y) + uyy (x, y) + λu(x, y) 1 − u(x, y)2 = 0 ∀ (x, y) ∈ [0, 1]2 , u(x, y) = 0 ∀ (x, y) ∈ ∂[0, 1]2 , (3.37) 0 ≤ u(x, y) ≤ 1 ∀ (x, y) ∈ [0, 1]2 , where λ ≥ 0. In fact, this PDE is known as the Allen-Cahn Equation. It was shown in [93], there exists a unique nontrivial solution for this problem if λ > λ0 = 2π 2 ≈ 19.7392, and there exists only the trivial zero solution if λ ≤ λ0 . Due to the bifurcation at λ0 , homotopy-like continuation methods, which start from a solution of a system with weak non-linearity to attempt the system with strong non-linearity, Pcannot be applied to solve (3.37). We fix λ = 22 and apply the SDPR method with ω = 2 and F (u) = − i,j ui,j . In order to study the efficiency of the various options of the SDPR method, we consider different settings: Dual SDP relaxations with and without additional local solver for the POP derived from (3.37), dual and primal SDP relaxations for a QOP equivalent to the POP, the grid-refining method starting from a coarse grid solution, tight and loose upper bounds. The numerical results are given in Table 3.1 and pictured in Figure 3.5. When applying dual SDP relaxations to the original POP, we observe that the solution provided by the SDPR method is very accurate even in the case no additional local method is used. Moreover, the size of the SDP relaxations and thus the computational cost increases rapidly for increasing Nx and Ny . One way to address this problem is to apply the transformation from POP into an equivalent QOP. Both, the primal and dual SDP relaxations are substantially smaller than the dual SDP relaxations for the original POP. But, ubd needs to be tightened when applying SDP relaxations to the QOP, in order to preserve numerical accuracy. Note, for this problem d-space sparsity is richer than correlative sparsity, as the size of the corresponding SDP relaxations and the resulting tC are smaller. The most efficient mean to obtain high resolution approximations to a solution of (3.37) is the grid-refining method. However, the grid-refining method relies on the assumption that the behavior of the discretized system does not change much for increasing Nx and Ny , which is not the case when starting from a very coarse discretization in general. 88 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS Therefore, SDP relaxations for the equivalent QOP are a promising tool to attempt a PDE problem on a higher resolution grid directly. Finally, we notice that the solution of the sparse SDP relaxation remains a sufficiently good initial guess for SQP, even if its feasibility error ǫsc is not that small. SDP relaxation Dual Dual + Grid-refining 3b Dual Dual Dual + Grid-refining 3b Dual Dual + Grid-refining 3b Dual Dual Primal Primal Primal Local solver SQP SQP none SQP SQP SQP SQP SQP SQP SQP none SQP POP to QOP no no no no no no no no yes yes yes yes ubd 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.45 0.45 0.6 0.6 n 16 1521 81 81 1521 49 225 196 196 196 841 841 Nx 6 41 11 11 41 9 17 16 16 16 31 31 Ny 6 41 11 11 41 9 17 16 16 16 31 31 ǫsc -1e-14 -1e-13 -1e-10 -4e-15 -1e-13 -9e-15 -1e-9 -5e-11 -8e-15 -5e-4 -5e-14 tC 3 426 418 420 1016 39 49 OOM 107 35 1133 3763 Table 3.1: SDPR method results for (3.37), where OOM stands for ’Out of Memory’. 0.4 0.4 0.5 0.35 0.4 u(x,y) 0.25 0.2 0.2 0.1 u(x,y) 0.3 0.3 0.5 0.35 0.4 0.3 0.3 0.25 0.2 0.2 0.1 0.15 0.15 0 1 0.1 0.8 0.8 0.6 0.4 y 0.05 0.4 0.2 0.1 0.8 1 0.6 0 1 1 0.6 0.8 0 0 0.6 0.4 y x 0.05 0.4 0.2 0.2 0.2 0 0 x Figure 3.5: Solution for (3.37) in case λ = 22 for (Nx , Ny ) = (6, 6) and (Nx , Ny ) = (41, 41). To examine whether the obtained solution is the only strictly positive one of the discretized PDE, we impose additional constraints | ux (x, y) | ≤ M | uy (x, y) | ≤ M ∀ (x, y) ∈ [0, 1]2 . (3.38) We apply the SDPR method with ω = 2 and Nx = Ny = 6 to (3.37) for λ = 22 under the additional constraints (3.38). For M > 0.8 we detect the positive solution obtained before. If we decrease M sufficiently, we obtain the zero solution. Hence, it seems there exists exactly one positive non-trivial solution to the discretization of (3.37), and this solution converges to the strictly positive solution of (3.37) for Nx , Ny → ∞. As another way to confirm the accuracy of the SDPR method, we take advantage of a further property of (3.37). It was shown in [15], a function u : [0, 1]2 → R that is a minimizer of the optimization problem 2 R u4 u 2 2 u + u − 2λ dx dy − minu: [0,1]2 →R 2 x y 2 4 [0,1] (3.39) s.t. u=0 on ∂[0, 1]2 , 0≤u≤1 on [0, 1]2 , 89 3.3. NUMERICAL EXPERIMENTS is a solution to (3.37). The integral to be minimized in this problem is called the energy integral. By discretizing (3.39) via a finite difference scheme it can be transformed into a POP analogously to a PDE of form (3.14). In opposite to (3.26) that we derive from (3.14), the objective function F is not of free choice but canonically given by the discretization of the objective function in (3.39). We apply the SDPR method with relaxation order ω = 2 to (3.37) and (3.39) on a 6 × 6- and a 11 × 11- grid and obtain an identical solution for both problems. These results are reported in Table 3.2, where ∆u, given by ∆u = max | ui,j − ûi,j |, i,j evaluates the deviation of the SDPR method solutions for both problems; ui,j denotes the SDPR solution to (3.37) and ûi,j the SDPR solution to (3.39). Problem (3.37) (3.39) (3.37) (3.39) Nx 6 6 11 11 Ny 6 6 11 11 tC 3 2 418 98 ǫobj 2e-14 1e-10 4e-15 2e-10 ǫsc -1e-14 -9e-15 - ∆u 2e-6 2e-6 9e-7 9e-7 Table 3.2: SDPR results for (3.37) and (3.39). The solutions to both problems are highly accurate and we note that the total computation time to minimize the energy integral is less than the time required to solve the polynomial optimization problem corresponding to (3.37). Finally, we compare the numerical performance of the SDPR method for (3.37) to existing solvers for nonlinear PDE problems. We apply the nonlinear solver from Matlab PDE toolbox to (3.37). The Matlab solver is FEM based and requires an initial guess to search for a solution of the PDE problem. When starting from the zero-solution or a number of random positive functions, this solvers detects the trivial solution. Even when choosing u0 with u0 (x, y) := 0.43 sin(πx) sin(πy) on [0, 1]2 as initial guess, the FEM solver detect the trivial solution not the non-trivial one, although u0 is close to the non-trivial solution (On P 41 × 41-grid: max | u − u0 |= 0.006, 4112 i,j | ui,j − u0i,j | = 0.003). Although the FEM solver finds the trivial solution in less than 60 seconds on a mesh of much higher resolution (67356 nodes, 134204 triangles) than those solved by the SDPR method, it needs a very good initial guess to find the more interesting, non-trivial solution. It is the advantage of the SDPR method, that no initial guess is required to find a accurate approximation of the strictly positive solution of (3.37). 3.3.2 Illustrative nonlinear PDE problems A problem in Yokota’s text book Simple ODE problems can be solved by the SDPR method with ease. To demonstrate this, consider the easy solvable nonlinear boundary value problem ü(x) + 81 u(x)u̇(x) − 4 − 14 x3 u(1) u(3) 10 ≤ u(x) =0 ∀ x ∈ [1, 3], = 17, = 43 3 , ≤ 20 ∀ x ∈ [1, 3]. (3.40) For details about problem (3.40) see [105]. Applying the SDPR method with ω = 2, objective function F , defined by Nx X ui , F (u) = i=1 and without using a locally fast convergent method, yields the highly accurate, stable solution, that is documented in Table 3.3 and pictured in Figure 3.6. 90 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS Nx 200 ǫsc 2e-9 ǫobj -4e-11 me -3 tC 104 Table 3.3: Numerical results for problem (3.40). 17 16 u u(x) 15 14 13 12 11 1 1.2 1.4 1.6 1.8 2 x 2.2 2.4 2.6 2.8 3 Figure 3.6: SDPR method solution u for problem (3.40). Nonlinear wave equation As an example of a hyperbolic PDE problem we study time periodic solutions of the nonlinear wave equation −uxx + uyy + u (1 − u) + 0.2 sin(2x) = 0 ∀ (x, y) ∈ [0, π] × [0, 2π], u(0, y) = u(π, y) = 0 u(x, 0) = u(x, 2π) −3 ≤ u(x, y) ≤ 3 ∀ y ∈ [0, 2π], ∀ x ∈ [0, π], (3.41) ∀ (x, y) ∈ [0, π] × [0, 2π]. As far as we have checked on the mathsci data base, there is no mathematical proof of the existence of periodic solution of this system. However, the SDPR method finds some periodic solutions. We observed the POP corresponding to problem (3.41) has various solutions. Therefore, the choice of the objective determines the solution found by the sparse SDP relaxation. We consider the functions X X ui,j , σi,j ui,j , F2 (u) = F1 (u) = i,j i,j as objective for (3.26), where σi,j (i = 1, . . . , Nx , j = 1, . . . , Ny ) are random variables that are uniformly distributed on [−0.5, 0.5]. We apply the SDPR method with ω = 2 and Newton’s method as a local solver. The results are enlisted in Table 3.4 and pictured in Figures 3.7 and 3.8. Imposing Variation bounds: Beside choosing different objective functions in the SDPR method, a second possibility to detect other solutions of a PDE is to impose additional constraints polynomial in the unknown functions. In 3.2.1 we introduced variation bounds (3.21) to restrict the space of functions we are searching for solutions. For (3.41) we impose the variation bounds | uy (x, y) |≤ 0.5 ∀ (x, y) ∈ (0, π) × (0, 2π). (3.42) 91 3.3. NUMERICAL EXPERIMENTS SDP relaxation Dual Dual + Grid-refining 3b Dual Dual + Grid-refining 3b objective F1 F1 F2 F2 Nx 5 33 5 33 Ny 6 40 5 33 ǫsc -4e-10 -3e-8 -3e-11 -5e-10 tC 151 427 19 86 Table 3.4: SDPR method results for (3.41). 2.5 2 2 1.5 1 0.5 u(x,y) u(x,y) 1.5 1 0.5 0 −0.5 0 −1 −0.5 8 −1.5 8 6 4 6 3 4 4 3 4 2 2 y 2 2 1 0 0 y x 1 0 0 x Figure 3.7: Solutions of (3.41) by SDPR method with objective F1 . If the SDPR method is applied to (3.41) with additional condition (3.42) and F1 as objective, another solution to the PDE problem is obtained, which is documented in Table 3.5 and pictured in Figure 3.9. Thus several solutions of (3.41) are detected by choosing different objective functions (3.26) and by imposing additional polynomial inequality constraints. SDP relaxation Dual Dual + Grid-refining 3b objective F1 F1 Nx 5 17 Ny 6 21 ǫsc -3e-15 -3e-14 tC 437 1710 Table 3.5: SDPR method results for (3.41) under additional constraint (3.42). A system of elliptic nonlinear PDEs As a PDE problem of two unknown functions in two variables, consider the following problem, where we distinguish two types of boundary conditions, Case I (Dirichlet condition) and Case II (Neumann condition). uxx + uyy + u 1 − u2 − v 2 = 0, vxx + vyy + v 1 − u2 − v 2 = 0, 0 ≤ u, v ≤ 5. ∀ (x, y) ∈ [0, 1]2 . (3.43) 92 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS 2.5 2 2 1.5 1.5 u(x,y) u(x,y) 2.5 1 0.5 1 0.5 0 8 0 8 6 4 6 3 4 4 3 4 2 2 2 1 0 y 2 0 1 0 y x 0 x Figure 3.8: SDPR solutions of (3.41) with objective F2 . 0.05 0.06 0.04 u(x,y) u(x,y) 0.02 0 0 −0.02 −0.05 8 −0.04 8 6 4 6 3 4 4 3 4 2 2 y 2 2 1 0 0 y x 1 0 0 x Figure 3.9: SDPR solutions of (3.41) under additional constraint (3.42) with objective F1 . Case I: u(0, y) u(x, 0) v(x, 0) or Case II: ux (0, y) uy (x, 0) vx (0, y) vy (x, 0) = 0.5y + 0.3 sin(2πy), = 0.4x + 0.2 sin(2πx), = v(x, 1) u(1, y) = 0.4 − 0.4y u(x, 1) = 0.5 − 0.5x = v(0, y) = v(1, y) = 0 ∀ y ∈ [0, 1], ∀ x ∈ [0, 1], ∀ x ∈ [0, 1], ∀ y ∈ [0, 1]. = −1, = 2x, = 0, = −1, ux (1, y) uy (x, 1) vx (1, y) vy (x, 1) ∀ y ∈ [0, 1], ∀ x ∈ [0, 1], ∀ y ∈ [0, 1], ∀ x ∈ [0, 1]. In both cases, we choose F (u, v) = P i,j =1 = x + 5 sin( πx 2 ) =0 =1 ui,j for the SDPR method. Case I. We apply the SDPR method with both primal and dual SDP relaxations exploiting sparsity. Also, as the degree of this PDE problem is three, we can apply the POP to QOP transformation to reduce the size of the SDP relaxations. Finally we apply the grid refining method to extend the coarse grid solutions to finer grids. When applying the dual SDP relaxations to the POP, lbd = 0 and ubd = 5 are given by (3.43). The bound ubd is tightened to ubd = 0.6 for the primal and dual SDP relaxations of the QOP in order to obtain accurate solutions. The numerical results of the SDPR method with ω = 2, and SQP as 93 3.3. NUMERICAL EXPERIMENTS a local solver are reported in Table 3.6. We observe, exploiting d-space sparsity and applying the primal SDP relaxations is very efficient for this problem. In fact, the d-space sparsity is richer than the correlative sparsity, as a comparison of the total computation time of primal and dual SDP relaxations for the QOP derived from (3.43) for Nx = Ny = 11 reveals. When applying the dual SDP relaxations, the grid-refining method is useful to extend coarse grid solutions to high resolution grids. The approximate solution for u(·, ·) is pictured in Figure 3.10, the corresponding v equals zero on the entire domain. SDP relaxation Dual Dual + Grid-refining 3b Dual Dual + Grid-refining 3b Dual + Grid-refining 3b Dual Primal Primal Transform POP to QOP no no no no no yes yes yes n 18 7938 32 162 722 162 162 338 Nx 5 65 6 11 21 11 11 15 Ny 5 65 6 11 21 11 11 15 ǫsc -4-e13 -2e-14 -4e-13 -2e-13 -4e-16 -9e-9 -3e-8 -3e-8 tC 2 12280 150 156 238 185 19 107 Table 3.6: Results of SDPR method for (3.43) in Case I. W 0.5 0.5 0.4 0.4 W u(x,y) 0.3 0.3 0.2 0.2 0.1 0.1 0 1 0 1 0.8 1 0.6 [ 0.8 1 0.6 0.8 0.6 0.4 0.4 0.2 y 0.2 0 0 x Z 0.8 0.6 0.4 [ 0.4 0.2 0.2 0 0 Z Figure 3.10: SDPR method solution u of (3.43) for Case I and two different discretizations. We compare the performance of the SDPR method for Case I of (3.43) to Matlab PDE toolbox. Starting from an arbitrary initial guess the Matlab solver detects the same solution as the SDPR method in 2 seconds on a mesh with 2667 nodes and 5168 triangles, and in 15 seconds on a mesh with 10501 nodes and 20672 triangles, c.f. Figure 3.11. Thus, the FEM solver from Matlab is much more efficient in finding the solution. However, the discretization of (3.43) under Dirichlet condition has exactly one real solution. In a more difficult PDE problem with many solutions a good initial guess is required for the Matlab solver to find a solution of interest. Case II. We apply the SDPR method with the same settings as in Case I, and compare the efficiency of primal and dual SDP relaxations and the grid-refining method. For the primal and dual SDP relaxations of the QOP the upper bounds are tightened to ubdu = 4 and ubdv = 1.5. The numerical performance of the SDPR method is reported in Table 3.7. The single solution (u, v) of the discretized differential equation is illustrated in Figure 3.12. As in Case I, we observe that the transformation from POP to QOP is efficient to reduce the size of the SDP relaxations while the accuracy of the approximation is preserved. Moreover, d-space sparsity is richer than correlative sparsity, as the primal SDP relaxations for n = 242 can be solved in 58s whereas solving the dual SDP relaxations for the same QOP requires 748s. 94 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS Height: u Height: u 0.5 0.5 0.45 0.45 0.4 0.4 0.35 0.35 0.3 0.3 0.25 0.25 0.2 0.2 0.15 0.15 0.1 0.1 0.05 0.05 0 1 0.8 1 0.6 0.8 0 1 0.8 1 0.6 0.6 0.4 0.8 0.4 0.2 0.2 0 0.6 0.4 0.4 0.2 0 0 0.2 0 Figure 3.11: Solution u of (3.43) by Matlab PDE Toolbox for mesh with 2667 nodes/ 5168 triangles (left) and 10501 nodes/ 20672 triangles (right). SDP relaxation Dual Dual + Grid-refining 3b Dual Dual + Grid-refining 3b Dual Primal Transform POP to QOP no no no no yes yes n 32 3042 50 242 242 242 Nx 6 41 7 13 13 13 Ny 6 41 7 13 13 13 ǫsc -7e-13 -3e-15 -1e-10 -6e-13 -5e-11 -6e-13 tC 128 2160 713 728 748 58 Table 3.7: Results of the SDPR method for (3.43) in Case I. Nonlinear parabolic equation Consider a nonlinear parabolic PDE problem of two dependent scalar functions 1 50 uxx − uy + 1 1 50 vxx − vy + u2 v − 4u + 3u − u2 v u(0, y) = u(1, y) v(0, y) = v(1, y) u(x, 0) v(x, 0) =0 =0 =1 =3 = 1 + sin(2πx) =3 ∀x ∈ [0, 1], y ≥ 0, ∀x ∈ [0, 1], y ≥ 0, ∀ y ≥ 0, ∀ y ≥ 0, ∀ x ∈ [0, 1], ∀ x ∈ [0, 1]. (3.44) In order to bring (3.44) into form (3.14), we need to cut y at y = T . Since problem (3.44) is parabolic, the solutions ((u, v)(Nx , Ny )) of the discretized problems converge to solutions (u(·, ·), v(·, ·)) of (3.44) P with Theorem 3.4. We apply the grid-refining method with strategy 3b, where F is given by F (u, v) = i,j ui,j and ω = 3. Furthermore, lbd ≡ 0 and ubd ≡ 5 are chosen as bounds for u and v. The grid-refinig method yields a highly accurate, stable solutions on a 33 × 65-grid; see Table 3.8 and Figure 3.13. 95 3.3. NUMERICAL EXPERIMENTS 3.5 W 1.1 3 1 X 2.5 v(x,y) u(x,y) 0.9 2 0.8 1.5 0.7 1 0.5 1 1 0.8 0.8 1 0.6 1 0.6 0.8 0.6 0.4 [ 0.2 y 0.2 0 0 0.8 0.6 0.4 0.4 [ Z 0.2 0.2 0 y x 0.4 0 Z x Figure 3.12: SDPR method solutions u (left) and v (right) of (3.43) for Case II. strategy initial SDPR method Grid-refining 3b Nx 5 33 Ny 9 65 me -4.12 -2.88 ǫsc -2e-10 -5e-11 Table 3.8: Results for diffusion problem (3.44). First order PDEs An optimization based approach to attempt first order PDE was proposed by Guermond and Popov [29, 30]. In [29] the following example of a first order PDE with a discontinuous solution is solved on a 40 × 40−grid: ux (x, y) u(0, y) u(0, y) = 0 ∀ (x, y) ∈ [0, 2] × [0.2, 0.8], = 1 if y ∈ [0.5, 0.8], = 0 if y ∈ [0.2, 0.5[. (3.45) Applying the SDPR method with an forward or central difference approximation for the first derivative in (3.45) we detect the discontinuous solution ( 1 if y ≥ 0.5 u(x, y) = 0 otherwise on a 40 × 40-grid. A more difficult first order PDE problem is given by ux (x, y) + u(x, y) − 1 = 0 u(0, y) = u(1, y) =0 0 ≤ u(x, y) ≤1 ∀ (x, y) ∈ [0, 1]2 , ∀ y ∈ [0, 1], ∀ (x, y) ∈ [0, 1]2 . (3.46) As can be seen easily and was pointed out in [30], problem (3.46) is not well-posed since the outflow boundary condition is over-specified. Problem (3.46) is discussed in detail in [30] and the authors obtained 1 an accurate approximation to the exact solution by L P approximation on a 10×10-grid. Applying the SDPR method with ω = 1 and objective function F (u) = i,j ui,j on a 10 × 10 grid, we obtain a highly accurate 96 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS 3 4 2.5 3.5 3 v(x,y) u(x,y) 2 1.5 1 2.5 2 0.5 1.5 0 10 1 8 6 0.6 8 0.6 4 1 0.8 1 10 0.8 4 0.2 0 y 0.4 6 0.4 2 0 0.2 2 x 0 y 0 x Figure 3.13: Solutions u (left) and v (right) for diffusion problem (3.44). approximation (| ǫsc |< 1e − 14) to this solution in less than 10 seconds in the case we choose a forward difference approximation for the first derivative. In the case of choosing a central or a backwards difference scheme the dual problem in the resulting SDP relaxation becomes infeasibe. Moreover, by applying the SDPR method on a 50 × 50-grid we are able to obtain a highly accurate approximation to the solution of (3.46) in less than 250 seconds, as pictured in Figure 3.14. 0.6 0.5 0.5 0.4 0.4 u(x,y) u(x,y) 0.6 0.3 0.2 0.2 0.1 0.1 0 1 0.3 0.8 0.6 0.4 0.2 0 1 0.8 0.6 0.4 0.2 0 0 1 0 0.2 0.8 0.4 0.6 0.6 0.4 0.8 0.2 0 1 y y x x Figure 3.14: Solution u by SDPR method for (3.46). 3.3.3 Reaction-diffusion equations An interesting class of PDE problems to analyze by the SDPR method is the class of reaction-diffusion equations. Many reaction-diffusion equations are systems of nonlinear PDE with multiple solutions. For some of them special unstable solutions exist that are difficult to detect by standard homotopy-like continuation methods. We demonstrate how the features of the SDPR method can be used to find special solutions of a reaction-diffusion equation, in particular interesting unstable ones. 97 3.3. NUMERICAL EXPERIMENTS A reaction-diffusion equations due to Mimura An exciting and difficult reaction-diffusion problem of two dependent functions is a problem by M. Mimura [64] which arises from the context of planktonic prey and predator models in biology. This problem is given below and we briefly call it Mimura’s problem. 1 1 ′′ − u(t)2 u(t) − u(t) v(t) = 0, 20 u (t) + 9 35 + 16u(t) = 0, 4 v ′′ (t) − 1 + 25 v(t) v(t) + u(t) v(t) u̇(0) = u̇(5) = v̇(0) = v̇(5) = 0, (3.47) 0 ≤ u(t) ≤ 14, 0 ≤ v(t) ≤ 14, ∀t ∈ [0, 5] . In [64] the problem is analyzed, and the existence of continuous solutions is shown in [82]. In order to construct a POP of the type (3.26), we consider different objective functions: PN F1 (u, v) = −u⌈ N ⌉ , F2 (u, v) = − i=1 ui , F3 (u, v) = −u2 , 2 (3.48) P F4 (u, v) = −uN −1 , F5 (u, v) = −u2 − uN −1 , F6 (u, v) = N i=1 (ui + vi ). First, we apply the SDPR method with ω = 3 and N = 5. In order to confirm the numerical results obtained for this very coarse grid, we apply PHoM [31], which is a C++ implementation of the polyhedral homotopy continuation method for computing all isolated complex solutions of a polynomial system of equations, to the system of discretized PDEs. In that case the dimension n of the polynomial system equals 6, as there are 2 unknown functions with 3 interior grid points each. PHoM finds 182 complex, 64 real and 11 nonnegative real solutions. Varying the upper and lower bounds for u2 and u4 and choosing one of the functions F1 , . . . , F5 as an objective function, all 11 solutions are detected accurately (| ǫsc |< 1e − 7) by the SDPR method, as enlisted in Table 3.9. u2 4.623 4.607 0.259 5.683 6.274 0.970 0.297 0.962 0.304 0.939 5.000 u3 6.787 6.930 6.930 2.971 0.177 7.812 7.932 7.932 8.045 6.787 5.000 u4 0.939 0.259 4.607 5.683 6.274 0.970 0.966 0.297 0.304 4.623 5.000 v2 9.748 9.737 5.166 10.388 10.638 5.735 5.230 5.729 5.234 5.659 10.000 v3 10.799 10.831 10.831 8.248 6.404 10.94 10.94 10.94 10.94 10.80 10.000 v4 5.659 5.166 9.737 10.388 10.638 5.735 5.729 5.230 5.234 9.748 10.000 objective F3 F3 F2 F3 F3 F3 F4 F3 F1 F4 F2 ubdu2 5 5 0.5 6 7 2 0.5 2 14 2 14 ubdu4 1.5 0.5 6 6 7 2 2 0.5 14 14 14 Table 3.9: SDPR method solutions of (3.47) for N = 5. The confirmation of our SparsePOP results by PHoM encourages us to solve Mimura’s problem for a higher discretization. Relaxation order ω = 3 is necessary to obtain an accurate solution in case N = 7 (Table 3.10, row 1). The upper bounds for u2 and uN −1 are chosen to be 1. When extending the grid size from 7 to 13, the accuracy of the SDPR solution deteriorates. Also, if we choose ω = 2 for the initial application of the SDPR method, or if Newton’s method is applied with another arbitrary starting point, or if we start for instance with N = 5 or N = 9, it is not possible to get an accurate solution. One possibility to overcome these difficulties is to start the grid-refining method with strategy 3b on a finer grid. We obtain a highly accurate stable solution 2teeth when we start with N = 7 and F2 as objective function, and a highly accurate stable solution 2,3peak when we start with N = 25 and F5 as objective function. See Table 3.10 and Figure 3.15. It seems reasonable to state that the SDPR method provides an appropriate initial guess for Newton’ s method, which leads to accurate solutions for sufficiently high discretizations. 98 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS SDP relaxation Dual Grid-ref 3b Grid-ref 3b Grid-ref 3b Grid-ref 3b Grid-ref 3b Grid-ref 3b Dual Grid-ref 3b Grid-ref 3b Grid-ref 3b QOP no no no no no no no no no no no Local solver Newton Newton Newton Newton Newton Newton Newton Newton Newton Newton Newton obj F2 F5 ubdu1 14 14 14 14 14 14 14 14 14 14 14 N 7 13 25 49 97 193 385 26 51 101 401 solution 2teeth 2,3peak ǫsc -5e-9 -2e+0 -2e-6 -1e-12 -5e-12 -2e-4 -5e-5 -1e-1 -5e-2 -2e-15 -6e-16 me -2.2 2.7 -0.9 0.1 0.2 0.3 0.2 2.09 -0.18 -0.07 -0.07 Table 3.10: Results of grid-refining strategy 3b. 12 12 10 10 8 8 6 6 u v 4 4 u v 2 0 2 0 0.5 1 1.5 2 2.5 x 3 3.5 4 4.5 5 0 0 0.5 1 1.5 2 2.5 x 3 3.5 4 4.5 5 Figure 3.15: Unstable solution 2teeth (left) and stable solution 2,3peak (right). As the most powerful approach we apply the grid-refining method with strategy 3b, ω1 = 3 and ωk = 2 for k ≥ 2. We obtain the highly accurate stable solutions 3peak and 4peak, that are documented in Table 3.11 and pictured in Figure 3.16. As objective function for the POPs to be solved in each iteration we choose the function FM from (3.31). Another way to attempt finer grids directly is to transform the POP derived from 3.47 into a QOP, and to apply both primal and dual SDP relaxations to that QOP. As reported in Table 3.11, the total computation time in the case N = 51 is reduced by two magnitudes under this method. A highly accurate solution is obtained when applying SQP as local solver in the SDPR method. Finally, we yield various stable and unstable solutions to Mimura’s problem, when choosing different functions as objective F and when tightening or loosening ubdu 1 . By the SDPR method we obtain the stable solutions 3peak, 4peak, 2,3peak and the unstable solutions 2teeth, peak3unstable, peak4unstable, 2valley. Reaction-diffusion equations from collision processes Another interesting class of reaction-diffusion equations arises from collision processes of particle-like patterns in dissipative systems. Various different input-output relations such as annihilation, repulsion, fusion, and even chaotic dynamics are observed after collision of these patterns. The reaction-diffusion equations 99 3.3. NUMERICAL EXPERIMENTS SDP Relaxation Dual Grid-ref 3a Grid-ref 3a Grid-ref 3a Dual Grid-ref 3a Grid-ref 3a Grid-ref 3a Dual Grid-ref 3b Dual Grid-ref 3b Dual Grid-ref 3b Dual Primal QOP no no no no no no no no no no no no no no yes yes Local solver Newton Newton Newton Newton Newton Newton Newton Newton SQP SQP SQP SQP SQP SQP SQP SQP obj F5 FM FM FM F1 FM FM FM F1 F1 F1 F1 F2 F2 F6 F6 ubdu1 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 14 14 0.5 0.5 0.5 0.5 N 26 51 101 201 26 51 101 201 51 201 51 201 51 201 51 51 solution ǫsc -2e-1 -4e-2 -4e-4 -3e-11 -1e-3 -4e-3 -3e-16 -2e-11 -5e-13 -1e-12 -1e-10 -4e-12 -1e-2 -3e-16 -1e-13 -1e-9 4peak 3peak 3peak peak3unstable peak4unstable 2valley 2valley me 2.09 -0.05 -0.02 -0.02 -0.12 -0.08 -0.08 -0.07 -0.07 -0.07 0.70 0.70 1.62 1.43 2.10 2.10 tC 203 224 383 1082 270 348 511 1192 470 501 393 427 806 868 16 8 Table 3.11: Results of grid-refining strategy 3a. describing the collision processes have special unstable solutions, so-called scattors, which are difficult to detect and attract lots of interest [73, 74, 75, 76]. We show how scattors are detected by the SDPR method. Gray-Scott model in 1D: As an first example we consider the stationary equation of a Gray-Scott model for the dynamics of two traveling pulses in one dimension from [74]: Du uxx (x) − u(x)v(x)2 + f (1 − u(x)) Dv vxx (x) + u(x)v(x)2 − (f + k)v(x) = 0 ∀ x ∈ [0, 4] , = 0 ∀ x ∈ [0, 4] , (3.49) where Du > 0, Dv > 0, and f > 0 and k > 0 are two parameters related to inflow and removal rate of chemical species. Moreover, we impose Neumann boundary conditions. The existence and the shape of stable solutions and scattors doublePeak depends heavily on the choice for f and k. We set the parameters to Du = 5e − 5, Dv = 2.5e − 5, f = 0.0198 and k = 0.0497859. For this setting a scattor withP positive eigenvalues (λ1 , λ2 , λ3 ) = (0.0639, 0.0638, 0.0023) was reported in [74]. We choose F (u, v) = − N i=1 ui , lbdu = lbdv = 0, ubdu = 1.0, ubdv = 0.8, lbdu 1 = lbdu N = 0.4, ubdu 1 = ubdu N = 0.6 and ubdu N = 0.9. 2 Applying the SDPR method with these settings yields the scattor doublePeak as reported in Table 3.12 and pictured in Figure 3.18. SDP Relaxation Dual Grid-ref 3b Grid-ref 3b QOP yes yes yes Local solver SQP SQP SQP ω 2 N 20 640 1280 solution doublePeak doublePeak λ1 0.0441 0.0639 0.0638 λ2 0.0434 0.0638 0.0637 λ3 0.0023 0.0023 tC 32 469 1247 Table 3.12: Results of the SDPR method for (3.49). In (3.49) the shape and number of scattors depends on the choice of the two parameters F and k. It is well known, that if we fix f there is a bifurcation point k0 , to be precise, there are no scattors for (3.49) if k < k0 and several scattors if k > k0 . For a bifurcation analysis by the SDPR method we fix f = 0.0270. 100 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS 12 12 u v 10 10 8 8 6 6 4 4 2 2 0 0 0.5 1 1.5 2 2.5 x 3 3.5 4 4.5 0 5 0 0.5 1 1.5 2 2.5 x 3 3.5 4 4.5 5 Figure 3.16: Stable solutions 4peak and 3peak. 12 12 u v u v 10 10 8 8 6 6 4 4 2 2 0 0 0.5 1 1.5 2 2.5 x 3 3.5 4 4.5 0 5 0 0.5 1 1.5 2 2.5 x 3 3.5 4 4.5 5 Figure 3.17: Unstable solutions peak4unstable and peak3unstable. Moreover, we define the norm of a numerical solution u by k u k= We define F (u, v) = − as follows: PN i=1 N X i=1 u2i ! 12 . u2i as objective function for the SDPR method. Bounds lbd and ubd are chosen lbdu i ( N 0.5, if i ∈ {1, . . . , ⌈ 10 ⌉} ∪ {⌈ 9N 10 ⌉, . . . , N } , = 0, else (3.50) ubdu = 0.8, lbdv = 0 and ubdv = 0.8. If we apply the SDPR method with ω = 2 and N = 256 to (3.49) for k ≤ 0.05281, we do not obtain any nontrivial solution. If we apply the SDPR method for the same settings to (3.49) with k increasing from 0.05282 to 0.0535 we obtain a scattor (u, v)1 with k u1 k increasing from 101 3.3. NUMERICAL EXPERIMENTS 1 0.9 0.8 u v 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 1.5 2 2.5 3 3.5 4 Figure 3.18: SDPR method detects double peak scattor for (3.49). 0.1682 to 0.1843. To obtain a second scattor requires to tighten the bounds lbdu and ubdu : ( N 0.4, if i ∈ {1, . . . , ⌈ 10 ⌉} ∪ {⌈ 9N 10 ⌉, . . . , N } , lbdui = 0, else ( N ⌉} ∪ {⌈ 9N 0.6, if i ∈ {1, . . . , ⌈ 10 10 ⌉, . . . , N } . ubdui = 0.8, else (3.51) The bounds for v remain unchanged. Applying the SDPR method with the tighter bounds (3.51) for k = 0.0535 yields a second solution (u, v)2 with k u2 k= 0.1566 6= 0.1843 =k u1 k. However, it is not possible to obtain accurate approximations for the second solution when applying the SDPR method for smaller choices of k. Therefore we apply a continuation technique based on the SDPR method to obtain the second solution for smaller k: Set k̃ = 0.0535, choose some stepsize ∆k and apply the SDPR method to (3.49) for k = k̃ − ∆k with objective function Gk̃ defined by Gk̃ (u, v) = N X i=1 ui − uk̃i 2 , where uk̃ solution of (3.49) for k = k̃. Update k̃ = k̃ − ∆k and iterate. Following this procedure we obtain (u, v)2 for k decreasing from 0.0535 to 0.05281 and k u2 k increasing from 0.1566 to 0.1655. Thus, the bifurcation point of (3.49) is k0 ≈ 0.05281 for f = 0.0270. The results of the SDPR method are illustrated in the bifurcation diagram in Figure 3.19. Three-component reaction-diffusion equation tion from [74]: Consider another one-dimensional steady state equa- Du uxx (x) + 2u(x) − u(x)3 − κ3 v(x) − κ4 w(x) + κ1 1 τ (Dv vxx (x) + u(x) − v(x)) 1 θ (Dw wxx (x) + u(x) − w(x)) −2 ≤ u(x), v(x), w(x) =0 =0 =0 ≤2 ∀ x ∈ [0, 0.5] , ∀ x ∈ [0, 0.5] , ∀ x ∈ [0, 0.5] , ∀ x ∈ [0, 0.5] , (3.52) under Neumann boundary conditions. We set the parameters to Du = 5e − 6, Dv = 5e − 5, Dw = 1e − 2, κ1 = −7, κ3 = 1, κ4 = 8.5, τ = 16.1328 and θ = 1. In [74] the existence of two scattors, twin-horn and fusion, is shown for this setting. We choose F (u, v, w) = −u⌈ N ⌉ . fusion has one positive eigenvalue and 2 102 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS 0.185 0.18 ||u1|| ||u2|| 0.175 0.17 0.165 0.16 0.155 0.0528 0.0529 0.053 0.0531 0.0532 0.0533 0.0534 0.0535 k Figure 3.19: Bifurcation diagram by the SDPR method (left) compared to the one (right) from [100]. twin-horn three positive eigenvalues (λ1 , λ2 , λ3 ) = (0.9069, 0.1297, 0.0138). Applying the SDPR method with SQP as local solver starting from N = 40 and subsequent application of the grid-refining method with strategy 3b, we obtain a highly accurate approximation of the scattor fusion pictured in Figure 3.20. 1.5 1 0.5 0 −0.5 −1 −1.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Figure 3.20: SDPR method detects fusion for (3.52), with u (blue), v (green) and w (red). Swift-Hohenberg equation Another type of reaction-diffusion equations arising from modeling pattern formations is given by SwiftHohenberg equations [80]. Swift-Hohenberg equations are interesting from the point of view of pattern formation because they have many qualitatively different stationary solutions. Differential equations with many solutions are of particular interest for applying the SDPR method. We examine a stationary SwiftHohenberg equation from [80]: −uxxxx(x) − 2ux (x) − (1 − α)u(x) − u(x)3 u(0) = u(L) = uxx (0) = uxx (L) = 0 ∀ x ∈ [0, L], = 0, (3.53) where α ∈ R. It is known that the shape and number of solutions depends on the choice for L. For our numerical analysis we set the parameters to L = 9 and α = 0.3. The forth order derivative is approximated 103 3.3. NUMERICAL EXPERIMENTS by uxxxx(xi ) ≈ 1 (ui+2 − 4ui+1 + 6ui − 4ui−1 + ui−2 ) . h4x We consider the following functions as objective functions: PN PN F1 (u) = − i=1 ui , F2 (u) = i=1 ui , F3 (u) = −u2 , F5 (u) = −u⌈ 2N ⌉ , F6 (u) = −u⌈ N ⌉ , F4 (u) = −u⌈ N ⌉ 3 3 (3.54) 4 We apply the SDPR method with ω = 3 for N = 40 and varying objective functions, and obtain five different solutions for (3.53) as reported in Table 3.13 and pictured in Figure 3.21. SDP relaxation Dual Dual Dual Dual Dual POP to QOP no no no no no Local Solver SQP SQP SQP SQP SQP Objective F1 F2 F3 F4 F5 ǫsc -1e-9 -3e-15 -3e-10 -3e-10 -1e-13 tC 29 32 31 69 58 Table 3.13: Results of the SDPR method with ω = 3, N = 40 and varying objective function. In a next step we investigate a more systematic approach to enumerate solutions of (3.53). Applying Algorithm 3.1 does not yield an accurate enumeration of solutions for (3.53). The numerical accuracy of the solutions deteriorates from the second iteration. This may be explained by the fact, that the relaxation of the quadratic constraints added in step 3 of Algorithm 3.1 is too weak to provide a good starting point for the initial solver. We therefore consider a variant of Algorithm 3.1. Instead of adding constraint (k−1) 2 (ui − ui to the POP, we consider 2 Pn i=1 bi ) ≥ ǫk for all i with bi = 1 POPs where we add the constraints (k−1) − ǫki (k−1) + ǫki ui ≤ ui or ui ≥ ui for every i with Pn bi = 1. It is clear, that the solution with kth smallest objective value is feasible for exactly one of the 2 i=1 bi POPs. This procedure has the advantage that the added linear constraint remains hard under the SDP relaxation, it has the disadvantage that many SDPs need to be solved in course of the algorithm. Therefore, we consider this modified enumeration algorithm for the SDPR method with small relaxation order. We apply this modified enumeration algorithm and SDPR method with N = 50, bk := b := (0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, . . ., 0), ǫk := ǫ := 0.1, POP to QOP transformation and ω = 2, F6 as objective and 6 iterations of the enumeration algorithm. In its first five iterations we obtain the same five solutions as those obtained by the SDPR method when choosing different objective functions, the output of the sixth iteration is the same as for the fifth one, although it is not a feasible solution for the POP from the sixth iteration. Our numerical results are reported in Table 3.14 and pictured in Figure 3.21. Note, applying the modified enumeration algorithm requires a longer time to obtain the five solutions to (3.53). However, enumerating the five solutions one by one with respect to the same objective function is a more systematic approach than choosing five arbitrary objective functions. 3.3.4 Differential algebraic equations The class of PDE problems (3.14) contains so called differential algebraic equations (DAE). These are differential equations, where the derivatives of several unknown functions do not occur explicitly. We 104 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS k 0 1 2 3 4 5 SDP relaxation Dual Dual Dual Dual Dual Dual POP to QOP yes yes yes yes yes yes Local Solver SQP SQP SQP SQP SQP SQP ǫsc -4e-15 -9e-11 -5e-10 -9e-13 -1e-11 -1e-11 F6 (u(k) ) -0.43 -0.17 0.00 0.17 0.43 0.43 tC 30 153 293 400 500 593 Table 3.14: Results of the modified enumeration algorithm and SDPR method with ω = 2, N = 50 and F6 as objective. 0.6 0.6 0.4 0.4 0.2 0.2 0 0 −0.2 −0.2 F 1 −0.4 −0.4 (0) F u 2 u(1) F 3 (2) u F 4 −0.6 −0.6 (3) u F 5 (4) u −0.8 0 1 2 3 4 5 6 7 8 −0.8 9 0 1 2 x 3 4 5 6 7 8 9 x Figure 3.21: SDPR method with varying objective for N = 40 (left) and modified enumeration algorithm with SDPR method for N = 50 (right). demonstrate for the following example that the SDPR method can be applied to solve DAE as well. Consider the DAE problem u̇1 (x) 0 0 u1 (0) = u3 (x), = u2 (x) (1 − u2 (x)) , = u1 (x) u2 (x) + u3 (x) (1 − u2 (x)) − x, = u0 . ∀ x ∈ [0, T ] (3.55) It is easy to see that two closed-form solutions u1 and u2 are given by u1 (x) u2 (x) = u0 + x2 2 , T 0, x T = (x, 1, 1) x ∈ [0, T ], x ∈ [0, T ]. We choose lbd ≡ 0 and ubd ≡ 10 for each function u1 , u2 and u3 and define two objective functions F1 and F2 , Nx Nx X X u2i . F2 (u) = u1i , F1 (u) = i=1 i=1 First we choose u0 = 0 and apply the SDPR method with F2 as an objective, and we obtain an highly accurate approximation for u2 , which is documented in Table 3.15 and Figure 3.22. 105 3.3. NUMERICAL EXPERIMENTS objective F2 F2 F1 F1 F1 F1 F1 F1 F1 F1 ω 2 2 2 2 2 2 2 2 3 4 Nx 100 200 200 200 10 20 30 40 30 30 u0 0 0 1 2 0.5 0.5 0.5 0.5 0.5 0.5 ǫobj 4e-10 3e-9 8e-9 3e-10 3e-10 3e-10 8e-10 7e-8 9e-9 8e-9 ǫsc -4e-10 -3e-9 -2e-6 -4e-8 -3e-7 -9e-6 -3e-3 -1e-1 -2e-3 -6e-4 tC 29 122 98 107 4 9 15 24 51 210 Table 3.15: Results of SDPR method for (3.55) with T = 2. 2 3 1.8 2.5 u1 1.6 u2, u3 1.4 u1(x), u2(x), u3(x) u1(x), u2(x), u3(x) 2 1.2 1 0.8 1.5 1 u1 0.6 u2 0.4 u3 0.5 0.2 0 0 0.2 0.4 0.6 0.8 1 x 1.2 1.4 1.6 1.8 2 0 0 0.2 0.4 0.6 0.8 1 x 1.2 1.4 1.6 1.8 2 Figure 3.22: Solutions u2 (left) and u1 (right) of DAE problem (3.55). Next we apply the SDPR method with F1 as objective. For u0 ∈ {1, 2} highly accurate approximations of u1 are obtained. An interesting phenomenon is observed in case u0 is small. For instance, if we choose u0 = 0.5 and ω = 2, we get a highly accurate solution for Nx = 10. But, as we increase Nx stepwise to 40, the accuracy decreases, although the relaxation order remains constant. For numerical details see Table 3.15. This effect can be slightly compensated by increasing ω, as demonstrated for the case Nx = 30. But due to the limited capacity of current SDP solvers it is not possible to increase ω as much as needed to obtain a high accuracy solution. However, we obtain highly accurate approximations to both solutions of (3.55) by the SDPR method even without applying a locally fast convergent method. 3.3.5 The steady cavity flow problem One of the most challenging present PDE problems is the numerical analysis of the Navier-Stokes equations. As a first step to attempt this class of PDE problems by the SDPR method we consider the steady cavity flow problem, which contains a steady state version of the Navier-Stokes equations. The steady cavity flow problem is a simple model of a flow with closed streamlines and is used for examining and validating numerical solution techniques in fluid dynamics. Although it has been discussed in the literature of numerical analysis of fluid mechanics (see, e.g., [40], [12], [32], [14], [98]), it is still an interesting problem to a number of researchers for a range of Reynolds numbers. The setting of the steady cavity flow problem is the following. 106 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS Let (v1 (x, y, t), v2 (x, y, t)) be the velocity of the two dimensional cavity flow of an incompressible fluid on the cavity region ABCD with the coordinates A = (0, 1), B = (0, 0), C = (1, 0), D = (1, 1). A - B D C It follows from the continuity equation of the incompressible fluid (preservation of the mass) that there exists a function ψ(x, y, t) such that ∂ψ ∂ψ = −v2 , = v1 . ∂x ∂y ∂v1 ∂v2 ∂x + ∂y =0 (3.56) ~ = rot v is called the vorticity. Since the last coordinate of v is 0, φ ~ can be written as Put v = (v1 , v2 , 0). φ (0, 0, φ(x, y, t)). The continuity equation and the Navier-Stokes equation (preservation of the momentum) can be written as follows in terms of ψ and φ. ∆ψ = −φ ∂ψ ∂φ ∂ψ ∂φ 1 ∂φ = − + ∆φ, ∂t ∂y ∂x ∂x ∂y R (3.57) (3.58) where the parameter R is called the Reynolds number. The steady cavity flow problem is (3.57) and (3.58) with the steady condition condition v1 (0, y) = v1 (x, 0) = v1 (1, y) = 0 v2 (0, y) = v2 (x, 0) = v2 (1, y) = 0 v1 (x, 1) = s, v2 (x, 1) = 0 on AB, BC, CD on AB, BC, CD on AD ∂φ ∂t = 0 and the boundary (3.59) (3.60) (3.61) Here s is the velocity of the stream out of the cavity ABCD. We set the boundary velocity at AD to s := 1. Due to its boundary conditions PDE problem (3.56), (3.57), (3.58), (3.59) - (3.61) is not of form (3.14). Therefore, we need to follow a specialized strategy to discretize this problem via a finite different scheme. We devide the square ABCD into a N × N mesh and define h := N1−1 . We translate the boundary conditions for v1 and v2 into boundary conditions for ψ and φ. It follows from (3.59), (3.60), (3.61) that the function ψ is constant on the boundaries AB, BC, CD, DA. Since ψ is continuous, we suppose that ψ = 0 on the boundaries. The boundary condition for φ is a little complicated. We derive it from the discussion in [98], p 162: Let us consider the case of the boundary AD first. We take a mesh point M on AD. Let P be the mesh point inside the cavity adjacent to M and P ′ the mirror image of P with respect to AD. We supposed that the size of the mesh is h. P′ M A h P D 107 3.3. NUMERICAL EXPERIMENTS We denote the value of ψ at the point P by ψ(P ) or ψP . Moreover, we have −φ(M ) = ∆ψ(M ) = ψyy ≈ . We need to determine the value of ψP ′ to get an approximate value of φ at M . By using the ψP ′ −ψP holds. Then, ψP ′ ≈ 2h + ψP . Therefore, central difference approximation, s = 1 = v2 = ∂ψ ∂y (M ) ≈ 2h we have 2ψP + 2h φM ≈ − . (3.62) h2 Analogously, we obtain 2ψP φM ≈ − 2 (3.63) h when M is a grid point on AB or BC or CD and P is the adjacent internal grid point of M . From this discussion follows, we obtain the following finite difference scheme for the steady cavity flow problem. ψP −2ψM +ψP ′ h2 1 gi,j (ψ, φ) = 0 2 gi,j (ψ, φ) ∀ 2 ≤ i, j ≤ N − 1, ∀ 2 ≤ i, j ≤ N − 1, =0 ψ1,j = ψN,j = 0 ψi,1 = ψi,N = 0 ψ φ1,j = −2 h2,j 2 ψ φN,j = −2 Nh−1,j 2 ψ φi,1 = −2 hi,2 2 ψ +h φi,N = −2 i,Nh−1 2 where 1 gi,j (ψ, φ) := ∀ j ∈ {1, . . . , N } , ∀ i ∈ {1, . . . , N } , ∀ j ∈ {1, . . . , N } , ∀ j ∈ {1, . . . , N } , ∀ i ∈ {1, . . . , N } , ∀ i ∈ {1, . . . , N } , (3.64) (3.65) (3.66) −4φi,j + φi+1,j + φi−1,j + φi,j+1 + φi,j−1 +R 4 (ψi+1,j − ψi−1,j )(φi,j+1 − φi,j−1 ) −R 4 (ψi,j+1 − ψi,j−1 )(φi+1,j − φi−1,j ), −4ψi,j + ψi+1,j + ψi−1,j + ψi,j+1 +ψi,j−1 + h2 φi,j . 2 gi,j (ψ, φ) := We call the polynomial system (3.64), (3.65), (3.66) the discrete steady cavity flow problem denoted as DSCF(R, N ). It depends on two parameters, the Reynolds number R and the discretization N of the cavity region ABCD = Ω = [0, 1]2 . Its dimension is given by n = 2(N − 2)2 . Remark 3.2 We conjecture that the discrete cavity flow problem DSCF(R, N ) has finite complex solutions. In other words, it defines a zero-dimensional ideal. We checked this conjecture up to N = 5 by Gröbner basis computation. DSCF(R, N ) is a sparse, polynomial system which we apply the SDPR method and Algorithm 3.1 to. The choice for F in 3.26 is motivated by the fact, that one is interested in the solution to the PDE problem which minimizes the kinetic energy given by Z Z ABCD ∂ψ ∂y 2 + −∂ψ ∂x 2 dxdy. (3.67) Thus, by discretizing (3.67) we yield the following function as a canonical choice for F : F (ψ, ω) = 1 4 X 2 2 2 2 ψi+1,j + ψi−1,j + ψi,j+1 + ψi,j−1 2≤i,j≤N −1 (3.68) −2ψi+1,j ψi−1,j − 2ψi,j+1 ψi,j−1 . We denote the optimization problem to minimize F subjected to DSCF(R, N ) as he steady cavity flow optimization problem CF(R, N ). It is characterized by a simple proposition. Proposition 3.4 a) CF(0, N ) is a convex quadratic program for any N . 108 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS b) CF(R, N ) is non-convex for any N , if R 6= 0. Proof: a) In case 0 all constraints are linear. Furthermore, the objective function can be written as P R = 1 2 + Fi,j , where F = i,j Fi,j 1 Fi,j (ψ, ω) = T ψi−1,j ψi+1,j 2 −2 −2 2 ψi−1,j ψi+1,j . 2 −2 It follows that is convex as positive semidefinite with eigenvalues 0 and 4. The −2 2 2 convexity of Fi,j follows analoguously. Thus, F can be written as a sum of convex function and is therefore convex as well. The proposition follows. 1 Fi,j 1 b) In case R 6= 0, the equality constraint function gi,j is indefinite quadratic. Thus, CF(R, N ) is a non-convex quadratic program. Solving CF(R, N ) by the SDPR method First, we apply the SDPR method with ω = 1 and Newton’s method as local solver to CF(100, N ). Highly accurate solutions with ǫsc < 1e − 10 are obtained for N ∈ {10, 15, 20}. By applying the grid-refining method we succeed in extending the solutions to grids of size 30 × 30 and 40 × 40, as pictured for N = 40 in Figure 3.23 and reported in Table 3.16. Thus, it seems reasonable to conclude, that the minimal energy solution of CF(100, N ) converges to a continuous solution of the steady cavity flow problem for N → ∞. The discrete steady cavity flow problem has multiple solutions. It is an advantage of the SDPR method to show that the minimal kinetic energy solution u⋆ (N ) of CF(R, N ) converges to an analytic solution for N → ∞. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Figure 3.23: (v1 , v2 ) for solution u of CF(100, 40). N 10 15 20 30 40 ω 1 1 1 1 1 ǫsc 4e-11 6e-16 6e-16 4e-11 4e-11 tC 14 255 948 1759 4156 F (u⋆ ) 0.0169 0.0313 0.0409 0.0503 0.0554 Table 3.16: Results for CF (100, N ) for increasing N . Second, we apply the SDPR method to CF(R, N ) for a much larger R. For example we examine CF (10000, N ) for N ∈ {8, . . . , 18}. For all tested discretizations we were able to find accurate solutions by the SDPR method with ω = 1 and SQP as local solver, c.f. Table 3.17 and Figure 3.24. 109 3.3. NUMERICAL EXPERIMENTS N 8 10 12 14 16 18 ω 1 1 1 1 1 1 ǫsc 2e-7 3e-10 1e-7 5e-9 4e-12 2e-8 F (u(k) ) 1.5e-6 3.2e-6 6.0e-6 1.1e-5 1.9e-5 3.9e-5 tC 7 21 49 99 199 501 Table 3.17: Results for CF (10000, N ) for increasing N . 1 1 1 0.9 0.9 0.9 0.8 0.8 0.8 0.7 0.7 0.7 0.6 0.6 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0.3 0.3 0.3 0.2 0.2 0.2 0.1 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 1 1 1 0.9 0.9 0.9 0.8 0.8 0.8 0.7 0.7 0.7 0.6 0.6 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0.3 0.3 0.3 0.2 0.2 0.2 0.1 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 Figure 3.24: Solutions of CF (10000, N ) of the SDPR method for N = 8 (left, top), N = 10 (center, top), N = 12 (right, top), N = 14 (left, bottom), N = 16 (center, bottom) and N = 18 (right, bottom). If we compare the pictures in Figure 3.24, it seems the SDPR(1) solution of CF (10000, 1, N ) evolves into some stream-like solution for increasing N . However, unlike the solutions of CF (100, 1, N ), we have not been able to expand this solution to a grid of higher resolution by the grid-refinement method. Therefore, it is possible the solution pictured in Figure 3.24 is a fake solution, which confirms that the Steady Cavity Flow problem becomes a hard problem for increasing Reynolds number. Enumerating the solutions of DSCF(R, N ) A further interesting question is to find all solutions of the cavity flow problem, in particular for large Reynolds number. Therefore, we examine the efficiency of Algorithm 3.1 for enumerating the solutions of DSCF(R, N ) with respect to their discretized kinetic energy. For the parameter bk ∈ {0, 1}n to be chosen in each iteration of Algorithm 3.1 we restrict ourselves to the case where bk is given by ( 1, if i ∈ {1, . . . , bk1 } ∪ { n2 + 1, . . . , n2 + bk2 }, k bi = 0, else. 110 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS 1 1 1 0.9 0.9 0.9 0.8 0.8 0.8 0.7 0.7 0.7 0.6 0.6 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0.3 0.3 0.3 0.2 0.2 0.2 0.1 0.1 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Figure 3.25: (v1 , v2 ) for solutions u(0) (left), u(1) (center) and u(2) (right) of CF(4000, 5) on the interior of [0, 1]2 . Thus bk is defined by the two parameters bk1 , bk2 ∈ {1, . . . , n2 }. The parameters ǫk1 and ǫk2 are corresponding to the constraints imposed by bk1 and bk2 , respectively. CF(4000,5): In a first setting we choose the discretization N = 5, i.e. the dimension is n = 2 · 32 = 18. This dimension is small enough to apply the Gröbner basis method to determine all complex solutions of DSCF(R, N ). Therefore, we are able to verify whether the solutions provided by Algorithm 3.1 are optimal. The computational results are given in Table 3.18. Comparing the solutions of the SDPR method to all k 0 1 2 ω 1 1 1 ǫk1 1e-3 1e-3 bk1 3 3 bk2 0 0 tC 2 5 8 ǫsc 2e-10 5e-4 5e-4 F (u(k) ) 4.6e-4 6.3e-4 1.0e-3 solution u(0) u(1) u(2) Table 3.18: Results of Algorithm 3.1 for CF(4000, 5). solutions of the polynomial system obtained by polyhedral homotopy method or Gröbner basis method, it turns out that the solutions u(0) , u(1) and u(2) indeed coincide with the three smallest energy solutions u(0)⋆ , u(1)⋆ and u(2)⋆ . The velocities (v1 , v2 ) derived from these three solutions via (3.56) are displayed in Figure 3.25. Note, that the third smallest energy solution u(2) shows a vortex in counter-clockwise direction, which may indicate that this solution is a fake solution. CF(20000,7): We apply Algorithm 3.1 with ω = 1 to CF(20000, 7) and obtain the results in Table 3.19. The two parameter settings (ǫ11 , b11 ) = (1e − 3, 1) and (ǫ11 , b11 ) = (1e − 6, 5) are not sufficient to obtain an other solution than u(0) , whereas (ǫ11 , b11 ) = (1e − 5, 5) yields u(1) , a solution of larger energy. After another iteration with (ǫ21 , b21 ) = (1e − 5, 5) we obtain a third solution u(3) of even larger energy. k 0 1 1 1 2 ω 1 1 1 1 1 ǫk1 1e-3 1e-6 1e-5 1e-5 bk1 1 5 5 5 bk2 0 0 0 0 tC 2 5 5 9 14 ǫsc 3e-7 5e-4 6e-6 5e-6 5e-6 F (u(k) ) 3.4e-4 3.4e-4 3.4e-4 5.9e-4 5.2e-3 solution u(0) u(0) u(0) u(1) u(2) Table 3.19: Results of Algorithm 3.1 for CF(20000, 7). 111 3.3. NUMERICAL EXPERIMENTS 1 1 1 0.9 0.9 0.9 0.8 0.8 0.8 0.7 0.7 0.7 0.6 0.6 0.6 0.5 0.5 0.5 0.4 0.4 0.4 0.3 0.3 0.3 0.2 0.2 0.2 0.1 0.1 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Figure 3.26: (v1 , v2 ) for solutions u(0) (left), u(1) (center) and u(2) (right) of CF(20000, 7) on [0, 1]2 . It is interesting to observe in Figure 3.26 that u(1) and u(2) are one-vortex solutions, whereas there seems to be no vortex in the smallest energy solution u(0) . CF(40000,7): Next, we examine CF (40000, 7), which is a good example to demonstrate that solving DSCF(R, N ) and CF(R, N ) are becoming more difficult for larger Reynolds numbers. As for the previous problem, the dimension of the POP is n = 50, which is too large to be solved by Gröbner basis. Our computational results are reported in Table 3.20. k 0 1 2 3 0 ω 1 1 1 1 2 ǫk1 5e-6 5e-6 8e-6 - bk1 5 5 5 - bk2 0 0 0 - tC 3 7 11 16 5872 ǫsc 2e-7 6e-9 3e-6 5e-6 8e-10 F (u(k) ) 3.4e-4 7.3e-4 5.9e-4 2.3e-4 2.6e-4 solution u(0) (1) u(1) (1) u(2) (1) u(3) (1) u(0) (2) Table 3.20: Results of Algorithm 3.1 for CF (40000, 7). Solution u(2) (1) is of smaller energy than u(1) (1), and u(3) (1) is even of smaller energy than u(0) (1). Thus, unlike the solutions for CF(20000,7) reported in Table 3.19, the solutions of CF(40000,7) are not enumerated in the correct order. This phenomenon can be explained by the fact, that the SDP relaxation with ω = 1 is not tight enough to yield a solution that converges to u⋆ under the local optimization procedure. The energy of u(0) (2) obtained by SDPR(2) is smaller than the one of u(0) (1), but it is not the global minimizer as well. In fact, Algorithm 3.1 with ω = 1 generates a better solution u(3) (1) (with smaller energy) in 3 iterations requiring 16 seconds computation time, compared to solution u(0) (2) obtained by applying the SDPR method with ω = 2 to CF (40000, 7) requiring 5872 seconds. Thus, despite failing to enumerate the smallest energy solutions in the right order with ω = 1, applying the enumeration algorithm with relaxation order ω = 1 is far more efficient than the original sparse SDP relaxation (2.18) with ω = 2 for approximating the global minimizer of POP (3.26). It is a future problem to make this construction systematic. Alternative finite difference scheme ∂φ ∂ψ ∂φ To derive DSCF(R, N ) we discretize the Jacobian ∂ψ ∂y ∂x − ∂x ∂y by the standard central difference scheme. Arakawa [3] showed that the standard central difference scheme does not keep important physical invariants. Therefore, Arakawa proposed an alternative finite difference discretization for the Jacobian, that is shown to preserve those invariants. We use this alternative scheme to derive an alternative discrete steady cavity flow problem ADSCF(R, N ) and solve it via the SDPR method. In ADSCF(R, N ), the finite difference 112 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS approximation for ∂ψ ∂φ ∂y ∂x − ∂ψ ∂φ ∂x ∂y is replaced by ∂ψ ∂φ ∂ψ ∂φ ∂y ∂x − ∂x ∂y (xi , yj ) ≈ 1 − 12h 2 [(φi,j−1 + φi+1,j−1 − φi,j+1 − φi+1,j+1 ) (ψi+1,j + ψi,j ) − (φi−1,j−1 + φi,j−1 − φi−1,j+1 − φi,j+1 ) (ψi,j + ψi−1,j ) + (φi+1,j + φi+1,j+1 − φi−1,j − φi−1,j+1 ) (ψi,j+1 + ψi,j ) − (φi+1,j−1 + φi+1,j − φi−1,j−1 − φi−1,j ) (ψi,j + ψi,j−1 ) + (φi+1,j − φi,j+1 ) (ψi+1,j+1 + ψi,j ) − (φi,j−1 − φi−1,j ) (ψi,j + ψi−1,j−1 ) + (φi,j+1 − φi−1,j ) (ψi−1,j+1 + ψi,j ) − (φi+1,j − φi,j−1 ) (ψi,j + ψi+1,j−1 )]. (3.69) Note, ADSCF(R, N ) is less sparse than DSCF(R, N ) and it is more difficult to derive accurate solutions by the SDPR method with relaxation order ω = 1. However, we succeed in solving ADSCF(R, N ) in some instances. For example, in Table 3.21 and Figure 3.27 we compare the minimum kinetic energy solutions obtained for DSCF(5000, N ) and ADSCF(5000, N ). It is interesting that the vortex in the minimum kinetic energy solution for ADSCF(5000, N ) is preserved for increasing N , whereas the vortex in solution for DSCF(5000, N ) seems to deteriorate. Problem ADSCF(5000,14) ADSCF(5000,16) DSCF(5000,14) DSCF(5000,16) ǫsc 7e-12 5e-10 1e-11 3e-10 tC 1304 2802 419 768 F (u⋆ ) 1.8e-4 3.1e-4 5.6e-4 1.1e-4 Table 3.21: Results for solving ADSCF(5000,N) compared to DSCF(5000,N). Solutions of CF(R, N ) for increasing R In order to understand why convergence of discrete approximations to the analytic solution is a lot more difficult to obtain for large R, we examine the behavior of the minimal energy solution of DSCF(R, N ) and CF(R, N ), respectively, for increasing Reynolds number R. The SDPR method is one possible approach to solve DSCF(R, N ). If ω is chosen sufficiently large, the output u of the SDPR method is guaranteed to accurately approximate the minimal energy solution u⋆ of CF(R′ , N ) and DSCF(R′ , N ), respectively. In order to show the advantage of the SDPR method we compare our results to solutions of DSCF(R′ , N ) obtained by the following standard procedure: Method 3.2 Naive homotopy-like continuation method 1. Choose the parameters R′ , N and a step size ∆R. 2. Solve DSCF(0, N ), i.e. a linear system, and obtain its unique solution u0 . 3. Increase Rk−1 by ∆R: Rk = Rk−1 + ∆R 4. Apply Newton’s method to DSCF(Rk , N ) starting from uk−1 . Obtain solution uk as an approximation to a solution of the discrete cavity flow problem. 5. Iterate 3. and 4. until the desired Reynold’s number R′ is reached. Note, the continuation method does not necessarily yield the minimal kinetic energy solution of DSCF(R, N ). Let u⋆ (R, N ) denote the global minimizer of CF(R, N ), the minimal energy is given by Emin (R, N ) = F (u⋆ (R, N )). Obviously, Emin (0, N ) = F (u0 (N )) holds. In a next step, the solution of DSCF(R, N ) obtained by the continuation method starting from u0 is denoted as ũ(R), and its energy as EC (R, N ) := F (ũ(R, N )). As illustrated for N = 5 in Figure 3.28, it is possible to find a continuation ũ of u0 for all R. 113 3.3. NUMERICAL EXPERIMENTS 1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 1 1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Figure 3.27: Solutions for ADSCF(5000,14) (top left), ADSCF(5000,16) (top right), DCSF(5000,14) (bottom left) and DCSF(5000,16) (bottom right). For N = 5 the dimension of DSCF(R, N ) is n = 18. This dimension is small enough to solve a polynomial system by Gröbner basis method and to determine all complex solutions of the system. Therefore, we can verify whether SDPR method detects the global minimizer of CF(R, N ) or not. It is worth pointing out, that we are able to find the minimal energy solution of CF(R, N ) by applying the SDP relaxation method, whereas this solution cannot be obtained by the continuation method. We observe applying the SDPR method with ω = 1 is sufficient to detect the global optimizer for R ≤ 10000, and for R ≥ 20000 the global optimizer is obtained by the SDPR method with ω = 2. In the case of N = 6 and N = 7 the dimension of the polynomial system is too large to be solved by Gröbner basis method for R > 0. For N = 6 the continuation method, and the SDPR method with ω = 1 and ω = 2 yield the same solution for all tested R. And in the case of N = 7 the continuation solution ũ(R) is detected by the SDPR method with ω = 1 as well, except the case R = 6000, where a solution with slightly smaller energy is detected, as documented in Table 3.23. Summarizing these results, F (u0 (N )) ≥ F (ũ(R, N )) for any of the tested R > 0. It is an advantage of the SDPR method to show, ũ(R, N ) is in general not the optimizer of CF (R, N ) for increasing R. In fact, for some settings we obtain far better approximations to the minimal energy solution than ũ(R, N ). Furthermore, Emin (R) and EC (R) are both decreasing in R. The behavior of EC , ESDPR and Emin coincides for all chosen discretizations N and motivates the following conjecture. Conjecture 3.1 Let discretization N be fixed. a) F (u0 (N )) = Emin (0, N ) ≥ Emin (R, N ) ≥ 0 ∀R ≥ 0. 114 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS R 0 100 500 1000 2000 4000 6000 8000 10000 30000 100000 NC 1 37 37 37 37 37 36 36 35 35 34 NR 1 13 13 13 13 17 16 16 17 17 16 EC 0.0096 0.0030 6.2e-4 5.4e-4 6.2e-4 6.3e-4 5.7e-4 5.2e-4 4.7e-4 4.5e-4 4.5e-4 ESDP R(1) 0.0096 0.0030 6.2e-4 5e-4 6.2e-4 4.6e-4 4.5e-4 4.5e-4 4.5e-4 4.5e-4 4.5e-4 ESDP R(2) 0.0096 0.0030 6.2e-4 5e-4 6.2e-4 4.6e-4 4.5e-4 4.5e-4 4.5e-4 2.5e-4 8.8e-5 Table 3.22: Numerical results for CF (R, 5), where ESDP R(ω) the discretized kinetic energy of the solution of SDPR method with relaxation order ω. 0.01 0.009 0.008 EC(R) ESDPR(1)(R) 0.007 ESDPR(2)(R) / Emin(R) 0.006 0.005 0.004 0.003 0.002 0.001 0 0 1 2 3 4 5 R 6 7 8 9 10 4 x 10 Figure 3.28: EC (R), ESDPR(1) (R), ESDPR(2) (R) and Emin (R) for N = 5. b) Emin (R, N ) → 0 for R → ∞. As an application, Conjecture 3.1 can be used as a certificate for the non-optimality of a feasible solution u′ of CF(R, N ) in the case F (u′ (R, N )) > Emin (0, N ). If it is possible to extend u0 to R via continuation method, ũ(R, N ) can serve as a non-optimality certificate in the case F (u′ (R, N )) > F (ũ(R, N )). 3.3.6 Optimal control problems A class of challenging problems involving differential equations that goes beyond the class of PDE problems (3.14) is optimal control, in particular nonlinear optimal control. To solve nonlinear optimal control problems (OCP) analytically is a challenging problem, even though powerful techniques such as the maximum principle and Hamilton-Jacobi-Bellman optimality equations exist. For numerical methods to solve OCP, we distinguish direct and indirect methods [18, 25, 94, 101]. But, in particular for OCPs with state constraints many numerical methods are difficult to use. A recent approach by Lasserre et al. [54] takes advantage of semidefinite programming (SDP) relaxations to generate a convergent sequence of lower bounds for the optimal value of an OCP. We demonstrate on the following examples that the SDPR method can be applied as well to solve OCPs numerically. An OCP can be discretized via finite difference approximations to obtain a POP satisfying a structured sparsity pattern. The POP we derive from an OCP is essentially of the form (3.26); the main difference being that we do not choose the objective function F , but that F is given as the discretization of the objective of the OCP. By applying the SDPR method to an OCP we (a) obtain 115 3.3. NUMERICAL EXPERIMENTS R EC ESDPR(1) 0 2.0e-2 2.0e-2 100 7.7e-3 7.7e-3 4000 4.1e-4 4.1e-4 6000 3.7e-4 3.6e-4 10000 3.4e-4 3.4e-4 Table 3.23: Numerical results for CF (R, 7), where ESDP R(ω) the discretized kinetic energy of the solution of SDPR method with relaxation order ω. a lower bound for its optimal value, and (b) unlike the approach in [54] we obtain approximations for the optimal value, the optimal control and trajectory. As in the PDE case, it is a feature of the SDPR method that state and/or control contraints can be incorporated by defining additional polynomial equality and inequality constraints. Control of production and consumption The following problem arises from the context of control of production and consumption of a factory. Let x(t) be the amount of output produced at time t ≥ 0, α(t) the control variable which denotes the fraction of output reinvested at time t ≥ 0, with 0 ≤ α(t) ≤ 1. The dynamics of the system are provided by the ODE problem ẋ(t) = kα(t)x(t) ∀ t ∈ [0, T ], x(0) = x0 , where k > 0 a constant modeling the growth rate of a reinvestment. It is the aim to maximize the functional Z T (1 − α(t)) x(t)dt, P (α(·), x(·)) = 0 i.e., the total consumption of the output, our consumption at a given time t being (1 − α(t))x(t). Thus, the control problem can be written as RT (1 − α(t)) x(t)dt max 0 s.t. ẋ(t) = k α(t)x(t) ∀ t ∈ [0, T ], (3.70) x(0) = x0 , 0 ≤ α(t) ≤1 ∀ t ∈ [0, T ]. The constraining ODE problem can be discretized by a finite difference scheme in the same way as (3.14). In contrast to the previous examples, we are not free to choose the objective function of (3.26). It is given by the objective function P (α(·), x(·)) of the optimal control problem (3.70). We obtain the POP’s objective function F by discretizing P (α(·), x(·)) as F (α, x) = N X i=1 (1 − αi )xi . It is easy to show with the Pontryagin Maximum Principle, see for example [60], the optimal control law α⋆ (·) is given by ( 1 if 0 ≤ t ≤ t⋆ ⋆ α (t) = 0 if t⋆ < t ≤ T for an appropriate switching time t⋆ , 0 ≤ t⋆ ≤ T . In the case k = 1 the switching time is given by t⋆ = T − 1. We apply the SDPR method to (3.70) with objective function F and ω = 2, and can confirm numerically t⋆ = T − 1 holds for k = 1. Our results are reported in Table 3.24 and illustrated in Figure 3.29. Thus, the solution of the control problem and in particular the optimal control law α⋆ are approximated accurately. Moreover, we observe that the switching time fits the predicted value. 116 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS T 3 3 4 4 Nt 200 300 200 300 ǫsc -3.9e-6 -1.3e-4 -4.5e-4 -2.7e-4 ǫobj 4.4e-9 3.9e-9 1.9e-8 3.9e-8 tC 102 354 126 367 t⋆ 2.00 2.00 3.00 3.00 Table 3.24: Results of the SDPR method for (3.70) with k = 1, x0 = 0.25. 2 6 1.8 5 1.6 x(t) alpha*(t) 1.4 4 x(t) alpha(t) 1.2 1 3 0.8 2 0.6 0.4 1 0.2 0 0 0.5 1 1.5 t 2 2.5 3 0 0 0.5 1 1.5 2 t 2.5 3 3.5 4 Figure 3.29: SDPR method solutions for (3.70), x(t) (blue) and α(t) (green) for Nt = 300, T = 3 (left) and T = 4 (right), respectively. Control of reproductive strategies of social insects As another example, consider a problem arising from reproductive strategies of social insects. max P (w(·), q(·), α(·)) s.t. ẇ(t) w(0) q̇(t) q(0) 0 ≤ α(t) = q(T ) = −µw(t) + b s(t)α(t) w(t) ∀ t ∈ [0, T ], = w0 , = −νq(t) + c(1 − α(t))s(t)w(t) ∀ t ∈ [0, T ], = q0 , ≤ 1 ∀ t ∈ [0, T ], (3.71) where w(t) the number of workers at time t, q(t) the number of queens, α(t) the control variable, which denotes the fraction of the colony effort devoted to increasing work force, µ the workers death rate, ν the queens death rate, s(t) a known rate at which each worker contributes to the bee-economy, b and c constants. It follows from Pontryagin Maximum Principle [60], the optimal control law α of problem (3.71) is a bang-bang control law for any rate s(t), i.e., α(t) ∈ {0, 1} for all t ∈ [0, 1]. For the SDPR method we choose as objective the function F given by F (w, q, α) = qNt , 117 3.3. NUMERICAL EXPERIMENTS which is a discretization of the objective function P (w(·), q(·), α(·)). Table 3.25 shows the numerical results, which are illustrated in Figure 3.30. s(t) 1 1 2 (sint + 1) Nt 300 300 ǫobj 2e-7 1e-4 ǫsc -2e-6 -4e-5 Table 3.25: Results of the SDPR method for (3.71) with T = 3, µ = 0.8, b = 1, w0 = 10, ν = 0.3, c = 1, q 0 = 1 and ω = 2. s(t)=1 s(t) = 0.5 (sin(t) +1) 14 10 w(t) 9 w(t) q(t) alpha(t) 12 q(t) 8 10 alpha(t) 7 6 8 5 6 4 3 4 2 2 1 0 0 0.5 1 1.5 t 2 2.5 3 0 0 0.5 1 1.5 t 2 2.5 3 Figure 3.30: SDPR method solutions for (3.71) for s(t) = 1 (left) and s(t) = 0.5(sin(t) + 1) (right). In the case of s(t) = 1, it is sufficient to choose w(t), q(t) ≤ 20 as upper bounds to get accurate results. For the more difficult problem with s(t) = 0.5(sin(t) + 1) it is necessary to tighten the upper bounds to w(t) ≤ 10 and q(t) ≤ 3, in order to obtain fairly accurate results. In both cases, the bang-bang control law is approximated with high precision. The double integrator Consider the optimal control problem given by min T s.t. ẋ1 (t) = x2 (t) ∀t ∈ [0, T ], ẋ2 (t) = u(t) ∀t ∈ [0, T ], x1 (0) = x1,0 , x1 (T ) = 0, x2 (0) = x2,0 , x2 (T ) = 0, x2 (t) ≥ −1, u(t) ∈ [−1, 1]. (3.72) Note, we can not apply the SDPR method directly to (3.72), since the length T of the domain is not specified. Furthermore as a system of first order PDE with terminal condition is overspecified, we replace 118 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS the constraints x1 (T ) = x2 (T ) = 0 by | x1 (T ) | + | x2 (T ) |≤ ǫ for a small ǫ > 0. We apply a standard coordinate transformation to (3.72) and obtain the equivalent problem min T s.t. ẋ1 (t) = T x2 (t) ∀t ∈ [0, 1], ẋ2 (t) = T u(t) ∀t ∈ [0, 1], x1 (0) = x1,0 , x2 (0) = x2,0 , | x1 (1) | + | x2 (1) |≤ ǫ, x2 (t) ≥ −1, u(t) ∈ [−1, 1]. (3.73) Optimal control problem (3.73) is of a form we can apply the SDPR method to. Lower bounds lbdu = lbdx = −1 and upper bound ubdu = 1 are given, and we choose ubdx = 10. Given some starting point x0 ∈ R2 it is the aim to find the minimal time T ⋆ to steer x(t) into the origin. For this simple problem it is possible to determine the minimal time T ⋆ (x0 ) analytically, c.f. [54]. Thus, for each choice ω (x0 )) and evaluate the performance of our approach. We apply x0 we can calculate the ratio min(sSDP T ⋆ (x0 ) the SDPR method with ω = 3 for discretization N = 50. We choose the same set of x0 as in [54], x0,1 ∈ {0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0} and x0,2 ∈ {−1.0, −0.8, −0.6, −0.4, −0.2, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0}. 3 (x0 )) for the 11 × 11 different values of x0 . Some entries are In Table 3.26 we report the fraction min(sSDP T ⋆ (x0 ) larger than 1, which can be explained by the fact that there is a discretization error for the medium scale discretization N = 50. Compared to the corresponding table in [54], we achieve better lower bounds for T ⋆ in most cases. Moreover, the SDPR method approximates optimal control and trajectory in addition to generating lower bounds for the optimal value. See Figure 3.31 for approximations of u⋆ and x⋆ before and after applying sequential quadratic programming with the SDP solution as initial guess. We observe, that the approximation (ũ, x̃) provided by the sparse SDP relaxation is already close to the highly accurate approximation (u, x) to the optimal solution of the discretized OCP obtained by additionally applying SQP. x0,1 \x0,2 0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 -1 0.8783 0.8158 0.7228 1.0139 1.0079 1.0141 1.0162 1.0185 1.0189 1.0196 1.0234 -0.8 0.6162 0.4756 0.9440 0.9975 1.0071 1.0124 1.0131 1.0156 1.0195 1.0182 1.0205 -0.6 0.5543 0.9060 0.9237 0.9971 1.0090 1.0119 1.0109 1.0135 1.0148 1.0187 1.0177 -0.4 0.5665 0.8756 0.9258 0.9886 0.9972 1.0050 1.0064 1.0114 1.0136 1.0150 1.0168 -0.2 0.8472 0.8281 0.9023 0.9991 0.9983 1.0009 1.0044 1.0086 1.0114 1.0133 1.0140 0 1.0000 0.7869 0.8708 0.9876 1.0035 0.9926 1.0018 1.0067 1.0069 1.0082 1.0109 0.2 0.8420 0.7362 0.8539 0.9382 1.0005 1.0086 1.0086 1.0042 1.0070 1.0085 1.0095 0.4 0.5447 0.7420 0.8495 0.9588 0.9962 1.0016 1.0076 1.0017 1.0062 1.0076 1.0082 Table 3.26: min(sSDP3 (x0 ))/T ⋆ (x0 ) for different choices of x0 . 0.6 0.8191 0.9068 0.9423 0.9507 0.9772 1.0020 1.0018 1.0065 1.0021 1.0065 1.0045 0.8 0.8858 0.9339 0.9692 0.9875 1.0025 1.0026 1.0043 0.9991 1.0009 1.0027 1.0028 1.0 0.9128 0.9537 0.9811 0.9848 0.9901 0.9961 0.9974 0.9967 0.9997 1.0010 1.0024 119 3.3. NUMERICAL EXPERIMENTS 1 1 0.8 0.8 x1 0.6 x1 0.6 x2 0.4 x2 0.4 u 0.2 0.2 0 0 −0.2 −0.2 −0.4 −0.4 −0.6 −0.6 −0.8 −0.8 −1 0 5 10 15 20 25 30 35 40 45 50 −1 u 0 5 10 15 20 25 30 35 40 45 50 Figure 3.31: Optimal control and trajectories for x0 = (0.8, −1) for Double Integrator OCP before (left) and after (right) applying SQP. The Brockett Integrator Consider the nonlinear optimal control problem given by min T s.t. ẋ1 (t) = u1 (t) ẋ2 (t) = u2 (t) ẋ3 (t) = u1 (t)x2 (t) − u2 (t)x1 (t) x1 (0) = x1,0 , x2 (0) = x2,0 , x3 (0) = x3,0 , x1 (T ) = 0, x2 (T ) = 0, x3 (T ) = 0, u1 (t)2 + u2 (t)2 ≤ 1. ∀t ∈ [0, T ], ∀t ∈ [0, T ], ∀t ∈ [0, T ], (3.74) Applying the same transformation as for (3.72), we bring (3.74) into a form we can apply the SDPR method to: min T s.t. ẋ1 (t) = T u1 (t) ∀t ∈ [0, 1], ẋ2 (t) = T u2 (t) ∀t ∈ [0, 1], ẋ3 (t) = T u1 (t)x2 (t) − T u2 (t)x1 (t) ∀t ∈ [0, 1], x1 (0) = x1,0 , (3.75) x2 (0) = x2,0 , x3 (0) = x3,0 , | x1 (1) | + | x2 (1) | + | x3 (1) |≤ ǫ, u1 (t)2 + u2 (t)2 ≤ 1, for some small ǫ > 0. As in example (3.72), it is the aim to find the minimal time T ⋆ (x0 ) to steer some point x(t) with x(0) = x0 ∈ R3 into the origin. For the lower and upper bounds of u and x, we choose lbdu = −1, ubdu = 1, lbdx = −5, ubdx = 5. For this optimal control problem it is possible to calculate the minimal time T ⋆ exactly [54]. Thus, we can compare the performance of the SDPR method to the approach in [54]. We apply the SDPR method 120 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS x0,2 \x0,3 0 1 2 3 0 0.0000 1.6049 2.7816 3.9705 1 1.0081 1.2827 2.0269 2.8498 2 2.0276 2.0337 2.1959 2.6177 3 3.0145 3.0125 3.0170 3.1906 Table 3.27: min(sSDP3 (x0 )) for x0,1 = 0 and (x0,2 , x0,3 ) ∈ {0, 1, 2, 3}2. x0,2 \x0,3 0 1 2 3 0 0.0000 2.5066 3.5449 4.3416 1 1.0000 1.7841 2.6831 3.4328 2 2.0000 2.1735 2.5819 3.0708 3 3.0000 3.0547 3.2088 3.4392 Table 3.28: T ⋆ (x0 ) for x0,1 = 0 and (x0,2 , x0,3 ) ∈ {0, 1, 2, 3}2. with ω = 3 to (3.75). The numerical results for N = 50 and x0,1 = 0, (x0,2 , x0,3 ) ∈ {0, 1, 2, 3}2 are reported in Table 3.27, the results for N = 30 and x0,1 = 1, (x0,2 , x0,3 ) ∈ {1, 2, 3}2 in Table 3.29. Again, min(sSDPw (x0 )) is larger than T ⋆ (x0 ) for some x0 , which is explained by the discretization error due to the medium scale choice N ∈ {30, 50}. This gap closes for N → ∞. In particular for choices x0 to be found in the lower left corner of Table 3.27 and 3.29, we obtain better lower bounds than [54]. Again, unlike the method in [54] we also obtain an accurate approximation of optimal control and trajectory, as pictured in Figure 3.32. x0,2 \x0,3 1 2 3 1 1.8862 2.6077 3.2969 2 2.4412 2.7737 3.2033 3 3.3145 3.4516 3.6618 Table 3.29: min(sSDP3 (x0 )) for x0,1 = 1 and (x0,2 , x0,3 ) ∈ {1, 2, 3}2. 121 3.3. NUMERICAL EXPERIMENTS x0,2 \x0,3 1 2 3 1 1.8257 2.5231 3.1895 2 2.3636 2.6856 3.1008 3 3.2091 3.3426 3.5456 Table 3.30: min(T ⋆ (x0 )) for x0,1 = 1 and (x0,2 , x0,3 ) ∈ {1, 2, 3}2. 2 2 x1 x1 x2 1.5 x2 1.5 x3 x3 u1 u1 u2 1 0.5 0.5 0 0 −0.5 −0.5 −1 0 5 10 15 20 25 30 35 40 45 u2 1 50 −1 0 5 10 15 20 25 30 35 40 45 50 Figure 3.32: Optimal control and trajectories for x0 = (0, 2, 1) for Brockett Integrator OCP before (left) and after (right) applying SQP. 122 CHAPTER 3. SDP RELAXATIONS FOR SOLVING DIFFERENTIAL EQUATIONS Chapter 4 Concluding Remarks and Future Research 4.1 Conclusion Hierarchies of SDP relaxations are a powerful tool to solve general, severely nonconvex POPs. However, to solve large scale POPs remains a very challenging task due to the limited capacity of contemporary SDP solvers. In this thesis we discussed two major approaches to attempt large scale POPs by reducing the size of the SDP relaxations. In the first one presented in 2.2, our focus has been on developing a theoretical framework consisting of the d- and r-space conversion methods to exploit structured sparsity, characterized by a chordal graph structure, via the positive semidefinite matrix completion for an optimization problem involving linear and nonlinear matrix inequalities. The two d-space conversion methods are provided for a matrix variable X in objective and/or constraint functions of the problem, which is required to be positive semidefinite. The methods decompose X into multiple smaller matrix variables. The two r-space conversion methods are aimed at a matrix inequality in the constraint of the problem. In these methods, the matrix inequality is converted into multiple smaller matrix inequalities. As mentioned in Remarks 2.3, 2.5 and 2.7, the d-space conversion method using clique trees and the r-space conversion method using clique trees have plenty of flexibilities in implementation. This should be explored further for increasing the computational efficiency. In 2.2.7 we constructed linear SDP relaxations for general quadratic SDP that exploit d- and r-space sparsity. When applying these relaxations to quadratic SDP arising from different applications, we observed that the computational performance is greatly improved compared to the classical SDP relaxations, which do not apply d- and r-space conversion methods. Of particular interest for the numerical analysis of differential equation is the linear SDP relaxation exploiting d-space sparsity. It reduces the size of SDP relaxations for POPs derived from certain differential equations a lot. It will be interesting topic to study the efficiency of d- and r-space conversion methods for further classes of nonlinear SDPs. In 2.3, we proposed four different heuristics to transform a general POP into a QOP. The advantage of this transformation is that the sparse SDP relaxation of order one can be applied to the QOP. The sparse SDP relaxation of order one is of vastly smaller size than the sparse SDP relaxation of minimal order ωmax for the original POP. By solving the sparse SDP relaxation of the QOP, approximates to the global minimizer of a large scale POP of higher degree can be derived. The reduction of the SDP relaxation and the gain in numerical tractability come at the cost of deteriorating feasibility and optimality errors of the approximate solution obtained by solving the SDP relaxation. In general the SDP relaxation of order one for the QOP is weaker than the SDP relaxation of order ωmax for the original POP. We discussed how to overcome this difficulty by imposing tighter lower and upper bounds for the components of the n-dimensional variable of a POP, by adding linear or quadratic Branch-and-Cut bounds, and by applying local convergent optimization methods such as sequential quadratic programming to the POP starting from the solution provided by the SDP relaxation for the QOP. The proposed heuristics have been demonstrated with success 123 124 CHAPTER 4. CONCLUDING REMARKS AND FUTURE RESEARCH on various medium and large-scale POPs. We have seen that imposing additional Branch-and-Cut bounds was necessary to yield accurate approximations to the global optimizer for some problems. However, for most problems it was crucial to choose the lower and upper bounds for the variable x sufficiently tight and to apply SQP to obtain a highly accurate approximation of the global optimizer. The total processing time could be reduced by up to three magnitudes for the problems we tested. For these reasons we think the proposed technique is promising to find first approximate solutions for POPs, whose size is too large to be solved by the more precise, original SDP relaxations due to Waki et al. Our most important application for both approaches to reduce the size of SDP relaxations is the numerical analysis of nonlinear differential equations. We were able to transform nonlinear PDE problems with polynomial structure into POPs. The description is based on the discretization of a PDE problem, the approximation of its partial derivatives by finite differences and the choice of an appropriate objective function. Due to the finite difference discretization, the POPs derived from PDEs satisfy both, correlative and domain-space sparsity patterns under fairly general assumptions. Therefore, we can apply dual standard form SDP relaxations exploiting correlative sparsity and primal standard form SDP relaxations exploiting domain-space sparsity efficiently. For many PDE problems the solution of the SDP relaxation is an appropriate initial guess for locally fast convergent methods like Newton’s method and SQP. However, we have seen it is often necessary to impose tighter or additional bounds to the POPs to derive highly accurate solutions. Moreover, we demonstrated how to choose an objective function and bounds for the unknown function, in order to detect particular solutions of a discretized PDE problem. In other words, one of the features of using the SDPR method instead of several existing methods is that a function space to find solutions may be translated into natural constraints for a sparse POP. In the case we have partial information about a particular solution we want to find, this information can be exploited by the SDPR method to provide an appropriate initial guess for a local method. The reduction techniques from Chapter 2 are highly efficient to solve POPs derived from PDEs for higher resolution grids and to obtain accurate discrete approximations for solutions of the continuous PDE problem. Another technique to extend solutions to finer and finer grid is the grid-refining method, which is efficient even when starting from solutions on very coarse grids. We have shown that the SDPR method is very promising for nonlinear differential equations with several solutions. One feature of the SDPR method is the ability to detect a particular solution. Another challenging problem is to enumerate all solutions of a discretized PDE problem and ultimately to enumerate accurate approximations for all solutions of the underlying continuous PDE problem. We proposed an algorithm based on the SDPR method to approximately enumerate all real solutions of a zero dimensional radical polynomial system with respect to a cost function. If the order of the SDP relaxations tends to infinity, we can guarantee the convergence of the algorithm’s output to the smallest kinetic energy solutions of the polynomial system. The algorithm can be applied successfully to enumerate the solutions of the discrete cavity flow problem with the kinetic energy of the flow as cost function. A variant of the enumeration algorithm has been applied to detect all solutions of an interesting reaction-diffusion equation. Since both, the enumeration algorithm and its variant, are based on the SDPR method that exploits sparsity, it is possible to attempt POPs of much larger scale than by the approaches in [35] and [55]. To conclude, the SDPR method constitutes a general purpose method for solving problems involving differential equations polynomial in the unknown functions. We demonstrated the potential of the SDPR method on differential equations arising from a range of areas: Elliptic, parabolic and hyperbolic PDE, reaction-diffusion equations, fluid dynamics, nonlinear optimal control, differential algebraic equations and first order PDEs. This list of differential equations we may analyze by the SDPR method is by no means complete. But it illustrates that the SDPR method provides a powerful tool to get new insights in the numerical analysis of differential equations. 4.2 Outlook on future research directions Efficient software for large scale POPs and their SDP relaxations remains a challenging field with many open problems. The research presented in this thesis motivates to look for answers to a number of questions: We discussed four heuristics to transform an arbitrary POP into an equivalent QOP. Moreover, we 4.2. OUTLOOK ON FUTURE RESEARCH DIRECTIONS 125 encountered that correlative sparsity and domain-space sparsity of a QOP can differ significantly. Therefore, the size of the resulting dual and primal form SDP relaxations may be of vastly different size. The question remains whether (a) there is a way to transform a POP into a QOP that enhances these types of sparsity, and (b) whether we may find a more general concept of sparsity for a QOP that combines these two types of sparsity. We have also seen that the approximation accuracy of the SDP relaxation for the QOP is weaker than for the original POP. It remains a future problem to strengthen the sparse SDP relaxation of order one for a QOP further. In that respect, it is desirable to find a systematic approach to tighten lower and upper bounds successively, without shrinking the optimal set of the POP. Furthermore, the additional quadratic constraints derived under the transformation algorithm allow to express some moments as linear combination of other moments. As proposed by Henrion and Lasserre in [35] and Laurent in [56] these linear combinations can be substituted in the moment and localizing matrices of the SDP relaxation to reduce the size of the moment vector y. Exploiting this technique will shrink the size of the sparse SDP relaxations for QOPs further and may enable us to solve POPs of even larger scale. Compared to the methods [35, 55] for finding all real solutions of a zero-dimensional radical polynomial system, our enumeration algorithm can be applied to problems of much larger scale. However, the numerical stability depends heavily on the choice of the parameters in the algorithm. The variant of the enumeration algorithm for the Swift-Hohenberg equation constitutes a promising first step to improve the numerical stability, since the additional linear constraints remain unchanged under the SDP relaxation. The idea of this variant may be exploited more systemically in future. Although we are able to solve some PDE problems with minimal relaxation order ω = ωmax , in many cases it is a priori not possible to predict the relaxation order ω which is necessary to attain an accurate solution. As the size of the sparse SDP relaxation increases polynomially in ω, the tractability of the SDP is limited by the capacity of current SDP solvers. It is a further challenging question whether we can characterize a class of differential equation problems that is guaranteed to be approximated accurately for a certain fixed relaxation order. At the moment there are only very few results for error bounds of SDP relaxations for general, nonconvex POPs [72]. Not every solution of a discretized differential equation is a discrete approximation for an actual solution of this differential equation, as we encountered in the analysis of the steady cavity flow problem. It is therefore interesting to close the gap between the discrete and the continuous world. An approach based on the SDPR method for narrowing this gap takes advantage of maximum entropy estimation [10, 53]. In this approach the solution of the SDPR method is used to compute discrete approximations to moments of a measure corresponding to the differential equation, and when applying maximum entropy estimation to these discretized moments we obtain a smooth approximation for a solution of the differential equation. This is the topic of some ongoing joint work with Jean Lasserre and Didier Henrion. Finally, we applied the SDPR method to a wide variety of problems involving differential equations. However, the classes of problems we may attempt by this approach are by no means exhausted. It will be an interesting topic of future research to apply the SDPR methods to challenging nonlinear differential equations satisfying a polynomial structure. Also, nonlinear optimal control seems an challenging area to apply this methodology to, as the numerical experiments for the simple optimal control problems presented in this thesis suggest. 126 CHAPTER 4. CONCLUDING REMARKS AND FUTURE RESEARCH Bibliography [1] J. Agler, J. W. Helton, S. McCullough, L. Rodman, Positive semidefinite matrices with a given sparsity pattern, Linear Algebra Appl. (1988), Vol. 107, pp. 101-149. [2] E.L. Allgower, D.J. Bates, A.J. Sommese, C.W. Wampler, Solution of polynomial systems derived from differential equations, Computing, 76 (2006), No. 1, pp. 1-10. [3] A. Arakawa, Computational design for long-term numerical integration of the equation of fluid motion: two dimensional incompressible flow, part I., Journal of Computational Physics 135 (1997), pp. 103-114. [4] J.R.S. Blair, B. Peyton, An introduction to chordal graphs and clique trees, Graph Theory and Sparse Matrix Computation, Springer Verlag (1993), pp. 1-29. [5] D. Bertsimas, C. Caramanis, Bounds on linear PDEs via semidefinite optimization, Math. Programming, Series A 108 (2006), pp. 135-158. [6] P. Biswas, Y. Ye, A distributed method for solving semidefinite programs arising from Ad Hoc Wireless Sensor Network Localization, Multiscale Optimization Methods and Applications, 69-84, SpringerVerlag [7] J. Bochnak, M. Coste, M.-F. Roy, Real Algebraic Geometry, Springer-Verlag (1998). [8] P.T. Boggs, J.W. Tolle, Sequential Quadratic Programming, Acta Numerica 4 (1995), pp. 1-50. [9] B. Borchers, SDPLIB 1.2, A library of semidefinite programming test problems, Optim. Methods Softw. (1999), 11-12, pp. 683-689. [10] J. Borwein, A.S. Lewis, On the convergence of moment problems, Trans. Am. Math. Soc., 325 (1991), pp. 249-271. [11] D. Braess, Finite Elements, Theory, fast solvers, and applications in solid mechanics, Cambridge University Press (2001). [12] O.R. Burggraf, Analytical and numerical studies of the structure of steady separated flows, J. Fluid Mech 24 (1966), pp. 113-151. [13] R. Courant, K. Friedrichs, H. Lewy, Über die partiellen Differenzengleichungen der mathematischen Physik, Math. Ann. 100 (1928), No. 1, pp. 32-74. [14] M. Cheng, K.C. Hung, Vortex structure of steady flow in a rectangular cavity, Computers & Fluids, Volume 35, Issue 10 (2006), pp. 1046-1062. [15] R. Courant, D. Hilbert, Methoden der Mathematischen Physik, Vol 1 (1931), Chapter 4, The method of variation. [16] R. Courant, Variational methods for the solution of problems of equilibrium and vibrations, Bull. Amer. Soc. 49 (1943), pp. 1-23. 127 128 BIBLIOGRAPHY [17] G. Dahlquist, Convergence and stability in the numerical integration of ordinary differential equations, Math. Scand. 4 (1956), pp. 33-53. [18] R. Fletcher, Practical Methods of Optimization. Vol. 1 Unconstrained Optimization, John Wiley, Chichester (1980). [19] I. Fried, Numerical Solutions of Differential Equations, Academic Press (1979). [20] K. Fujisawa, S. Kim, M. Kojima, Y. Okamoto, M. Yamashita, User’s Manual for SparseCoLO: Conversion Methods for SPARSE COnic-form Linear Optimization Problems, Research reports on Mathematical and Computing Sciences B-453, Tokyo Institute of Technology. [21] M. Fukuda, M. Kojima, K. Murota, K. Nakata Exploiting sparsity in semidefinite programming via matrix completion I: General framework, SIAM J. Optim., 11 (2000), pp. 647-674. [22] M. Fukuda, M. Kojima, Branch-and-Cut Algorithms for the Bilinear Matrix Inequality Eigenvalue Problem, Computational Optimization and Applications, 19 (2001), pp. 79-105. [23] B.G. Galerkin, Series solution of some problems in elastic equilibrium of rods and plates, Vestn. Inzh. Tech. 19 (1915), pp. 897-908. [24] A. George, J.W. Liu, Computer Solution of Large Sparse Positive Definite Systems, Prentice-Hall (1981). [25] P.E. Gill, W. Murray, M.H. Wright, Practical Optimization, Academic Press, London, New York (1981). [26] M. Goemans, D.P. Williamson, Improved Approximation Algorithms for Maximum Cut and Satisfiability Problems Using Semidefinite Programming, Journal of the ACM, 42 (1995), No. 6, pp. 1115-1145. [27] D. Gottlieb, S. Orszag, Numerical Analysis of Spectral Methods: Theory and Applications, SIAM, Philadelphia (1977). [28] R. Grone, C. R. Johnson, E. M. Sá, H. Wolkowitz, Positive definite completions of a partial hermitian matrices, Linear Algebra Appl. 58 (1984), pp. 109-124. [29] J.L. Guermond, A finite element technique for solving first-order PDEs in LP , SIAM Journal Numerical Analysis 42 (2004), No. 2, pp. 714-737. [30] J.L. Guermond, B. Popov, Linear advection with ill-posed boundary conditions via L1 -minimization, International Journal of Numerical Analysis and Modeling 4 (2007), No. 1, pp. 39-47. [31] T. Gunji, S. Kim, M. Kojima, A. Takeda, K. Fujisawa, and T. Mizutani, PHoM - a Polyhedral Homotopy Continuation Method for Polynomial Systems, Research Reports on Mathematical and Computing Sciences, Dept. of Math. and Comp. Sciences, Tokyo Inst. of Tech., B-386 (2003) [32] K. Gustafson, K. Halasi, Cavity flow dynamics at higher Reynolds number and higher aspect ratio, Journal of Computational Physics 70 (1987), pp. 271-283. [33] W. Hao, J.D. Hauenstein, B. Hu, Y. Liu, A.J. Sommese, Y.-T. Zhang, Multiple stable steady states of a reaction-diffusion model on zebrafish dorsal-ventral patterning, Discrete and Continuous Dynamical Systems, Series S, To appear. [34] J.D. Hauenstein, A.J. Sommese, C.W. Wampler, Regeneration Homotopies for Solving Systems of Polynomials, Mathematics of Computation, To appear. [35] D. Henrion, J.B. Lasserre, Detecting global optimality and extracting solutions in GloptiPoly, Chapter in D. Henrion, A. Garulli, editors, Positive polynomials in control. Lecture Notes in Control and Information science, Springer Verlag (2005), Berlin. BIBLIOGRAPHY 129 [36] D. Henrion, J. B. Lasserre, Convergent relaxations of polynomial matrix inequalities and static output feedback, IEEE Trans. Automatic Control (2006), 51, pp. 192-202. [37] C. W. J. Hol, C. W. Scherer, Sum of squares relaxations for polynomial semidefinite programming, Proc. Symp. on Mathematical Theory of Networks and Systems (MTNS), Leuven, Belgium, 2004. [38] R. Horst, P.M. Pardalos, N.V. Thoai, Introduction to Global Optimization, Kluwer Academic Publishers (2000). [39] B. Huber, B. Sturmfels, A polyhedral method for solving sparse polynomial systems, Math. of Comp. 64 (1995), pp. 1541-1555. [40] M. Kawaguti, Numerical solution of the Navier-Stokes equations for the flow in a two dimensional cavity, J. Phys. Soc. Jpn. 16 (1961), pp. 2307-2315. [41] S. Kim, M. Kojima, Exact solutions of some nonconvex quadratic optimization problems via SDP and SOCP relaxations, Computational Optimization and Applications, 26 (2003), pp. 143-154. [42] S. Kim, M. Kojima, M. Mevissen, M. Yamashita, Exploiting Sparsity in Linear and Nonlinear Matrix Inequalities via Positive Semidefinite Matrix Completion, Mathematical Programming, To Appear. [43] S. Kim, M. Kojima, H. Waki, Exploiting Sparsity in SDP Relaxation for Sensor Network Localization, SIAM Journal of Optimization 20 (2009), No. 1, pp. 192-215. [44] S. Kim, M. Kojima H. Waki, M. Yamashita, SFSDP: a Sparse Version of Full Semidefinite Programming Relaxation for Sensor Network Localization Problems, Research Report B-457, Department of Mathematical and Computing Sciences, Tokyo Institute of Technology (2009). [45] K. Kobayashi, S. Kim, M. Kojima, Correlative sparsity in primal-dual interior-point methods for LP, SDP and SOCP, Appl. Math. Optim. (2008), 58, pp. 69-88. [46] M. Kojima, Sums of Squares Relaxations of Polynomial Semidefinite Programs, Research Report B-397, Department of Mathematical and Computing Sciences, Tokyo Institute of Technology (2003). [47] M. Kojima, S. Kim, H. Waki, Sparsity in sums of squares of polynomials, Mathematical Programming, 103 (2005), pp. 45-62. [48] M. Kojima, M. Muramatsu, An Extension of Sums of Squares Relaxations to Polynomial Optimization Problems over Symmetric Cones, Math. Programming (2007), 110, pp. 315-336. [49] M. Kojima, M. Muramatsu, A note on sparse SOS and SDP relaxations for polynomial optimization problems over symmetric cones, Comput. Optim. Appl. (2009), 42, pp. 31-41. [50] W. Kutta, Beitrag zur näherungsweisen Integration totaler Differentialgleichungen, Zeitschrift Math. Physik 46 (1901), pp. 435-453. [51] J.B. Lasserre, Global optimization with polynomials and the problem of moments, SIAM Journal on Optimization, 11 (2001), pp. 796-817. [52] J.B. Lasserre, Convergent SDP-Relaxations in Polynomial Optimization with Sparsity, SIAM Journal on Optimization, 17 (2006), No. 3, pp. 822-843. [53] J.B. Lasserre, Semidefinite programming for gradient and Hessian computation in maximum entropy estimation, Proc. IEEE Conf. Dec Control, 2007. [54] J.B. Lasserre, D. Henrion, C. Prieur, E. Trelat, Nonlinear optimal control via occupation measures and LMI-relaxations, SIAM Journal on Control and Optimization, 47 (2008), pp. 1649-1666. [55] J.B. Lasserre, M. Laurent, P. Rostalski, Semidefinite characterization and computation of real radical ideals, Foundations of Computational Mathematics, Vol. 8 (2008), No. 5, pp. 607-647. 130 BIBLIOGRAPHY [56] M. Laurent, Sums of squares, moment matrices and optimization over polynomials, Emerging Applications of Algebraic Geometry, Vol. 149 of IMA Volumes in Mathematics and its Applications (2009), M. Putinar and S. Sullivant (eds.), Springer, pp. 157-270. [57] P.D. Lax, R.D. Richtmyer, Survey of the stability of linear finite difference equations, Comm. Pure Appl. Math. 9 (1956), pp. 267-293. [58] R.J. LeVeque, Finite Volume Methods for Hyperbolic Problems, Cambridge University Press (2002). [59] G.R. Liu, S.S. Quek, The Finite Element Method, A practical course, Elsevier (2003). [60] J. Macki, A. Strauss, Introduction to Optimal Control Theory, Springer-Verlag (1982), pp. 108. [61] M. Mevissen, M. Kojima, J. Nie and N. Takayama, Solving partial differential equations via sparse SDP relaxations, Pacific Journal of Optimization, 4 (2008), No. 2, pp. 213-241. [62] M. Mevissen, M. Kojima, SDP Relaxations for Quadratic Optimization Problems Derived from Polynomial Optimization Problems, Asia-Pacific Journal for Operations Research 27 (2010), No. 1, pp. 1-24. [63] M. Mevissen, K. Yokoyama and N. Takayama, Solutions of Polynomial Systems Derived from the Cavity Flow Problem, Proceedings of the 2009 International Symposium on Symbolic Computation, 2009, pp. 255 - 262. [64] M. Mimura, Asymptotic Behaviors of a Parabolic System Related to a Planktonic Prey and Predator Model, SIAM Journal on Applied Mathematics, 37 (1979), no. 3, pp. 499-512. [65] A.R. Mitchell, D.F. Griffiths, The Finite Difference Method in Partial Differential Equations, John Wiley and Sons (1980). [66] J.J. More, B.S. Garbow and K.E. Hillstrom, Testing unconstrained optimization software, ACM Trans. Math. Software, 7 (1981), pp. 17-41. [67] K.G. Murty, S.N. Kabadi, Some NP-complete problems in quadratic and nonlinear programming, Mathematical Programming, 39 (1987), pp. 117-129. [68] K. Nakata, K. Fujisawa, M. Fukuda, M. Kojima, K. Murota Exploiting sparsity in semidefinite programming via matrix completion II: Implementation and numerical results, Math. Programming, 95 (2003), pp. 303-327. [69] Ju. E. Nesterov, A. S. Nemirovski, Interior Point Polynomial Methods in Convex Programming: Theory and Applications, SIAM, Philadelphia, PA, 1994. [70] Y. Nesterov, Squared functional systems and optimization problems, in J.B.G. Frenk, C. Roos, T. Terlaky, and S. Zhang, editors, High Performance Optimization, pp. 405-440. Kluwer Academic Publishers (2000). [71] J. Nie, Sum of squares method for sensor network localization, Computational Optimization and Applications 43 (2009), No. 2, pp. 151-179. [72] J. Nie, An Approximation Bound Analysis for Lasserre’s Relaxation in Multivariate Polynomial Optimization, preprint (2009). [73] Y. Nishiura, D. Ueyama, Spatio-temporal chaos for the Gray-Scott model, Physica D, 150 (2001), pp. 137 - 162. [74] Y. Nishiura, T. Teramoto, K. Ueda, Dynamic transitions through scattors in dissipative systems, Chaos, 13 (2003), No. 3, pp. 962 - 972. BIBLIOGRAPHY 131 [75] Y. Nishiura, T. Teramoto, K. Ueda, Scattering of traveling spots in dissipative systems, Chaos, 15 (2005), 047509. [76] Y. Nishiura, T. Teramoto, X. Yuan, K. Ueda, Dynamics of traveling pulses in heterogeneous media, Chaos, 17 (2007), 037104. [77] J. Nocedal, S.J. Wright, Numerical Optimization, Series in Operations Research, Springer, New York 2006. [78] M. Noro, K. Yokoyama, A modular method to compute the rational univariate representation of zerodimensional ideals, Journal of Symbolic Computation 28 (1999), pp. 243–263. [79] P.A. Parrilo, Semidefinite programming relaxations for semialgebraic problems, Math. Programming, 96 (2003), pp. 293 - 320. [80] L.A. Peletier, V. Rottschäfer, Pattern selection of solutions of the Swift-Hohenberg equation, Physica D, 194 (2004), pp. 95 - 126. [81] M. Putinar, Positive Polynomials on Compact Semi-algebraic Sets, Indiana Univ. Math. Journal 42 (1993), No. 3, pp. 969-984 [82] J. Rauch, J. Smoller, Qualitative theory of the FitzHugh-Nagumo equations, Advances in Mathematics, 27 (1978), pp. 12-44. [83] L. Rayleigh, On the theory of resonance, Trans. Roy. Soc. A 161 (1870), pp. 77 - 118. [84] W. Ritz, Über eine neue Methode zur Lösung gewisser Variationsprobleme der mathematischen Physik, Journal für die reine und angewandte Mathematik 135 (1908), pp. 1-61. [85] F. Rouillier, Solving zero-dimensional systems through the rational univariate representation, Applicable Algebra in Engineering, Communication and Computing 9 (1999), pp. 433–461. [86] C. Runge, Über die numerische Auflösung von Differentialgleichungen, Math. Ann. 46 (1895), pp. 167 -178. [87] K. Schmüdgen, The K-moment problem for compact semi-algebraic sets, Math. Ann. 289 (1991), pp. 203-206. [88] M. Schweighofer, Optimization of polynomials on compact semialgebraic sets, SIAM J. Optimization 15 (2005), pp. 805-825. [89] H.D. Sherali, C.H. Tuncbilek, A global optimization algorithm for polynomial programming problems using a reformulation-linearization technique, Journal of Global Optimization, 2 (1992), pp. 101-112. [90] H.D. Sherali, C.H. Tuncbilek, New reformulation-linearization technique based relaxations for univariate and multivariate polynomial programming problems, Operations Research Letters, 21 (1997), 1, pp. 1-10. [91] N.Z. Shor, Class of global minimum bounds of polynomial functions, Cybernetics, 23 (1987), 6, pp. 731-734. [92] N.Z. Shor, Nondifferentiable Optimization and Polynomial Problems, Kluwer (1998). [93] J. Smoller, Shock Waves and Reaction-Diffusion Equations, Springer-Verlag (1983), pp. 106. [94] J. Stoer, R. Bulirsch, Introduction to Numerical Analysis, 3rd edition, Springer-Verlag, New York (2002). [95] J.F. Sturm, SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones, Optimization Methods and Software, 11 and 12 (1999), pp. 625-653. 132 BIBLIOGRAPHY [96] J.C. Strikwerda, Finite Difference Schemes and Partial Differential Equations, Wadsworth and Brooks (1989). [97] M. Tabata, A finite difference approach to the number of peaks of solutions for semilinear parabolic problems, J. Math. Soc. Japan, 32 (1980), pp. 171-192. [98] T. Takami, T. Kawamura, Solving Paritial Differential Equations with Difference schemes, Tokyo University Press (1994). [99] M.J. Turner, R.M. Clough, H.C. Martin, L.J. Topp, Stiffness and deflection analysis of complex structures, J. Aeron. Sci. 23 (1956), pp. 805-823, pp. 854. [100] T. Teramoto, Personal communication. [101] O. Von Stryk, R. Bulirsch, Direct and indirect methods for trajectory optimization, Ann. Oper. Res. 37 (1992), pp. 357-373. [102] H. Waki, S. Kim, M. Kojima, M. Muramatsu, Sums of squares and semidefinite program relaxations for polynomial optimization problems with structured sparsity, SIAM Journal of Optimization 17 (2006) 218-242. [103] H. Waki, S. Kim, M. Kojima, M. Muramatsu, SparsePOP: a Sparse Semidefinite Programming Relaxation of Polynomial Optimization Problems, Research Reports on Mathematical and Computing Sciences, Dept. of Math. and Comp. Sciences, Tokyo Inst. of Tech., B-414 (2005). [104] M. Yamashita, K. Fujisawa, M. Kojima, Implementation and evaluation of SDPA 6.0 (SemiDefinite Programming Algorithm 6.0), Optimization Methods and Software 18 (2003), pp. 491-505. [105] Yokota, http://next1.cc.it-hiroshima.ac.jp/MULTIMEDIA/numeanal2/node24.html. [106] O.C. Zienkiewicz, R.L. Taylor, J.Z. Zhu, The Finite Element Method, Its Basis and Fundamentals, Elsevier (2005).

- Similar pages
- Optimization Techniques for Semi-supervised Support Vector Machines
- Presentation
- Regularity and Lyapunov stabilization of weak entropy solutions to scalar conservation laws
- Nonnegative Local Coordinate Factorization for Image Representation
- Circuit tuning for high-performance custom digital design.
- Alan Turing Institute
- Fast Poisson solvers for thermal analysis
- Efficient time-domain simulation and optimization of digital FET circuits.
- C164CI-8EM CA, V1.9
- Presentation