Invariant manifold theory is a fundamental tool in the study of local dynamics near invariant structures in smooth evolution systems. It ensures the existence of nonlinearly invariant structures from linear ones. The theory has been well developed for diffeomorphisms, ODEs, semilinear PDEs, and some quasilinear parabolic PDEs. However, it becomes subtle for quasilinear or more nonlinear PDEs due to regularity issues when there is no smoothing effect. In this talk, we consider a class of nonlinear PDEs whose linearizations satisfy certain energy estimates. We prove that the linear exponential dichotomy implies the existence of local stable/unstable manifolds of the equilibria. In particular the result applies to a class of nonlinear Hamiltonian PDEs including the capillary gravity water waves of one or two fluids, quasilinear wave and Schr\odinger equations, KdV type equations, etc., for which the linear analysis is also discussed. Basically, for such systems under certain conditions, spectral instability implies the existence of stable and unstable manifolds, which in particular yields the nonlinear instability in rough Sobolev norms and/or the existence of solutions decaying in high Sobolev norms. This is a joint work with Jalal Shatah.
