The prospects of gene therapy have generated unprecedented interest in the properties and structures of complexes of nucleic acids (NAs) with cationic liposomes (CLs), which are used as nonviral NA carriers in worldwide clinical trials. An improved understanding of the mechanisms of action of CL-NA complexes is required to enable their widespread therapeutic use. In prior studies of CL-mediated DNA delivery, membrane charge density (sigma(M)) was identified as a key parameter for transfection efficiency (TE) of lamellar (L(alpha)(C)) CL-DNA complexes. The TE of CL-DNA complexes containing cationic lipids with headgroup valencies from 1+ to 5+ follows a universal bell-shaped curve as a function of sigma(M). As we report here, the TE of CL-DNA complexes containing new multivalent lipids with dendritic headgroups (DLs) strongly deviates from this curve at high sigma(M). We have investigated four DLs, MVLG2 (4+), MVLG3 (8+), MVLBisG1 (8+), and MVLBisG2 (16+), in mixtures with neutral 1,2-dioleoyl-sn-glycerophosphatidyl-choline (DOPC). To understand the TE behavior, we have performed X-ray diffraction (XRD), optical microscopy, and cryo-TEM studies of the DL/DOPC mixtures and their DNA complexes. XRD reveals a complex phase behavior of DL-DNA complexes which strongly depends on the headgroup charge. MVLG2(4+)/DOPC-DNA complexes exhibit the lamellar phase at all molar fractions of DL, Phi(DL). In stark contrast, MVLBisG2(16+)/ DOPC-DNA complexes remain lamellar only for Phi(DL) < or = 0.2. In a narrow regime around Phi(DL) = 0.25, the hexagonal phase H(I)(C), consisting of a hexagonal lattice of cylindrical lipid micelles and a DNA honeycomb lattice, is formed. At Phi(DL) > 0.3, XRD suggests formation of a distorted H(I)(C) phase. For Phi(DL) > or = 0.5 under high salt conditions, this phase coexists with a bundle phase of DNA condensed by the depletion-attraction effect of DL micelles. The transitions at high sigma(M) from the lamellar phase to the new hexagonal phases of DL-DNA complexes coincide with the deviation from the universal TE behavior of lamellar complexes. The observed high TE, which is independent of sigma(M), strongly suggests a novel mechanism of action for these DL-DNA complex phases.