Retinotopic maps are plastic in response to changes in sensory input; however, the experience-dependent instructive cues that organize retinotopy are unclear. In animals with forward-directed locomotion, the predominant anterior to posterior optic flow activates retinal ganglion cells in a stereotyped temporal to nasal sequence. Here we imaged retinotectal axon arbor location and structural plasticity to assess map refinement in vivo while exposing Xenopus tadpoles to visual stimuli. We show that the temporal sequence of retinal activity driven by natural optic flow organizes retinotopy by regulating axon arbor branch dynamics, whereas the opposite sequence of retinal activity prevents map refinement. Our study demonstrates that a spatial to temporal to spatial transformation of visual information controls experience-dependent topographic map plasticity. This organizational principle is likely to apply to other sensory modalities and projections in the brain.