Current visualization design toolkits help users to explore data and create visualizations. However, most of these systems do not record the executed actions during the visualization construction process with semantic context. In this paper, we present VisAct, a visualization design system based on semantic actions, that helps average users to construct visualizations step by step. Our system contributes a set of action-based visualization components and a high-level grammar for semantic actions. VisAct also guides the visualization construction process and provides an action tracker for history management and data collection. We demonstrate the usability of VisAct by visualizations and a plugin application. Finally, we conduct a user study to evaluate the efficiency and effectiveness of our system.