With Augmented Reality, also termed AR, a view of the real world is augmented by superimposing computer-generated graphics, thereby enriching or enhancing the perception of the reality. Today, lots of applications benefit from AR in different areas, such as education, medicine, navigation, construction, gaming, and multiple other areas, using primarily head-mounted AR displays and AR on hand-held smart devices. Tablets and phones are highly suitable for AR, as they are equipped with high resolution screens, good cameras and powerful processing units, while being readily available to both industry and home use. They are used with video see-through AR, were the live view of the world is captured by a camera in real time and subsequently presented together with the computer graphics on the display. In this thesis I put forth our recent work on improving video see-through Augmented Reality graphics and interaction for hand-held devices by applying and utilizing user perspective. On the rendering side, we introduce a geometry-based user perspective rending method aiming to align the on screen content with the real view of the world visible around the screen. Furthermore, we introduce a device calibration system to compensate for misalignment between system parts. On the interaction side we introduce two wand-like direct 3D pose manipulation techniques based on this user perspective. We also modified a selection technique and introduced a new one suitable to be used with our introduced manipulation techniques. Finally, I present several formal user studies, evaluating the introduced techniques and comparing them with concurrent state-of-the-art alternatives.