Methods, systems, and apparatuses are described herein for improving computer authentication processes using vocal confidence processing. A request for access to an account may be received. An authentication question may be provided to a user. Voice data indicating one or more vocal utterances by the user in response to the authentication question may be received. The voice data may be processed, and a first confidence score that indicates a degree of confidence of the user when answering the authentication question may be determined. An overall confidence score may be modified based on the first confidence score. Based on determining that the overall confidence score satisfies a threshold, data preventing the authentication question from being used in future authentication processes may be stored. The data may be removed when a time period expires.