You are almost correct; “didn’t” (or “did not” in its uncontracted form) is considered past-tense. Since you’ve already shifted the tense to the past, you no longer need to do so for the following verb - you are “already in” the past! So “didn’t go” vs “didn’t went”, “didn’t build” Vs “didn’t built”. The verbs following a past-tense auxiliary verb retain their simple form.