Ssh-add always fails when opening a new terminal

Here's the output:

➜  ~ ssh-add
Could not open a connection to your authentication agent.
➜  ~ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XXXXXXkMT84G/agent.58624; export SSH_AUTH_SOCK;
SSH_AGENT_PID=58625; export SSH_AGENT_PID;
echo Agent pid 58625;
➜  ~ ssh-add
Could not open a connection to your authentication agent.
➜  ~

I just want to be able to run ssh-add, be asked for my password once and then that's it.