Toads- Philip Larkin

This poem kind of confused me. I got the fact that there were two toads and each represented something different. The first toad was called "work". The speaker begins the poem by stating that he works hard six days a week only to "pay a few bills". The speaker is saying that he is working too much and too hard and is not able to enjoy life. The "lecturers, lispers, losels" work at a minimum and still survive and lead happy lives. The speaker is envious of these people even though they are very poor. However, the phrase "to should stuff your pension!" states that the reader is too scared to take a chance and leave his money behind. The second toad is introduced after this. I'm not sure exactly what the second toad represents. The phrase "and will never allow me to blarney my way to getting the fame and the girl and the money all at one sitting" make me think that the second toad is the voice of reason within the speaker. The speaker wants to step away from his high-profile job so that he can enjoy the simple things in life and indulge in some pleasures. However, the voice of reason is telling him that he can't abandon his job because that will mean no income which will lead to difficulties down the road. I feel like I may be completely wrong on this but that was my initial interpretation.