To get back the advance I paid

Dear Sir, I and my brother went to see a house , it is a town ship and only few houses has been build the builder told us that he will finish it within 2 years. Now when we went he told us that day was last to get a discount which they were providing as special rate. And asked us to pay advance and we told the builder and told we cannot buy as we dont have money and I have already bank loan , and I dont even have papers for bank loan. He told us it is ok he will arrange loan. We paid 5000 on sunday and on monday he sent a person to my office for more advance and I paid 50000 for 1 shop and a house. Then after paying he started asking money I paid Rs.7,71000 total (we took loan from a friend thought after getting loan we will return). And the builder was not saying about loan we asked him then he said you dont have proper documents and u already have loan, you will not get loan. Then we told him that in that condition we will buy the shop as we cannot afford to buy the house. He says he will not return the money of the house we paid 5.51000 , we then told him to put that amount in the shop he says he wont do that too. Now I am stuck, I cant pay for the house as I dont have money, And the money I gave him he says neither he will return nor he will adjust it with the shop. Is there any ways that I can get out of this. Please advice. He has given an agreement