In order to restrict that we can add an entry in web.config file of that Site Collections web application.
Add the following block
<location path="_vti_bin">
<system.web>
<authorization>
<allow users="sps\user" />
<deny users="*" />
</authorization>
</system.web>
</location>