Git does not support downloading parts of the repository. You have to download all of it. But you should be able to do this with GitHub.
When you view a file it has a link to the “raw” version. The URL is constructed like so
By filling in the blanks in the URL, you can use Wget or cURL or whatever to download a single file. Again, you won’t get any of the nice version control features used by Git by doing this.
Update: I noticed you mention this doesn’t work for binary files. You probably shouldn’t use binary files in your Git repository, but GitHub has a download section for each repository that you can use to upload files. If you need more than one binary, you can use a .zip file. The URL to download an uploaded file is: