From: Digest To: "OS/2GenAu Digest" Date: Sat, 12 Dec 2009 00:00:34 EST-10EDT,10,1,0,7200,4,1,0,7200,3600 Subject: [os2genau_digest] No. 1888 Reply-To: X-List-Unsubscribe: www.os2site.com/list/ ************************************************** Friday 11 December 2009 Number 1888 ************************************************** Subjects for today 1 Re: os2site dot com : Ian Manners" 2 Re: Slow writing to USB 2.0 : Ian Manners" 3 Re: os2site dot com : Ed Durrant 4 Re: Slow writing to USB 2.0 : Wayne 5 Re: Slow writing to USB 2.0 : Wayne **= Email 1 ==========================** Date: Fri, 11 Dec 2009 00:55:22 +1100 (EDT) From: "Ian Manners" Subject: Re: os2site dot com Hi Ed, >I wonder if there is any way to spot when a multiple file download is >occurring? i.e. someone is trying to mirror your site. Problem thats been happening a lot, especially in the past 4-5 weeks is some smart arse has set themselves up to use different UA (User Agents), on different IP address's from all over the world, for a couple of hours each. Each one is below the blacklist threshold (which I cant tighten up any more). So each block looks like a normal user. I can see the pattern occurring when I view the logs after sorting. I can setup a rule for that but it will impact other people to the extent of making normal site usage impractical, not to mention very annoying. Users are blacklisted for X time if they a) use Y concurrent download threads b) exceed Y amount of data c) use various UA's d) use a fake UA on a wrong IP (ie, Ukriane with a Google UA) e) try to access or download trap files. Those that keep doing it end up on the permanent blacklist in the firewall. >Perhaps supporting transfer only via a HTTP access from a "portal" on >your website, that would only let someone download one file at a time >could be an approach - a "door keeper" if you like. I have it set for 4 threads per IP per file, anything lower leads to problems. There is a preset limit that if exceeded leads to the user being blacklisted for a period of time as well >Probably turn off FTP straight away. Can you tell if the sites that are >accessing are using HTTP or FTP ? HTTP, you want FTP access, you have to ask me :-) It's amazing how many people blast me over my access rules to what I thought was my website, my answer is normally, you pay for the data, you can download the data :o) I figure as it is essentially a free service, the website can take the rest of the month off while I think about it at my leisure, anyone that wants urgent access can let me know and I'll email them a username/password sometime next week, if its ultra urgent I'll email them the file or email back a URL :-) The next 3 days I'm going to be very busy on my real life things. Cheers Ian Manners http://www.os2site dot com/ -------------------------------------------------- http://www./melbpc/ - The Melbourne OS/2 SIG **= Email 2 ==========================** Date: Fri, 11 Dec 2009 00:58:07 +1100 (EDT) From: "Ian Manners" Subject: Re: Slow writing to USB 2.0 Hi Peter, >I can explain that part. The check to see whether the sender is a >legitimate list member is done on the address in the SMTP command "MAIL >FROM:" rather than on the "From:" header line. The reasoning here is >that, although spammers can fake either address, there's a much higher >probability that a spammer will fake the "From:". Yes, now you mention it I can remember this coming up in the past and I think you said the same thing :-) Cheers Ian Manners http://www.os2site dot com/ -------------------------------------------------- http://www./melbpc/ - The Melbourne OS/2 SIG **= Email 3 ==========================** Date: Fri, 11 Dec 2009 05:46:40 +1100 From: Ed Durrant Subject: Re: os2site dot com Ian Manners wrote: > Hi Ed, > > >> I wonder if there is any way to spot when a multiple file download is >> occurring? i.e. someone is trying to mirror your site. >> > > Problem thats been happening a lot, especially in the past 4-5 > weeks is some smart arse has set themselves up to use different > UA (User Agents), on different IP address's from all over the world, > for a couple of hours each. Each one is below the blacklist > threshold (which I cant tighten up any more). So each block looks > like a normal user. > > I can see the pattern occurring when I view the logs after sorting. > > I can setup a rule for that but it will impact other people to the extent > of making normal site usage impractical, not to mention very > annoying. > > Users are blacklisted for X time if they > > a) use Y concurrent download threads > b) exceed Y amount of data > c) use various UA's > d) use a fake UA on a wrong IP (ie, Ukriane with a Google UA) > e) try to access or download trap files. > > Those that keep doing it end up on the permanent blacklist in the > firewall. > > >> Perhaps supporting transfer only via a HTTP access from a "portal" on >> your website, that would only let someone download one file at a time >> could be an approach - a "door keeper" if you like. >> > > I have it set for 4 threads per IP per file, anything lower leads to > problems. There is a preset limit that if exceeded leads to the > user being blacklisted for a period of time as well > > >> Probably turn off FTP straight away. Can you tell if the sites that are >> accessing are using HTTP or FTP ? >> > > HTTP, you want FTP access, you have to ask me :-) > > It's amazing how many people blast me over my access rules to > what I thought was my website, my answer is normally, you > pay for the data, you can download the data :o) > > I figure as it is essentially a free service, the website can take > the rest of the month off while I think about it at my leisure, anyone > that wants urgent access can let me know and I'll email them > a username/password sometime next week, if its ultra urgent > I'll email them the file or email back a URL :-) > > The next 3 days I'm going to be very busy on my real life things. > > Cheers > Ian Manners > http://www.os2site dot com/ > > -------------------------------------------------- > > http://www./melbpc/ - The Melbourne OS/2 SIG > Its a real shame that someone would want to do this, without respect for the fact that you get stung for the network bandwidth used. Another idea, presuming this is a "bot" doing this, how difficult would it be to implement a "catchya" feature (I think that's what it is called, where a human readable set of random letters and numbers are displayed and have to be entered before access to a file is given? Needs some thinking through, so I agree with you closing off all access in the interim is probably a good idea - it may of itself stop the bot and they may give up in the meantime. -- Cheers/2 Ed eComStationAustralia podcast RSS feed http://eComStationAustralia.podbean dot com/feed or iTunes -------------------------------------------------- http://www./melbpc/ - The Melbourne OS/2 SIG === **= Email 4 ==========================** Date: Fri, 11 Dec 2009 07:14:32 +0930 From: Wayne Subject: Re: Slow writing to USB 2.0 ** Reply to note from smee.wayne+caf_=datablitz=three dot com dot au at gmail dot com Thu, 10 Dec 2009 07:56:57 +0800 > > [attachments have been removed] > -------------------------------------------------- > > http://www./melbpc/ - The Melbourne OS/2 SIG === Gmail does have problems at times. This email seems to be originated from 1 of my gmail accounts but I didn't send it. Nor have I sent anything vaguely like it anywhere & nothing to os2genau. Really seems like gmail is at fault. 1 of my gmail accounts is forwarded to three dot com dot au. Cheers Wayne -------------------------------------------------- http://www./melbpc/ - The Melbourne OS/2 SIG === **= Email 5 ==========================** Date: Fri, 11 Dec 2009 06:56:15 +0930 From: Wayne Subject: Re: Slow writing to USB 2.0 ** Reply to note from smee.wayne+caf_=datablitz=three dot com dot au at gmail dot com Thu, 10 Dec 2009 07:56:57 +0800 > > [attachments have been removed] > -------------------------------------------------- > > http://www./melbpc/ - The Melbourne OS/2 SIG === Gmail does have problems at times. This email seems to be originated from 1 of my gmail accounts but I didn't send it. Nor have I sent anything vaguely like it anywhere & nothing to os2genau. Really seems like gmail is at fault. 1 of my gmail accounts is forwarded to three dot com dot au. Cheers Wayne -------------------------------------------------- http://www./melbpc/ - The Melbourne OS/2 SIG ===