Hi all, I would like to filter access by a portion of the URL. I have a HLS streaming server where the URLs are in the form: http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed) The server is behind DST-NAT I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL. Here has been my naive approach that did not work. I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8" And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150 I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct. Any advise appreciated. TIA, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
Interesting one. The Layer 7 stuff is only for the data within the packets, it's not for the URL as such, you will need to find something unique in the data stream for each of those connections and then try and match that.
From my experience find some really simple data to match within the packet, any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL Hi all, I would like to filter access by a portion of the URL. I have a HLS streaming server where the URLs are in the form: http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed) The server is behind DST-NAT I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL. Here has been my naive approach that did not work. I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8" And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150 I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct. Any advise appreciated. TIA, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Malcolm, When doing this from commandline, you need to be very particular about syntax. Give this a try: /ip firewall layer7-protocol add name=hls1 regexp="\\ /hls/1/1\\.m3u8" /ip firewall layer7-protocol add name=hls2 regexp="\\ /hls/2/2\\.m3u8" /ip firewall layer7-protocol add name=hls3 regexp="\\ /hls/3/3\\.m3u8" I haven't tested this myself, I based it on this: https://forum.mikrotik.com/viewtopic.php?t=130481 You may find it easier to edit via Winbox, where you don't have to worry about escaping the RouterOS commandline as well as escaping the regex statement :) Regards, Philip Loenneker | Network Engineer | TasmaNet -----Original Message----- From: Public <public-bounces@talk.mikrotik.com.au> On Behalf Of Paul Julian Sent: Friday, 31 May 2019 6:18 PM To: 'MikroTik Australia Public List' <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL Interesting one. The Layer 7 stuff is only for the data within the packets, it's not for the URL as such, you will need to find something unique in the data stream for each of those connections and then try and match that.
From my experience find some really simple data to match within the packet, any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul -----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL Hi all, I would like to filter access by a portion of the URL. I have a HLS streaming server where the URLs are in the form: http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed) The server is behind DST-NAT I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL. Here has been my naive approach that did not work. I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8" And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150 I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct. Any advise appreciated. TIA, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Friday, 31 May 2019 6:18 PM To: 'MikroTik Australia Public List' <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Interesting one.
The Layer 7 stuff is only for the data within the packets, it's not for
such, you will need to find something unique in the data stream for each of those connections and then try and match that. From my experience find some really simple data to match within the
any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL
Hi all,
I would like to filter access by a portion of the URL.
I have a HLS streaming server where the URLs are in the form:
http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed)
The server is behind DST-NAT
I need to allow / block access from different IP public addresses based on
http request header always includes the resource path requested, and almost always also the server hostname in plain text. So long as the request is not encrypted, it is possible to do request filtering using L7 inspection just on the first packet of the TCP request, and then add connection mark to deal with the entire request and response streams. So technique described by Malcolm is valid - EXCEPT that since the packet final destination is not the router itself, then forward chain is the correct filter chain to use. Input chain is only where the packet final destination is the router itself. Also, since L7 content inspection is quite CPU intensive, I would encourage to also limit the filter rule to first packet in the connection request only, i.e: add action=drop chain=forward comment="Drop all traffic to CCTV except allowed" \ dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes \ protocol=tcp src-address=!203.123.168.150 \ tcp-state=new It may not make much difference in this case where http request header is likely to fit inside one packet anyway, but if there is ever a need for the remote client to upload data to the server, then limit to the tcp-new packet will prevent inspection of every subsequent packet in the data stream. Cheers! the URL as packet, the
" hls/1/1.m3u8" part of the URL.
Here has been my naive approach that did not work.
I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8"
And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150
I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct.
Any advise appreciated.
TIA, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Thanks all for your advice. I will work on it next week and report back. Cheers. Malcolm Faed Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm +61 2 9939 4377 Unit 24 / 9 Powells Road, Brookvale - NSW 2100 avcomm.com.au This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Sat., 1 Jun. 2019, 17:54 Mike Everest, <mike@duxtel.com> wrote:
http request header always includes the resource path requested, and almost always also the server hostname in plain text. So long as the request is not encrypted, it is possible to do request filtering using L7 inspection just on the first packet of the TCP request, and then add connection mark to deal with the entire request and response streams.
So technique described by Malcolm is valid - EXCEPT that since the packet final destination is not the router itself, then forward chain is the correct filter chain to use. Input chain is only where the packet final destination is the router itself.
Also, since L7 content inspection is quite CPU intensive, I would encourage to also limit the filter rule to first packet in the connection request only, i.e:
add action=drop chain=forward comment="Drop all traffic to CCTV except allowed" \ dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes \ protocol=tcp src-address=!203.123.168.150 \ tcp-state=new
It may not make much difference in this case where http request header is likely to fit inside one packet anyway, but if there is ever a need for the remote client to upload data to the server, then limit to the tcp-new packet will prevent inspection of every subsequent packet in the data stream.
Cheers!
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Friday, 31 May 2019 6:18 PM To: 'MikroTik Australia Public List' <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Interesting one.
The Layer 7 stuff is only for the data within the packets, it's not for the URL as such, you will need to find something unique in the data stream for each of those connections and then try and match that. From my experience find some really simple data to match within the packet, any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL
Hi all,
I would like to filter access by a portion of the URL.
I have a HLS streaming server where the URLs are in the form:
http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed)
The server is behind DST-NAT
I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL.
Here has been my naive approach that did not work.
I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8"
And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150
I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct.
Any advise appreciated.
TIA, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi all, Thanks all for your advice. After a packet capture and working through a few experiments.... 0040 81 d6 47 45 54 20 2f 68 6c 73 2f 36 2f 36 2e 6d .ÖGET /hls/6/6.m 0050 33 75 38 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 3u8 HTTP/1.1..Us 0060 65 72 2d 41 67 65 6e 74 3a 20 4d 58 50 6c 61 79 er-Agent: MXPlay /ip firewall layer7-protocol add comment="Match HLS stream for NHK" name=NHK regexp="\\/hls\\/6\\/6.m3u8" (applied using GUI) /ip firewall address-list add address=xxx.xxx.xxx.xxx comment="Streaming Whitelist" list=\ "NHK Streaming Whitelist" add address=yyy.yyy.yyy.yyy comment="Streaming Whitelist" list=\ "NHK Streaming Whitelist" /ip firewall filter add action=drop chain=forward comment=\ "Drop all traffic to NHK except allowed by whitelist" dst-address=\ 10.zzz.zzz.zzz dst-port=8060 layer7-protocol=NHK log=yes protocol=tcp \ src-address-list="!NHK Streaming Whitelist" Using 'new' connection to lower CPU did not seem to work. I need to investigate further why. Regards, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Sat, 1 Jun 2019 at 18:15, Malcolm Faed <malcolm@avcomm.com.au> wrote:
Thanks all for your advice. I will work on it next week and report back.
Cheers.
Malcolm Faed Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm +61 2 9939 4377 Unit 24 / 9 Powells Road, Brookvale - NSW 2100 avcomm.com.au
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
On Sat., 1 Jun. 2019, 17:54 Mike Everest, <mike@duxtel.com> wrote:
http request header always includes the resource path requested, and almost always also the server hostname in plain text. So long as the request is not encrypted, it is possible to do request filtering using L7 inspection just on the first packet of the TCP request, and then add connection mark to deal with the entire request and response streams.
So technique described by Malcolm is valid - EXCEPT that since the packet final destination is not the router itself, then forward chain is the correct filter chain to use. Input chain is only where the packet final destination is the router itself.
Also, since L7 content inspection is quite CPU intensive, I would encourage to also limit the filter rule to first packet in the connection request only, i.e:
add action=drop chain=forward comment="Drop all traffic to CCTV except allowed" \ dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes \ protocol=tcp src-address=!203.123.168.150 \ tcp-state=new
It may not make much difference in this case where http request header is likely to fit inside one packet anyway, but if there is ever a need for the remote client to upload data to the server, then limit to the tcp-new packet will prevent inspection of every subsequent packet in the data stream.
Cheers!
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Friday, 31 May 2019 6:18 PM To: 'MikroTik Australia Public List' <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Interesting one.
The Layer 7 stuff is only for the data within the packets, it's not for the URL as such, you will need to find something unique in the data stream for each of those connections and then try and match that. From my experience find some really simple data to match within the packet, any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL
Hi all,
I would like to filter access by a portion of the URL.
I have a HLS streaming server where the URLs are in the form:
http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed)
The server is behind DST-NAT
I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL.
Here has been my naive approach that did not work.
I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8"
And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150
I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct.
Any advise appreciated.
TIA, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Mal, Use of 'new' to limit CPU may not have significant effect because the vast majority of http requests are likely to be just a single packet anyhow.. Perhaps if reverse the check to detect /allowed/ traffic rather than detect disallowed, then you can choose action of 'fast-track connection' instead which may provide some performance benefit. What kind of router hardware are you using for this purpose? L7 packet inspection is best managed by one of the multi-core CPU models like 1100, 4011 or CCR. Cheers, Mike.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Monday, 3 June 2019 3:49 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Hi all, Thanks all for your advice.
After a packet capture and working through a few experiments....
0040 81 d6 47 45 54 20 2f 68 6c 73 2f 36 2f 36 2e 6d .ÖGET /hls/6/6.m 0050 33 75 38 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 3u8 HTTP/1.1..Us 0060 65 72 2d 41 67 65 6e 74 3a 20 4d 58 50 6c 61 79 er-Agent: MXPlay
/ip firewall layer7-protocol add comment="Match HLS stream for NHK" name=NHK regexp="\\/hls\\/6\\/6.m3u8" (applied using GUI)
/ip firewall address-list add address=xxx.xxx.xxx.xxx comment="Streaming Whitelist" list=\ "NHK Streaming Whitelist" add address=yyy.yyy.yyy.yyy comment="Streaming Whitelist" list=\ "NHK Streaming Whitelist"
/ip firewall filter add action=drop chain=forward comment=\ "Drop all traffic to NHK except allowed by whitelist" dst-address=\ 10.zzz.zzz.zzz dst-port=8060 layer7-protocol=NHK log=yes protocol=tcp \ src-address-list="!NHK Streaming Whitelist"
Using 'new' connection to lower CPU did not seem to work. I need to investigate further why.
Regards, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
On Sat, 1 Jun 2019 at 18:15, Malcolm Faed <malcolm@avcomm.com.au> wrote:
Thanks all for your advice. I will work on it next week and report back.
Cheers.
Malcolm Faed Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm +61 2 9939 4377 Unit 24 / 9 Powells Road, Brookvale - NSW 2100 avcomm.com.au
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
On Sat., 1 Jun. 2019, 17:54 Mike Everest, <mike@duxtel.com> wrote:
http request header always includes the resource path requested, and almost always also the server hostname in plain text. So long as the request is not encrypted, it is possible to do request filtering using L7 inspection just on the first packet of the TCP request, and then add connection mark to deal with the entire request and response streams.
So technique described by Malcolm is valid - EXCEPT that since the packet final destination is not the router itself, then forward chain is the correct filter chain to use. Input chain is only where the packet final destination is the router itself.
Also, since L7 content inspection is quite CPU intensive, I would encourage to also limit the filter rule to first packet in the connection request only, i.e:
add action=drop chain=forward comment="Drop all traffic to CCTV except allowed" \ dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes \ protocol=tcp src-address=!203.123.168.150 \ tcp-state=new
It may not make much difference in this case where http request header is likely to fit inside one packet anyway, but if there is ever a need for the remote client to upload data to the server, then limit to the tcp-new packet will prevent inspection of every subsequent packet in the data stream.
Cheers!
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Friday, 31 May 2019 6:18 PM To: 'MikroTik Australia Public List' <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Interesting one.
The Layer 7 stuff is only for the data within the packets, it's not for the URL as such, you will need to find something unique in the data stream for each of those connections and then try and match that. From my experience find some really simple data to match within the packet, any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL
Hi all,
I would like to filter access by a portion of the URL.
I have a HLS streaming server where the URLs are in the form:
http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed)
The server is behind DST-NAT
I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL.
Here has been my naive approach that did not work.
I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8"
And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7- protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150
I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct.
Any advise appreciated.
TIA, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE- yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
Hi Mike, It is a repurposed Firebox X1250E . I am not sure what core it has. We don't have a huge throughput so it is working fine. It will soon to be replaced by a HP DL360G7 3.2GHz with 6 Ethernet ports for BGP. (Big step up!) I mention this as there is a lot of (mis) information regarding compatibility around, but the G7 loaded RouterOS and identified the Ethernet ports without issue. Regards, Mal *Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au Av-Comm Office: +61 2 9939 4377 Mobile+61 424 957 053 Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au [image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. On Mon, 3 Jun 2019 at 16:03, Mike Everest <mike@duxtel.com> wrote:
Hi Mal,
Use of 'new' to limit CPU may not have significant effect because the vast majority of http requests are likely to be just a single packet anyhow..
Perhaps if reverse the check to detect /allowed/ traffic rather than detect disallowed, then you can choose action of 'fast-track connection' instead which may provide some performance benefit.
What kind of router hardware are you using for this purpose? L7 packet inspection is best managed by one of the multi-core CPU models like 1100, 4011 or CCR.
Cheers, Mike.
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Monday, 3 June 2019 3:49 PM To: MikroTik Australia Public List <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Hi all, Thanks all for your advice.
After a packet capture and working through a few experiments....
0040 81 d6 47 45 54 20 2f 68 6c 73 2f 36 2f 36 2e 6d .ÖGET /hls/6/6.m 0050 33 75 38 20 48 54 54 50 2f 31 2e 31 0d 0a 55 73 3u8 HTTP/1.1..Us 0060 65 72 2d 41 67 65 6e 74 3a 20 4d 58 50 6c 61 79 er-Agent: MXPlay
/ip firewall layer7-protocol add comment="Match HLS stream for NHK" name=NHK regexp="\\/hls\\/6\\/6.m3u8" (applied using GUI)
/ip firewall address-list add address=xxx.xxx.xxx.xxx comment="Streaming Whitelist" list=\ "NHK Streaming Whitelist" add address=yyy.yyy.yyy.yyy comment="Streaming Whitelist" list=\ "NHK Streaming Whitelist"
/ip firewall filter add action=drop chain=forward comment=\ "Drop all traffic to NHK except allowed by whitelist" dst-address=\ 10.zzz.zzz.zzz dst-port=8060 layer7-protocol=NHK log=yes protocol=tcp \ src-address-list="!NHK Streaming Whitelist"
Using 'new' connection to lower CPU did not seem to work. I need to investigate further why.
Regards, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE-yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
On Sat, 1 Jun 2019 at 18:15, Malcolm Faed <malcolm@avcomm.com.au> wrote:
Thanks all for your advice. I will work on it next week and report back.
Cheers.
Malcolm Faed Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm +61 2 9939 4377 Unit 24 / 9 Powells Road, Brookvale - NSW 2100 avcomm.com.au
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
On Sat., 1 Jun. 2019, 17:54 Mike Everest, <mike@duxtel.com> wrote:
http request header always includes the resource path requested, and almost always also the server hostname in plain text. So long as the request is not encrypted, it is possible to do request filtering using L7 inspection just on the first packet of the TCP request, and then add connection mark to deal with the entire request and response streams.
So technique described by Malcolm is valid - EXCEPT that since the packet final destination is not the router itself, then forward chain is the correct filter chain to use. Input chain is only where the packet final destination is the router itself.
Also, since L7 content inspection is quite CPU intensive, I would encourage to also limit the filter rule to first packet in the connection request only, i.e:
add action=drop chain=forward comment="Drop all traffic to CCTV except allowed" \ dst-address='my public ip' dst-port=4545 layer7-protocol=CCTV log=yes \ protocol=tcp src-address=!203.123.168.150 \ tcp-state=new
It may not make much difference in this case where http request header is likely to fit inside one packet anyway, but if there is ever a need for the remote client to upload data to the server, then limit to the tcp-new packet will prevent inspection of every subsequent packet in the data stream.
Cheers!
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Paul Julian Sent: Friday, 31 May 2019 6:18 PM To: 'MikroTik Australia Public List' <public@talk.mikrotik.com.au> Subject: Re: [MT-AU Public] Filtering by part of URL
Interesting one.
The Layer 7 stuff is only for the data within the packets, it's not for the URL as such, you will need to find something unique in the data stream for each of those connections and then try and match that. From my experience find some really simple data to match within the packet, any work with layer 7 for me has always been hit and miss for me, sometimes it seems that the data needs to be in regex, sometimes not, it's strange.
Regards Paul
-----Original Message----- From: Public [mailto:public-bounces@talk.mikrotik.com.au] On Behalf Of Malcolm Faed Sent: Friday, 31 May 2019 5:57 PM To: MikroTik Australia Public List Subject: [MT-AU Public] Filtering by part of URL
Hi all,
I would like to filter access by a portion of the URL.
I have a HLS streaming server where the URLs are in the form:
http://123.123.123.123:4545/hls/1/1.m3u8 http://123.123.123.123:4545/hls/2/2.m3u8 http://123.123.123.123:4545/hls/3/3.m3u8 (These can not be changed)
The server is behind DST-NAT
I need to allow / block access from different IP public addresses based on the " hls/1/1.m3u8" part of the URL.
Here has been my naive approach that did not work.
I tried creating a layer 7 filter add comment="Match HLS stream for CCTV" name=CCTV regexp="\/hls\/1\/1.m3u8"
And using this in the filter add action=drop chain=input comment="Drop all traffic to CCTV except allowed" dst-address='my public ip' dst-port=4545 layer7- protocol=CCTV log=yes protocol=tcp src-address=!203.123.168.150
I also tried the forward chain based on the wiki, but since this is behind a DST NAT, I think input is correct.
Any advise appreciated.
TIA, Mal
*Malcolm Faed*Network Broadcast Engineer malcolm@avcomm.com.au
Av-Comm
Office: +61 2 9939 4377 Mobile+61 424 957 053
Unit 24 / 9 Powells Road, Brookvale, NSW 2100, Australia. avcomm.com.au
[image: Twitter] <https://twitter.com/AvCommSatellite>[image: Google Plus] <https://plus.google.com/+AvcommAustralia/>[image: Youtube] <https://www.youtube.com/channel/UCO8ZtcnwoTH7e54LAndE- yw>[image: Linkedin] <https://www.linkedin.com/company-beta/6583589/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorised disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Av-Comm is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au
http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
_______________________________________________ Public mailing list Public@talk.mikrotik.com.au http://talk.mikrotik.com.au/mailman/listinfo/public_talk.mikrotik.com.au
participants (4)
-
Malcolm Faed -
Mike Everest -
Paul Julian -
Philip Loenneker