Squid with Cache Youtube Puas rasanya, akhirnya bisa caching youtube (dan yang similar) :D. Setelah sekian bulan “mangkrak” akhirnya jalan juga walau masih di mesin virtualbox. Howto ini bukan untuk para pembenci youtube dan googlemap. Tapi untuk youtube dan googlemap lovers. bahan referensi yang jadi bacaan. http://www.mail-archive.com/
[email protected]/msg54605.html http://www.mail-archive.com/
[email protected]/msg51076.html http://wiki.squid-cache.org/Features/StoreUrlRewrite http://wiki.squid-cache.org/Features/StoreUrlRewrite/RewriteScript Versi yang saya pakai adalah squid-2.7.STABLE3, tidak tahu dukungan untuk versi yang lain. 1. buat script untuk manipulasi youtube. 2. 3. 4. 5. 6. 7.
#!/usr/bin/perl $|=1; while (<>) { @X = split; $url = $X[0]; $url =~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)&.*@squid://videos .youtube.INTERNAL/ID=$3@; 8. $url =~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)$@squid://videos.y outube.INTERNAL/ID=$3@; 9. $url =~s@^http://(.*?)/videodownload\?(.*)docid=(.*?)$@squid://videos. google.INTERNAL/ID=$3@; 10.$url =~s@^http://(.*?)/videodownload\?(.*)docid=(.*?)&.*@squid://video s.google.INTERNAL/ID=$3@; print "$url\n"; }
11. Lalu di squid.conf-nya edit seperti yang dibawah ini: 12.acl store_rewrite_list url_regex ^http://(.*?)/get_video\? 13.acl store_rewrite_list url_regex ^http://(.*?)/videodownload\? 14.cache allow store_rewrite_list 15. 16.# Had to uncomment this again, because I couln'd login to google mail using IE6 (firefox had no trouble): 17.acl QUERY urlpath_regex cgi-bin \? 18.cache deny QUERY 19. 20.refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private 21.refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999 override-expire ignore-no-cache ignore-private 22.
23.storeurl_access allow store_rewrite_list 24.storeurl_access deny all 25. storeurl_rewrite_program /usr/local/bin/store_url_rewrite
Hasilnya bisa dilihat di access-log, pada saat mengakses video yang sama, akan langsung hit. # grep youtube access.log | grep TCP_HIT 1214834411.379 735 192.168.1.89 TCP_HIT/200 1604459 GET http://youtube.com/get_video?video_id=2d55BSiJdM&t=OEgsToPDskKrwAAE_vVIhOqMhPqmPDUQ - NONE/- video/flv 1214834487.090 818 192.168.1.94 TCP_HIT/200 1604459 GET http://youtube.com/get_video?video_id=2d55BSiJdM&t=OEgsToPDskLGVqEnxKjLEN4DGA3HYGse - NONE/- video/flv 1214836269.353 4383 192.168.1.91 TCP_HIT/200 9533167 GET http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskKeQxYVvYZ 7fgEIW4UNC_U- - NONE/- video/flv 1214836514.802 3757 192.168.1.91 TCP_HIT/200 9533167 GET http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskIEwsTb26L iGFc96hBUUa9Z - NONE/- video/flv
Satu pesan dari Horacio Herrera Gonzalez, karena basic scriptnya tidak spesifik ke url tertentu, maka : Warning! This code may match other sites not related to YT or GV. He he he he, watching your bandwidth. Karena beberapa user merasa kesulitan untuk mengaplied caching youtube. Langkah dibawah adalah urutan di server saya. 1. Saya pakai distro TSL 3.05, dengan squid squid-2.7.STABLE3 2. ./configure \ 3. --sysconfdir=/etc/squid \ 4. --prefix=/usr \ 5. --enable-async-io \ 6. --enable-removal-policies=lru,heap \ 7. --disable-delay-pools \ 8. --disable-wccp \ 9. --disable-wccp2 \ 10.--enable-kill-parent-hack \ 11.--enable-snmp \ 12.--enable-default-err-languages=English --enable-errlanguages=English \ 13.--enable-linux-netfilter \ 14.--disable-auth
15. config hasil parsing ^# dari squid.conf 16.acl all src all 17.acl manager proto cache_object 18.acl localhost src 127.0.0.1/32
19.acl to_localhost dst 127.0.0.0/8 20.acl localnet src 10.0.0.0/8 # RFC1918 possible internal network 21.acl localnet src 172.16.0.0/12 # RFC1918 possible internal network 22.acl localnet src 192.168.0.0/16 # RFC1918 possible internal network 23.acl SSL_ports port 443 24.acl Safe_ports port 80 # http 25.acl Safe_ports port 21 # ftp 26.acl Safe_ports port 443 # https 27.acl Safe_ports port 70 # gopher 28.acl Safe_ports port 210 # wais 29.acl Safe_ports port 1025-65535 # unregistered ports 30.acl Safe_ports port 280 # http-mgmt 31.acl Safe_ports port 488 # gss-http 32.acl Safe_ports port 591 # filemaker 33.acl Safe_ports port 777 # multiling http 34.acl CONNECT method CONNECT 35.http_access allow manager localhost 36.http_access deny manager 37.http_access deny !Safe_ports 38.http_access deny CONNECT !SSL_ports 39.http_access allow localnet 40.http_access deny all 41.icp_access allow localnet 42.icp_access deny all 43.http_port 3128 transparent 44.hierarchy_stoplist cgi-bin ? 45.cache_mem 6 MB 46.maximum_object_size_in_memory 32 KB 47.memory_replacement_policy heap GDSF 48.cache_replacement_policy heap LFUDA 49.cache_dir aufs /nfs/cache 20000 16 256 50.maximum_object_size 64 MB 51.cache_swap_low 98 52.cache_swap_high 99 53.access_log /var/log/squid/access.log squid 54.cache_log /var/log/squid/cache.log 55.cache_store_log none 56.log_fqdn off 57.storeurl_rewrite_program /etc/squid/store_url_rewrite 58.acl store_rewrite_list url_regex ^http://(.*?)/get_video\? 59.acl store_rewrite_list url_regex ^http://(.*?)/videodownload\? 60.storeurl_access allow store_rewrite_list 61.storeurl_access deny all 62.cache allow store_rewrite_list 63.acl QUERY urlpath_regex cgi-bin \? 64.cache deny QUERY 65.refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private 66.refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999 override-expire ignore-no-cache ignore-private 67.refresh_pattern ^ftp: 1440 20% 10080 68.refresh_pattern ^gopher: 1440 0% 1440 69.refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 70.refresh_pattern . 0 20% 4320
71.quick_abort_min 0 72.quick_abort_max 0 73.quick_abort_pct 98 74.acl apache rep_header Server ^Apache 75.broken_vary_encoding allow apache 76.vary_ignore_expire on 77.cache_effective_user squid 78.cache_effective_group squid 79.log_icp_queries off 80.ipcache_size 2048 81.ipcache_low 98 82.ipcache_high 99 83.memory_pools off 84.reload_into_ims on 85.coredump_dir /usr/var/cache pipeline_prefetch on
Caching photobucket Kontribusi apit1, caching untuk photobucket yang banyak di pakai di friendster. di store_url_rewrite $url =~s@^http://(.*?)/albums\?&.*@squid://images.photobucket.INTERNAL/ID=$3 @; $url =~s@^http://(.*?)/albums\?$@squid://images.photobucket.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/albums\?&.*@squid://videos.photobucket.INTERNAL/ID=$3 @; $url =~s@^http://(.*?)/albums\?$@squid://videos.photobucket.INTERNAL/ID=$3@;
di squid.conf acl store_rewrite_list url_regex ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? acl store_rewrite_list url_regex ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\? refresh_pattern ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? 43200 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\? 43200 90% 999999 override-expire ignore-no-cache ignore-private
Hasilnya TCP_HIT/200 5474813 GET http://vid264.photobucket.com/albums/ii163/shannonwiseman12/DSCN0212.fl v - NONE/- text/plain
Hasilnya bisa dilihat di access-log, pada saat mengakses video yang sama, akan langsung hit. # grep youtube access.log | grep TCP_HIT 1214834411.379 735 192.168.1.89 TCP_HIT/200 1604459 GET http://youtube.com/get_video?video_id=2d55BSiJdM&t=OEgsToPDskKrwAAE_vVIhOqMhPqmPDUQ - NONE/- video/flv 1214834487.090 818 192.168.1.94 TCP_HIT/200 1604459 GET http://youtube.com/get_video?video_id=2d55BSiJdM&t=OEgsToPDskLGVqEnxKjLEN4DGA3HYGse - NONE/- video/flv 1214836269.353 4383 192.168.1.91 TCP_HIT/200 9533167 GET http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskKeQxYVvYZ 7fgEIW4UNC_U- - NONE/- video/flv 1214836514.802 3757 192.168.1.91 TCP_HIT/200 9533167 GET http://youtube.com/get_video?video_id=i6cKRT12jgw&t=OEgsToPDskIEwsTb26L iGFc96hBUUa9Z - NONE/- video/flv
Satu pesan dari Horacio Herrera Gonzalez, karena basic scriptnya tidak spesifik ke url tertentu, maka : Warning! This code may match other sites not related to YT or GV. He he he he, watching your bandwidth. Karena beberapa user merasa kesulitan untuk mengaplied caching youtube. Langkah dibawah adalah urutan di server saya. 1. Saya pakai distro TSL 3.05, dengan squid squid-2.7.STABLE3 2. ./configure \ 3. --sysconfdir=/etc/squid \ 4. --prefix=/usr \ 5. --enable-async-io \ 6. --enable-removal-policies=lru,heap \ 7. --disable-delay-pools \ 8. --disable-wccp \ 9. --disable-wccp2 \ 10.--enable-kill-parent-hack \ 11.--enable-snmp \ 12.--enable-default-err-languages=English --enable-errlanguages=English \ 13.--enable-linux-netfilter \ 14.--disable-auth
Squid.conf config hasil parsing ^# dari squid.conf acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access deny all icp_access allow localnet icp_access deny all http_port 3128 transparent hierarchy_stoplist cgi-bin ? cache_mem 6 MB maximum_object_size_in_memory 32 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /nfs/cache 20000 16 256 maximum_object_size 64 MB cache_swap_low 98 cache_swap_high 99 access_log /var/log/squid/access.log squid cache_log /var/log/squid/cache.log cache_store_log none log_fqdn off storeurl_rewrite_program /etc/squid/store_url_rewrite acl store_rewrite_list url_regex ^http://(.*?)/get_video\? acl store_rewrite_list url_regex ^http://(.*?)/videodownload\? storeurl_access allow store_rewrite_list storeurl_access deny all cache allow store_rewrite_list acl QUERY urlpath_regex cgi-bin \? cache deny QUERY
refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 overrideexpire ignore-no-cache ignore-private refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 quick_abort_min 0 quick_abort_max 0 quick_abort_pct 98 acl apache rep_header Server ^Apache broken_vary_encoding allow apache vary_ignore_expire on cache_effective_user squid cache_effective_group squid log_icp_queries off ipcache_size 2048 ipcache_low 98 ipcache_high 99 memory_pools off reload_into_ims on coredump_dir /usr/var/cache pipeline_prefetch on
Caching photobucket Kontribusi apit1, caching untuk photobucket yang banyak di pakai di friendster. di store_url_rewrite $url =~s@^http://(.*?)/albums\?&.*@squid://images.photobucket.INTERNAL/ID=$3 @; $url =~s@^http://(.*?)/albums\?$@squid://images.photobucket.INTERNAL/ID=$3@; $url =~s@^http://(.*?)/albums\?&.*@squid://videos.photobucket.INTERNAL/ID=$3 @; $url =~s@^http://(.*?)/albums\?$@squid://videos.photobucket.INTERNAL/ID=$3@;
di squid.conf acl store_rewrite_list url_regex ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\?
acl store_rewrite_list url_regex ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\? refresh_pattern ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\? 43200 90% 999999 override-expire ignore-no-cache ignore-private refresh_pattern ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\? 43200 90% 999999 override-expire ignore-no-cache ignore-private
Hasilnya TCP_HIT/200 5474813 GET http://vid264.photobucket.com/albums/ii163/shannonwiseman12/DSCN0212.fl v - NONE/- text/plain