HPC Pack install guide : Science,Server & Web : (c)Rupert S
****
The point of this document is to make installing and managing HPC High Performance Computation possible,
To be clear A person or organisation that wishes to have high performance cloud nodes with docker..
Have the capacity to install Microsoft HPC pack & Docker, The configuration settings are at the top,
In the lower half are the docker VM's you can potentially use to University,Gamer & Corporate advantage.
Clearly the bat files make install easier than running individual commands one by one..
Both the bat files and the commands are in the document,
The work in this document took months & hereby is offered to the research community.
Firstly HPC pack should be installed on windows; However the docker configuration works on Linux..
However you have to first install Docker on for example Ubuntu,
Of-course the main server for Microsoft HPC pack needs to be windows (Preferably Server 2008/2016/2018),
Nodes can be Windows 7 to 10 & Server 2008 to 2018 & Linux (Mainly REDHAT,CentOS,Ubuntu,SuSE)
Personally CentOS rocks for servers, Ubuntu has professional server configurations & lots of options,RedHAT is the main line of secure server that registers all security & tests all.
In my experience RedHAT offers large corporation support; As do Ubuntu & CentOS & Obviously Microsoft offers full confidence.
Rupert S
****
Node configuration guidelines:
Head node,Workstation/Server
Nodes (potentially 2+)
Cloud nodes (potentially limitless)
RS
****
Example Plans :
Include every 8 to 256 core Compute node (Win Serve 2 Win 10) into the : Cfana Ridge Connect(tm)Global Compute Network: Operational & functional: For every HPC Workstation 8 Core to 256 Core, 4gb RAM to 4Tb RAM
****
Linux HPC Node install
****# (c)RS Updates & Security
# Creates an efficient & Cohesive update policy that does not require huge thought..
# Also Security SSL
# Run as ADMIN
# https://is.gd/Security_BAT
https://www.microsoft.com/en-us/download/details.aspx?id=54507 Microsoft HPC Pack 2016 including linux
Router/Firewall ports:53,80,123,319,320,443,853,1812,1813,2377,2408,2480,3128,3130,5353,7680,7844,8080,11211
Router/Firewall ports HPC Pack: 53,80,123,319,320,443,853,1812,1813,1856,2377,2408,2480,3128,3130,5353,5800,5801,5969,5970,5974,5999,6729,6730,7680,7844,7997,8677,8080,9087,9090,9091,9092,9094,9095,9096,9794,9892,9893,11211
https://is.gd/HPCData_P
Windows server: Step 1
install Microsoft HPC Pack - Microsoft HPC Pack server/Workstation/Unmanaged server/node
Head Node
127.0.0.1,dn.n-helix.com,node.ns4v.icu,worldcommunitygrid.org,lhcathome.cern.ch,cern.ch,boinc.bakerlab.org,cloudflareworkers.com,cloudflare.com,cloud.sfr.fr,sfrcloud.sfr.fr,webrlb01.cern.ch,boincai01.cern.ch,8.8.4.4,1.1.1.1
You will need a server certificate such as a comodo or digicert or microsoft.. for the next page of hpc pack install,
A node address such as https://www.dynu.com for the node
https://www.gogetssl.com/compare-domain-sslcertificates/
https://certifytheweb.com/home/
https://letsencrypt.org/docs/client-options/
powershell Install-PackageProvider -Name NuGet -Force
powershell Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
powershell Install-Package -Name docker -ProviderName DockerMsftProvider -Force
or run one of these instead of step 1 & 2:
For Intel GPU : https://is.gd/DockerIntelServerInstall
For Nvidia GPU : https://is.gd/DockerNvidiaServerInstall
For AMD GPU : https://is.gd/DockerAMDServerInstall
Windows 7/8/10: Step 1
install Microsoft HPC Pack node or workstation/server
Head Node
127.0.0.1,dn.n-helix.com,node.ns4v.icu,worldcommunitygrid.org,lhcathome.cern.ch,cern.ch,boinc.bakerlab.org,cloudflareworkers.com,cloudflare.com,cloud.sfr.fr,sfrcloud.sfr.fr,webrlb01.cern.ch,boincai01.cern.ch,8.8.4.4,1.1.1.1
You will need a server certificate such as a comodo or digicert or microsoft.. for the next page of hpc pack install,
A node address such as https://www.dynu.com for the node
https://www.gogetssl.com/compare-domain-sslcertificates/
https://certifytheweb.com/home/
https://letsencrypt.org/docs/client-options/
powershell Install-PackageProvider -Name NuGet -Force
powershell Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
powershell Install-Package -Name docker -ProviderName DockerMsftProvider -Force
and then install docker CE
https://docker.com
run one of these after installing hpc pack & docker: Instead of Step 2
For Intel GPU : https://is.gd/DockerIntelServerInstall
For Nvidia GPU : https://is.gd/DockerNvidiaServerInstall
For AMD GPU : https://is.gd/DockerAMDServerInstall
For Nvidia GPU : https://is.gd/DockerNvidiaServerInstall
For AMD GPU : https://is.gd/DockerAMDServerInstall
Step 2 for both : Enable Kubernets in docker and set available CPU's & Resources then >
or do the admin : CMD List or run the script above >admin : CMD
docker run --name railgun-memcached -d -p=127.0.0.1:11211:11211 -e https_proxy=http://127.0.0.1:3128 -e http_proxy=http://127.0.0.1:3128 -e ftp_proxy=http://127.0.0.1:3128 --restart=always memcached:latest
docker run -d --name=railgun -p 2408:2408 -e RG_ACT_TOKEN=ENTERapiTOKENHERE -e RG_ACT_HOST=127.0.0.1 -e RG_LOG_LEVEL=1 -e RG_WAN_PORT=2408 -e https_proxy=http://127.0.0.1:3128 -e http_proxy=http://127.0.0.1:3128 -e ftp_proxy=http://127.0.0.1:3128 --link railgun-memcached:memcached --restart=always rungeict/cloudflare-railgun
docker run -d --name=squid-memcached -p=127.0.0.1:3128:3128 -e RG_ACT_HOST=127.0.0.1 -e RG_LOG_LEVEL=1 -e https_proxy=http://127.0.0.1:3128 -e http_proxy=http://127.0.0.1:3128 -e ftp_proxy=http://127.0.0.1:3128 --link railgun-memcached:memcached --restart=always minimum2scp/squid
docker run -d --name=nuster-memcache -p=127.0.0.1:8080:8080 -p=127.0.0.1:443:443 -p=127.0.0.1:80:80 -e RG_ACT_HOST=127.0.0.1 -e RG_LOG_LEVEL=1 -e https_proxy=http://127.0.0.1:3128 -e http_proxy=http://127.0.0.1:3128 -e ftp_proxy=http://127.0.0.1:3128 --restart=always nuster/nuster
docker run --name binddns-resolver -d --restart=always -e https_proxy=http://127.0.0.1:3128 -e http_proxy=http://127.0.0.1:3128 -e ftp_proxy=http://127.0.0.1:3128 -p 53:53/tcp -p 53:53/udp -p 5353:5353/tcp -p 5353:5353/udp -p 10000:10000/tcp --volume /srv/docker/bind:/data sameersbn/bind
# to configure and update binddns-resolver : https://localhost:10000
# alternative requires to be configured docker run --name=binddns-resolver -d -p 53:53/udp -p 53:53 -d -p 5353:5353/udp -p 5353:5353 -e https_proxy=http://127.0.0.1:3128 -e http_proxy=http://127.0.0.1:3128 -e ftp_proxy=http://127.0.0.1:3128 -d --restart=always --volume /srv/docker/bind:/data unixtastic/bind-dnssec-resolver
docker run --name nax-waf -e BACKEND_IP=127.0.0.1 -e KIBANA_PASSWORD=Safe5 -d --link railgun-memcached:memcached --restart=always scollazo/naxsi-waf-with-ui
docker exec squid-memcached curl http://127.0.0.1/
****
When you enable Kubernet in docker in the console (CMD) : kubectl: Kubernets ingress controller security: If you use cloudflared & or a local certificate : the HPC Certificate you got: https://kubernetes.io/
(CMD):
kubectl create secret generic localhost --from-file="$HOME/.
****
Selenium runs once - you need to re run it on boot
https://store.docker.com/community/images/selenium/standalone-chrome
https://github.com/SeleniumHQ/docker-selenium
docker run -d -it --rm -p=0.0.0.0:9222:9222 -p=0.0.0.0:443:443 --link railgun-memcached:memcached --name=chrome-headless-selenium -v /tmp/chromedata/:/data selenium/standalone-chrome
port : 4444
****
*waf firewall*
https://hub.docker.com/r/scollazo/naxsi-waf-with-ui/
docker pull scollazo/naxsi-waf-with-ui
*zend php stack is licenced*
https://hub.docker.com/_/php-zendserver/
docker pull php-zendserver
*nginx php stack*
https://hub.docker.com/r/richarvey/nginx-php-fpm/
docker pull richarvey/nginx-php-fpm
*node js*
https://hub.docker.com/r/mhart/alpine-node/docker pull mhart/alpine-node
Proposed Base Configuration for faster node.js
https://nodejs.org/en/download/package-manager/
*Node.js package command suggestions:
npm install -g --save npm@latest amd random cacache pacote node-cache requirejs amd jquery crypto-js zlibjs pulverized @types/jqueryui get-google-fonts google-font-installer font-awesome @fortawesome/fontawesome-free bootstrap angular-bootstrap bootstrap-css-only util texture-compressor compress-images cdnjs jsdom canvas express @tensorflow/tfjs @tensorflow/tfjs-backend-wasm @tensorflow/tfjs-backend-webgpu @tensorflow/tfjs-backend-webgl onnxruntime-common onnxruntime-node onnxruntime-web onnxruntime-react-native @cloudflare/wrangler @cloudflare/ai
npm audit fix --force
#AfterWork
npm publish
ipfs-npm publish
npm audit fix --force
#AfterWork
npm publish
ipfs-npm publish
linux node.js:
sudo apt-get update && sudo apt upgrade -y && sudo apt-get install cgroup-tools npm -y && sudo npm install --save npm@latest && sudo npm install --save npm@latest amd random cacache pacote node-cache requirejs amd jquery crypto-js zlibjs pulverized @types/jqueryui get-google-fonts google-font-installer font-awesome @fortawesome/fontawesome-free bootstrap angular-bootstrap bootstrap-css-only util texture-compressor compress-images cdnjs jsdom canvas express @tensorflow/tfjs @tensorflow/tfjs-backend-wasm @tensorflow/tfjs-backend-webgpu @tensorflow/tfjs-backend-webgl onnxruntime-common onnxruntime-node onnxruntime-web onnxruntime-react-native @cloudflare/wrangler @cloudflare/ai && sudo npm audit fix --force && sudo npm publish && sudo ipfs-npm publish
https://hub.docker.com/r/minimum2scp/squid/
docker pull minimum2scp/squid
http://www.squid-cache.org/Intro/
A squid config file : https://is.gd/squid_conf
Needs to be in the folder > /etc/squid
*Nuster http Cache - needs configuration !*
https://github.com/jiangwenyuan/nuster
https://hub.docker.com/r/nuster/nuster/
docker pull nuster/nuster
*railgun*
https://www.cloudflare.com/docs/railgun/installation.html
https://www.cloudflare.com/docs/railgun/daemon_execution.html
https://hub.docker.com/r/rungeict/cloudflare-railgun/
docker pull rungeict/cloudflare-railgun
https://www.cloudflare.com/website-optimization/railgun/?r=1
*bind DNS*
https://hub.docker.com/r/sameersbn/bind/#introduction
docker pull sameersbn/bind
# to configure and update binddns-resolver : https://localhost:10000
****
dhcp servers
freeradius dhcp servers
https://hub.docker.com/r/freeradius/freeradius-server/
docker pull freeradius/freeradius-server
docker run -d --name my-radius -p 1812-1813:1812-1813/udp --restart=always freeradius/freeradius-server
RackHD dhcp server
https://rackhd.readthedocs.io/en/latest/
https://github.com/RackHD/
https://hub.docker.com/r/rackhd/isc-dhcp-server/
docker pull rackhd/isc-dhcp-server
docker run -d --name rackhd-dhcp -p 1812-1813:1812-1813/udp --restart=always rackhd/isc-dhcp-server
Alpine dhcp
https://hub.docker.com/r/joebiellik/dhcpd/
docker pull joebiellik/dhcpd
****
https://hackernoon.com/a-kubernetes-guide-for-docker-swarm-users-c14c8aa266cc
https://blog.alexellis.io/you-need-to-know-kubernetes-and-swarm/
Traefik : Swarm cloud single ip/site url; header wall that keeps all nodes hidden within to handle web traffic from the singular web portal to inside the node array list (funnels Data, Hence > traefik)
https://github.com/bee42/traefik-with-docker
https://github.com/bee42/docker-orchestration-swarming
https://hub.docker.com/r/bee42/docker-orchestration-swarming/
docker pull bee42/docker-orchestration-swarming
****
Open science grid including tensor flow & GPU
https://hub.docker.com/u/opensciencegrid/
swarm elasticsearch
https://hub.docker.com/r/agoryachev/docker-swarm-elasticsearch/
****
Media server
https://hub.docker.com/r/kurento/kurento-media-server/
docker pull kurento/kurento-media-server
docker run -d --name kms-mediaserver -p 8888:8888 kurento/kurento-media-server
Image neural enhancer
https://github.com/alexjc/neural-enhance
https://hub.docker.com/r/alexjc/neural-enhance/
docker pull alexjc/neural-enhance
caffe gpu
https://hub.docker.com/r/tleyden5iwx/caffe-gpu-master/
docker pull tleyden5iwx/caffe-gpu-master
****
Neural networks : SVM : Support vector machine learning
Commands to advantage algebra, differential maths & statistics:
SVM ML Spec : Green world Support vector machine, Machine learning and data min
https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/
https://scikit-learn.org/stable/modules/svm.html
https://towardsdatascience.com/https-medium-com-pupalerushikesh-svm-f4b42800e989
https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47?gi=51274a92cf9b
http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf
****
Boinc HPC Computing
https://github.com/BOINC/boinc-client-docker
https://github.com/marius311/boinc-server-docker/blob/master/docs/cookbook.md
https://cosmicmar.com/2016/02/14/boinc-server-docker-1/
Our suggestion of boinc detection of docker (docker launch takes 15 seconds plus so boinc would have to wait for the process to initiate in the windows task manager before..
The initial detection system would launch for Container VM & also pull user data into the VM (the necessary ID from project add device & also set the VM and OS use for special case use and non generic project data,
Of special note is the use of Linux containers; This container potentially could batch multiple projects with finesse; In particular Science Linux and Alpine (for size and usability) (science kernel version is possible)
docker pull alpine - check this out!
*alpine is a tiny Linux with many apps very flexible 5mb*
https://hub.docker.com/_/alpine/
docker pull alpine
Finally special contained secure environments for HPC Boinc Networking science is realised with relative technical aplume & thusly easily usable by the average; Non average user.
boinc node network on alpine linux with rasberry-pi
https://github.com/firepear/homefarm
****
http://science.n-helix.com/2018/09/hpc.html - a bit about boinc in terms of Docker & Microsoft HPC Pack
http://science.n-helix.com/2017/04/boinc.html - HPC, Optimisation & Boinc - programming
****
Squid for Windows & Linux
http://www.squid-cache.org/
easy install for windows
http://squid.diladele.com/
http://packages.diladele.com/squid/3.5.28/squid.msi
https://techencyclopedia.wordpress.com/2018/04/07/how-to-install-and-configure-squid-proxy-on-windows/
Squid.conf for easy setup of squid settings for cache : https://is.gd/Squid_conf
Needs to be in the folder > /etc/squid
****
A squid config file : https://is.gd/squid_conf
Needs to be in the folder > /etc/squid
*Nuster http Cache - needs configuration !*
https://github.com/jiangwenyuan/nuster
https://hub.docker.com/r/nuster/nuster/
docker pull nuster/nuster
*railgun*
https://www.cloudflare.com/docs/railgun/installation.html
https://www.cloudflare.com/docs/railgun/daemon_execution.html
https://hub.docker.com/r/rungeict/cloudflare-railgun/
docker pull rungeict/cloudflare-railgun
https://www.cloudflare.com/website-optimization/railgun/?r=1
*bind DNS*
https://hub.docker.com/r/sameersbn/bind/#introduction
docker pull sameersbn/bind
# to configure and update binddns-resolver : https://localhost:10000
****
dhcp servers
freeradius dhcp servers
https://hub.docker.com/r/freeradius/freeradius-server/
docker pull freeradius/freeradius-server
docker run -d --name my-radius -p 1812-1813:1812-1813/udp --restart=always freeradius/freeradius-server
RackHD dhcp server
https://rackhd.readthedocs.io/en/latest/
https://github.com/RackHD/
https://hub.docker.com/r/rackhd/isc-dhcp-server/
docker pull rackhd/isc-dhcp-server
docker run -d --name rackhd-dhcp -p 1812-1813:1812-1813/udp --restart=always rackhd/isc-dhcp-server
Alpine dhcp
https://hub.docker.com/r/joebiellik/dhcpd/
docker pull joebiellik/dhcpd
****
https://hackernoon.com/a-kubernetes-guide-for-docker-swarm-users-c14c8aa266cc
https://blog.alexellis.io/you-need-to-know-kubernetes-and-swarm/
Further reading for swarm node configuration
https://github.com/docker/labs/blob/master/swarm-mode/beginner-tutorial/README.mdTraefik : Swarm cloud single ip/site url; header wall that keeps all nodes hidden within to handle web traffic from the singular web portal to inside the node array list (funnels Data, Hence > traefik)
https://github.com/bee42/traefik-with-docker
https://github.com/bee42/docker-orchestration-swarming
https://hub.docker.com/r/bee42/docker-orchestration-swarming/
docker pull bee42/docker-orchestration-swarming
****
Open science grid including tensor flow & GPU
https://hub.docker.com/u/opensciencegrid/
swarm elasticsearch
https://hub.docker.com/r/agoryachev/docker-swarm-elasticsearch/
****
Media server
https://hub.docker.com/r/kurento/kurento-media-server/
docker pull kurento/kurento-media-server
docker run -d --name kms-mediaserver -p 8888:8888 kurento/kurento-media-server
Image neural enhancer
https://github.com/alexjc/neural-enhance
https://hub.docker.com/r/alexjc/neural-enhance/
docker pull alexjc/neural-enhance
caffe gpu
https://hub.docker.com/r/tleyden5iwx/caffe-gpu-master/
docker pull tleyden5iwx/caffe-gpu-master
****
Neural networks : SVM : Support vector machine learning
Commands to advantage algebra, differential maths & statistics:
SVM ML Spec : Green world Support vector machine, Machine learning and data min
https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/
https://scikit-learn.org/stable/modules/svm.html
https://towardsdatascience.com/https-medium-com-pupalerushikesh-svm-f4b42800e989
https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47?gi=51274a92cf9b
http://web.mit.edu/6.034/wwwbob/svm-notes-long-08.pdf
****
Boinc HPC Computing
https://github.com/BOINC/boinc-client-docker
https://github.com/marius311/boinc-server-docker/blob/master/docs/cookbook.md
https://cosmicmar.com/2016/02/14/boinc-server-docker-1/
Our suggestion of boinc detection of docker (docker launch takes 15 seconds plus so boinc would have to wait for the process to initiate in the windows task manager before..
The initial detection system would launch for Container VM & also pull user data into the VM (the necessary ID from project add device & also set the VM and OS use for special case use and non generic project data,
Of special note is the use of Linux containers; This container potentially could batch multiple projects with finesse; In particular Science Linux and Alpine (for size and usability) (science kernel version is possible)
docker pull alpine - check this out!
*alpine is a tiny Linux with many apps very flexible 5mb*
https://hub.docker.com/_/alpine/
docker pull alpine
Finally special contained secure environments for HPC Boinc Networking science is realised with relative technical aplume & thusly easily usable by the average; Non average user.
boinc node network on alpine linux with rasberry-pi
https://github.com/firepear/homefarm
****
http://science.n-helix.com/2018/09/hpc.html - a bit about boinc in terms of Docker & Microsoft HPC Pack
http://science.n-helix.com/2017/04/boinc.html - HPC, Optimisation & Boinc - programming
****
Squid for Windows & Linux
http://www.squid-cache.org/
easy install for windows
http://squid.diladele.com/
http://packages.diladele.com/squid/3.5.28/squid.msi
https://techencyclopedia.wordpress.com/2018/04/07/how-to-install-and-configure-squid-proxy-on-windows/
Squid.conf for easy setup of squid settings for cache : https://is.gd/Squid_conf
Needs to be in the folder > /etc/squid
****
Some ambitious specs for a VM
8 cores, 8Gb ram, 1 GPU (2gb ram), 20GB HDD per process ^^
That being said : Hyper V VM allows Dynamic ram as an option & this means the ram can be used by the system if the VM is not using it..
Docker has a similar method in that the Docker VM uses a pre-defined space :
In my case 16gb and 8 cores, This is sub allocated along with linux swap ..
To processes in containers, Containers can use most varieties of linux including alpine (very small memory footprint) Alpine is suitable for science projects as is the Docker Kernel.
(c)RS : Help the community discover cures & improve the community: https://is.gd/ProcessorLasso
