Performance Benchmarks on CentOS 7 Linux

In a scenario where a VM is moved to different underlying hardware, it is generally a good idea to validate CPU, memory, disk IO and network.

CPU Benchmark

sysbench cpu --cpu-max-prime=20000 run

sysbench threads --num-threads=10 --thread-yields=0 --max-requests=100000000 --thread-locks=1 run

Memory Benchmark

sysbench memory --memory-block-size=1M --memory-total-size=100G run
sysbench memory --memory-total-size=10G run

File IO

sysbench fileio --file-total-size=5G prepare; sysbench fileio --file-total-size=5G --file-test-mode=rndrw --time=300 --max-requests=0 run
# Clean up
sysbench fileio --file-total-size=5G cleanup

Network latency, upload and download

wget https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py; ./speedtest.py; rm -f ./speedtest.py

Office 365 Send As an Alias

If you want to have a single mailbox on Office 365 and be able to send as aliases of that mailbox, you will need to do some work around as it is not really support by Microsoft, see:

1 – Create Distribution List

  1. Create distribution group for the desired email address (ensuring is does not exist as an alias or otherwise in the tenant)
  2. Add desired destination mailbox as a member
  3. Open the Exchange Admin center
  4. Select “recipients” (side navbar) -> Select “groups” (top) -> Select the distribution group you just created, click the pencil icon to edit
  5. Select “group delegation” add your main mailbox user to the ‘Send As’ list
  6. Wait for approx 30 mins for Office 365 to provision the distribution list and update contact lists
  7. Optionally set up message rules in your mailbox to ensure emails to the distribution list email address are put into a specific folder

2 – Send As the distribution list via Outlook (Windows)

  1. In your Outlook client, create a new message
  2. If you cant see the From box, click ‘Options’, Click ‘From’
  3. Click on the now display ‘From’ dropbox and select ‘Other email address’
  4. Click on the ‘From…’ in the popup box
  5. Click on the ‘Offline Global Address List’, select ‘All Distribution Lists’, select your desired From address.

3 – Exchange Online

  1. Create new message
  2. Click the ellipsis to the right of the send button
  3. Right click on the from address, click remove
  4. Start typing the address you want to send from, select it from the drop down autocompleter

3D CAD Fundamental – Week 3

Building a toy house module

Looks primarily at changing object shapes, introducing the move too and the 2-point arch tool. Using double click for repetition of push/pull tool also proved to be convenient. We then used the move tool to alter slopes of surfaces, including using the up key to match slope and then height of another surface.

Next up is the arc tool, which has 4 variants:

  • Arc – Main point of this method determines where the center point of the arc will be
  • 2 Point Arc – select two points that will be the width of the arc
  • 3 Point Arc – Firts 2 points determine form, and the third point gives that exact length Ideal for irregularly shaped objects
  • Pie

The week 3 assignment was creating a house to match a floor, wall and roof plan. Unfortunately it appears that the assignment specification had a couple of slight errors. This was a bit of a time waste and student from the previous course had reported it so it is a bit disappointing that the course writers have not noticed/corrected it: https://www.coursera.org/learn/3d-cad-fundamental/discussions/weeks/3/threads/QTxAZ5UGEeir3xJNYGdMZA

Again the first pass took a while and was quite difficult, but a complete redraw took only 5 mins. When drawing structures like this, with eves and and sloped roofs it is important to complete a room (minus the eves and roof thickness) to make slope matching easier.

week 3 simple house

Free Golang IDE (s) on macos (Visual Studio Code / vim)

Visual Studio Code

Visual Studio Code is a now is Microsoft’s now OpenSource IDE that runs on windows, macos and linux!

Simple set up guide here: https://rominirani.com/setup-go-development-environment-with-visual-studio-code-7ea5d643a51a. Assuming go is installed and ready to do – the download, install and setup took about 5 minutes. Everything just works out of the box and its much less dependency on complex config files and plugins (vs vim).


Vim (abandoned this for Microsoft Visual Code)

Install these if they are not already:

  • Customise ~/.vimrc to enable and configure your plugins and shortcut keys
  • Once th ~/.vimrc is added run :GoInstallBinaries to get vim-go’s dependencies

Shortcut keys in this vimrc:

  • \ + b -> build
    • if errors occur toggle forward and back through them with ctrl + n and ctrl + m
    • close quick fix dialogue boxes with \ + a
  • \ + i -> install
  • dif (whilst on func def, delete all contents for func)

Autocompletion sucks though 🙁 so adding neocomplete is a must).

With existing versions of brew installed vim and the introduced dependency of xcode makes the setup time high. I went through this in the past and after a fairly long hiatus from writing code if find nothing is working quite right.

 

Download all Evernote attachments via Evernote API with Python

Python script for downloading snapshots of all attachments in all of your Evernote notebooks.

Downloading Google Drive with Python via Drive API

Python script for downloading snapshots of all file in your google drive, including those shared with you.

source: https://github.com/SecurityShift/tools/blob/master/backup_scripts/google_drive_backup.py

downloading a youtube playlist and converting to mp3 (osx/linux(ubuntu))

Downloading youtube audio and converting mp3 is very simple now thanks to these guys: https://rg3.github.io/youtube-dl/

Installing:

Contents:

Usage: sh GetPlaylist.sh [youtube playlist URL] [output_directory]

Example:

To get the youtube playlist URL view the playlist by clicking on its name, no videos will be playing, then click on share and the url will be highlighted:

youtube-playlist
click on share and the URL will appear

Virtual Enigma Machine Implementation

After watching a video on numberphile thought it would be interesting to implement a virtual version on the Enigma cipher machine.

Step 1: Install Scala and Eclipse IDE for Scala.

http://www.scala-lang.org/downloadshttp://scala-ide.org/

Step 2: Review the numberphile video and how the Enigma machine works

Step 3: Implement each function of the machine

  1. Rotors
    • Rotor choices, rotors have different circuitry (0-24) – static value for each message
    • Initial rotor combination (0-25) – static value for each message
    • Rotor increment – Faster rotor, middle rotor, slow rotor
  2. Plug board
    • 10 wires
    • swapping 2 letters
    • static value for each message

Step 4: Write an interface between the functions and a basic interface (text?)

Step 5: Test!

Conclusion: Using Scala (trying anyway) the implementation is not very complex. What is more difficult is replicating the original machines where the output character could not equal the input character. The limitation Turing used to crack the machines. Might see if I can implement that later then test out the cracking method. Not sure how easy it would be to break the current implementation…

CODE (EnigmaMachine.scala, EnigmaMachine.class):

 

 

MapReduce Programming Model and Hadoop

After hearing the praises of Hadoop I had a brief look it and the Map/Reduce paradigm. Most info was read from Wikipedia and references in wiki articles. In particular, this paper from Jeffrey Dean and Sanjay Ghemawat.

Hadoop is an opensource software framework that implements MapReduce and Google File System. The aim is to enable easy and robust deployment of highly parallel data set processing programs. It is important to note that the MapReduce model is applicable to embarrassingly parallel problems. Processing can occur on data that is stored in a database or filesystem.

Map/Reduce refers to the steps in the model:

Map: A master node takes an input problem and divides it into sub-problems, passing them to worker nodes. Worker nodes can further divide sub-problems.  For example, in the problem of counting word occurrence in a document the Map function will output a key/value pair every time it sees a a specified work – ie: (“searchterm”, 1).

Reduce: The reduce function takes the list of word(key)/values and sums the occurrences:

(foo, (1)

(bar, 1)

(foo, 1)

(bar, 1)

(foo, 1)

The output of reduce in this case could be: (foo, 3).

The MapReduce model becomes powerful when considering giant datasets can be processed in large clusters very efficiently using this model.

Hadoop Run-time Takes care of:

  • Parallelalization
  • Partitioning of input data
  • Scheduling programs execution across machines
  • Handling machine failures
  • Managing inter-machine communication

Hadoop aims to enable developers with little distributed programming experience to utilize compute resources such as EC2.

With the emergence of ‘big data’ and the apparent value that can be extrapolated from massive databases/datastores, many organisations have found the limits of traditional relational databases. Hadoop has such a big buzz because it can pass the processing boundaries of relational database software and enable the extrapolation of value. The video below is a decent explanation of this point by data scientists at SalesForce.com


Parsing HTML with Python

Came across an interesting scenario this week where I decided to try and use Python for parsing HTML. Turns out to quite straight forward and something I could imagine using more often in the future.

We were trying to grab the a dynamic link from this page at twitch http://www.twitch.tv/directory/StarCraft%20II:%20Wings%20of%20Liberty.

The link we were after was the most viewed channel at any particular time:

The most viewed channel is always in this location/element

First attempt was to load up that page in an <iframe> allowing our own JavaScript to grab the relevant URL then forward. Twitch.tv however forces forwarding when iframes are detected, presumably for proprietary reasons. So javascript was not really an option.

After experimenting with Python’s lxml http://lxml.de/ it was really straight forward and effective. It is apparently quite efficient using C core (http://stackoverflow.com/a/6494811/692180) too. The module I used is documented quite well here: http://lxml.de/lxmlhtml.html. With a little bit of trial an error a brief python script successfully grabs the relevant information.

Then using PHP’s popen() method I can simply call the python script and use the return value as a php variable for a header redirect.

Links to source:

get_link.py – Python script

get_link.php – PHP calling python and using return value :

 

 

Python source: