Categories
InfoSec Notes ITOps Random

Eramba Community 2019 in Docker (docker-compose)

Eramba is an excellent open source Governance Risk and Compliance tool. Recently (10-SEP-2019), a new major release of the community version came out. Previously I used https://github.com/digitorus/eramba which was based on https://hub.docker.com/r/k0st/alpine-eramba/ to start eramba instances quickly with docker and docker-compose.

As I could not find an updated version of these for the new release I have made one. The repo for this, 2019 community version (specifically c2.4.1) can be found here: https://github.com/markz0r/eramba-community-docker

Follow the steps in README.md and you should be testing the new eramba in no time.

Mar, 2020: Updated for community edition 2.8.1

Thanks to the team at Eramba for making the tool available for all!

Categories
Random

Performance Benchmarks on CentOS 7 Linux

In a scenario where a VM is moved to different underlying hardware, it is generally a good idea to validate CPU, memory, disk IO and network.

CPU Benchmark

sysbench cpu --cpu-max-prime=20000 run

sysbench threads --num-threads=10 --thread-yields=0 --max-requests=100000000 --thread-locks=1 run

Memory Benchmark

sysbench memory --memory-block-size=1M --memory-total-size=100G run
sysbench memory --memory-total-size=10G run

File IO

sysbench fileio --file-total-size=5G prepare; sysbench fileio --file-total-size=5G --file-test-mode=rndrw --time=300 --max-requests=0 run
# Clean up
sysbench fileio --file-total-size=5G cleanup

Network latency, upload and download

wget https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py; ./speedtest.py; rm -f ./speedtest.py
Categories
Random

Office 365 Send As an Alias

If you want to have a single mailbox on Office 365 and be able to send as aliases of that mailbox, you will need to do some work around as it is not really support by Microsoft, see:

1 – Create Distribution List

  1. Create distribution group for the desired email address (ensuring is does not exist as an alias or otherwise in the tenant)
  2. Add desired destination mailbox as a member
  3. Open the Exchange Admin center
  4. Select “recipients” (side navbar) -> Select “groups” (top) -> Select the distribution group you just created, click the pencil icon to edit
  5. Select “group delegation” add your main mailbox user to the ‘Send As’ list
  6. Wait for approx 30 mins for Office 365 to provision the distribution list and update contact lists
  7. Optionally set up message rules in your mailbox to ensure emails to the distribution list email address are put into a specific folder

2 – Send As the distribution list via Outlook (Windows)

  1. In your Outlook client, create a new message
  2. If you cant see the From box, click ‘Options’, Click ‘From’
  3. Click on the now display ‘From’ dropbox and select ‘Other email address’
  4. Click on the ‘From…’ in the popup box
  5. Click on the ‘Offline Global Address List’, select ‘All Distribution Lists’, select your desired From address.

3 – Exchange Online

  1. Create new message
  2. Click the ellipsis to the right of the send button
  3. Right click on the from address, click remove
  4. Start typing the address you want to send from, select it from the drop down autocompleter
Categories
Random

3D CAD Fundamental – Week 3

Building a toy house module

Looks primarily at changing object shapes, introducing the move too and the 2-point arch tool. Using double click for repetition of push/pull tool also proved to be convenient. We then used the move tool to alter slopes of surfaces, including using the up key to match slope and then height of another surface.

Next up is the arc tool, which has 4 variants:

  • Arc – Main point of this method determines where the center point of the arc will be
  • 2 Point Arc – select two points that will be the width of the arc
  • 3 Point Arc – Firts 2 points determine form, and the third point gives that exact length Ideal for irregularly shaped objects
  • Pie

The week 3 assignment was creating a house to match a floor, wall and roof plan. Unfortunately it appears that the assignment specification had a couple of slight errors. This was a bit of a time waste and student from the previous course had reported it so it is a bit disappointing that the course writers have not noticed/corrected it: https://www.coursera.org/learn/3d-cad-fundamental/discussions/weeks/3/threads/QTxAZ5UGEeir3xJNYGdMZA

Again the first pass took a while and was quite difficult, but a complete redraw took only 5 mins. When drawing structures like this, with eves and and sloped roofs it is important to complete a room (minus the eves and roof thickness) to make slope matching easier.

week 3 simple house
Categories
GoLang Web Application Random

Free Golang IDE (s) on macos (Visual Studio Code / vim)

Visual Studio Code

Visual Studio Code is a now is Microsoft’s now OpenSource IDE that runs on windows, macos and linux!

Simple set up guide here: https://rominirani.com/setup-go-development-environment-with-visual-studio-code-7ea5d643a51a. Assuming go is installed and ready to do – the download, install and setup took about 5 minutes. Everything just works out of the box and its much less dependency on complex config files and plugins (vs vim).


Vim (abandoned this for Microsoft Visual Code)

Install these if they are not already:

brew install vim
# Note that is is executing arbitrary code from an the vim-go repo 
curl -fLo ~/.vim/autoload/plug.vim --create-dirs https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim 
git clone https://github.com/fatih/vim-go.git ~/.vim/plugged/vim-go
  • Customise ~/.vimrc to enable and configure your plugins and shortcut keys
  • Once th ~/.vimrc is added run :GoInstallBinaries to get vim-go’s dependencies

Shortcut keys in this vimrc:

  • \ + b -> build
    • if errors occur toggle forward and back through them with ctrl + n and ctrl + m
    • close quick fix dialogue boxes with \ + a
  • \ + i -> install
  • dif (whilst on func def, delete all contents for func)

Autocompletion sucks though 🙁 so adding neocomplete is a must).

With existing versions of brew installed vim and the introduced dependency of xcode makes the setup time high. I went through this in the past and after a fairly long hiatus from writing code if find nothing is working quite right.

 

Categories
Random

Download all Evernote attachments via Evernote API with Python

Python script for downloading snapshots of all attachments in all of your Evernote notebooks.

#!/usr/bin/python
import json, os, pickle, httplib2, io
import evernote.edam.userstore.constants as UserStoreConstants
import evernote.edam.type.ttypes as Types
from evernote.api.client import EvernoteClient
from evernote.edam.notestore.ttypes import NoteFilter, NotesMetadataResultSpec
from datetime import date

# Pre-reqs: pip install evernote 
# API key from https://dev.evernote.com/#apikey

os.environ["PYTHONPATH"] = "/Library/Python/2.7/site-packages"

CREDENTIALS_FILE=".evernote_creds.json"
LOCAL_TOKEN=".evernote_token.pkl"
OUTPUT_DIR=str(date.today())+"_evernote_backup"

def prepDest():
    if not os.path.exists(OUTPUT_DIR):
        os.makedirs(OUTPUT_DIR)
        return True
    return True

# Helper function to turn query string parameters into a 
# source: https://gist.github.com/inkedmn
def parse_query_string(authorize_url):
    uargs = authorize_url.split('?')
    vals = {}
    if len(uargs) == 1:
        raise Exception('Invalid Authorization URL')
    for pair in uargs[1].split('&'):
        key, value = pair.split('=', 1)
        vals[key] = value
    return vals

class AuthToken(object):
    def __init__(self, token_list):
        self.oauth_token_list = token_list

def authenticate():
    def storeToken(auth_token):
        with open(LOCAL_TOKEN, 'wb') as output:
            pickle.dump(auth_token, output, pickle.HIGHEST_PROTOCOL)    

    def oauthFlow():
        with open(CREDENTIALS_FILE) as data_file:    
            data = json.load(data_file)
            client = EvernoteClient(
                consumer_key = data.get('consumer_key'),
                consumer_secret = data.get('consumer_secret'),
                sandbox=False
            )
        request_token = client.get_request_token('https://assetowl.com')
        print(request_token)
        print("Token expired, load in browser: " + client.get_authorize_url(request_token))
        print "Paste the URL after login here:"
        authurl = raw_input()
        vals = parse_query_string(authurl)
        auth_token=client.get_access_token(request_token['oauth_token'],request_token['oauth_token_secret'],vals['oauth_verifier'])
        storeToken(AuthToken(auth_token))
        return auth_token

    def storeToken(auth_token):
        with open(LOCAL_TOKEN, 'wb') as output:
            pickle.dump(auth_token, output, pickle.HIGHEST_PROTOCOL)    

    def getToken():
        store_token=""
        if os.path.isfile(LOCAL_TOKEN):
            with open(LOCAL_TOKEN, 'rb') as input:
              clientt = pickle.load(input)
            store_token=clientt.oauth_token_list
        return store_token

    try:
        client = EvernoteClient(token=getToken(),sandbox=False)
        userStore = client.get_user_store()
        user = userStore.getUser()
    except Exception as e:
        print(e)
        client = EvernoteClient(token=oauthFlow(),sandbox=False)
    return client

def listNotes(client):
    note_list=[]
    note_store = client.get_note_store()
    filter = NoteFilter()    
    filter.ascending = False
    spec = NotesMetadataResultSpec(includeTitle=True)
    spec.includeTitle = True
    notes = note_store.findNotesMetadata(client.token, filter, 0, 100, spec)
    for note in notes.notes:
        for resource in note_store.getNote(client.token, note.guid, False, False, True, False).resources:
            note_list.append([resource.attributes.fileName, resource.guid])
    return note_list


def downloadResources(web_prefix, res_array):
    for res in res_array:
        res_url = "%sres/%s" % (web_prefix, res[1])
        print("Downloading: " + res_url + " to " + OUTPUT_DIR + res[0])
        h = httplib2.Http(".cache")
        (resp_headers, content) = h.request(res_url, "POST",
                                        headers={'auth': DEV_TOKEN})
        with open(os.path.join(OUTPUT_DIR, res[0]), "wb") as wer:
            wer.write(content)

def main():
    if prepDest():
        client = authenticate()
        user_store=client.get_user_store()
        web_prefix = user_store.getPublicUserInfo(user_store.getUser().username).webApiUrlPrefix
        downloadResources(web_prefix, listNotes(client))

if __name__ == '__main__':
    main()

Categories
Random

Downloading Google Drive with Python via Drive API

Python script for downloading snapshots of all file in your google drive, including those shared with you.

source: https://github.com/SecurityShift/tools/blob/master/backup_scripts/google_drive_backup.py


Categories
Random

downloading a youtube playlist and converting to mp3 (osx/linux(ubuntu))

Downloading youtube audio and converting mp3 is very simple now thanks to these guys: https://rg3.github.io/youtube-dl/

Installing:

# on Ubuntu
sudo apt-get install youtube-dl avconv
curl https://raw.githubusercontent.com/SecurityShift/tools/master/scripts/random/GetYoutubePlaylist.sh -o GetYoutubePlaylist.sh
chmod 755 GetYoutubeAudio.sh

https://raw.githubusercontent.com/SecurityShift/tools/master/backup_scripts/google_drive_backup.py

# on OSX
brew install youtube-dl libav
curl https://raw.githubusercontent.com/SecurityShift/tools/master/scripts/random/GetYoutubePlaylist.sh -o GetYoutubePlaylist.sh
chmod 755 GetYoutubeAudio.s

Contents:


Usage: sh GetPlaylist.sh [youtube playlist URL] [output_directory]

Example:

sh ./GetYoutubePlaylist.sh http://www.youtube.com/playlist?list=PL702CAF4AD2AED35B ~/Music

To get the youtube playlist URL view the playlist by clicking on its name, no videos will be playing, then click on share and the url will be highlighted:

youtube-playlist
click on share and the URL will appear

Categories
Functional Programming - Scala Random

Virtual Enigma Machine Implementation

After watching a video on numberphile thought it would be interesting to implement a virtual version on the Enigma cipher machine.

Step 1: Install Scala and Eclipse IDE for Scala.

http://www.scala-lang.org/downloadshttp://scala-ide.org/

Step 2: Review the numberphile video and how the Enigma machine works

Step 3: Implement each function of the machine

  1. Rotors
    • Rotor choices, rotors have different circuitry (0-24) – static value for each message
    • Initial rotor combination (0-25) – static value for each message
    • Rotor increment – Faster rotor, middle rotor, slow rotor
  2. Plug board
    • 10 wires
    • swapping 2 letters
    • static value for each message

Step 4: Write an interface between the functions and a basic interface (text?)

Step 5: Test!

Conclusion: Using Scala (trying anyway) the implementation is not very complex. What is more difficult is replicating the original machines where the output character could not equal the input character. The limitation Turing used to crack the machines. Might see if I can implement that later then test out the cracking method. Not sure how easy it would be to break the current implementation…

CODE (EnigmaMachine.scala, EnigmaMachine.class):

 

 

// Virtual reconstruction of what the enigma machines did
// The replication is of the improved version where the input could result in the same output

object EnigmaMachine {
//////////////// ENCRYPTION/DECRYPTION FUNCTIONS \\\\\\\\\\\\\\\\\\\\\
    //to replicate the simple machine circuit chars are converter to 0 - 25
    def convertMessageToSimpleInt(rawInput: String):List[Int] = {
        val userMessage = rawInput.toUpperCase
      (for (x <- userMessage) yield if (x != 32) (x.toInt - 65) else 26).toList
    }
    
    //for all integers in the list, convert
    def convertMessageToString(input: List[Int]): String = {
      (for (x <- input) yield if (x == 26) " " else (x + 65).toChar).mkString
    } 
    
    def enigmaGo(rotorChoices: List[Int], rotorPositions: List[Int], inputMessage: List[Int], encrypt: Boolean): List[Int] = {    
      // incrementing the rotors is dont based on the number of previous chars.. instead of the previous position
      def incrementRots(nth: Int): List[Int] = {
        val rot0 = (rotorPositions(0) + nth) % 26 //fast rotor
        val rot1 = (rotorPositions(1) + (nth / 26)) % 26 //medium rotor
        val rot2 = (rotorPositions(2) + ((nth / 26) /26)) % 26 //slow rotor
        List(rot0, rot1, rot2)
      }
      //applies the rotor encryption then the scramble board
      def transmuteElement(element: Int, rotPosApp: List[Int]): Int = {
        val rotorTotal = rotorList(rotorChoices(0))(rotPosApp(0)) +
              rotorList(rotorChoices(1))(rotPosApp(1)) +
              rotorList(rotorChoices(2))(rotPosApp(2)) 
        
      def scramble(inputVal:Int):Int = plugMap1.find(x => (x._1 == inputVal) || (x._2 == inputVal)) match {
            case Some(y) => if (y._1 == inputVal) y._2 else y._1 
          case None => inputVal
      }
        if (element == 26) element
        else if (encrypt) (scramble(element) + rotorTotal) % 26 else { 
            //decrypt is a bit messy atm :( 
            if (element - (rotorTotal % 26) < 0) scramble(element - (rotorTotal % 26) + 26) else scramble(element - (rotorTotal % 26))
          }     
      }
      
      //Accumulator hold the input and output lists, when input list is empty, return output.. also tail recursive
      def enigmaAccu(remInput: List[Int], rotPos: List[Int], accumOutput: List[Int]): List[Int] = {
              if (remInput.isEmpty) accumOutput
            else enigmaAccu(remInput.tail, incrementRots(accumOutput.length), accumOutput:+(transmuteElement(remInput.head, rotPos)))
      }
      
      enigmaAccu(inputMessage, rotorPositions, Nil) //start the process
    }
////////////////////   \\\\\\\\\\\\\\\\\\\\\\\\\
    
//////////////// UI STUFF \\\\\\\\\\\\\\\\\\\\\
    def useMachine(rotorChoices: List[Int], rotorStartVals: List[Int], activity: String) = {
            spamLine
            val inputMessage = convertMessageToSimpleInt(askInput("Please enter a message to be " + activity + ", use only a-z: "))
            val resultingMessage = enigmaGo(rotorChoices, rotorStartVals, inputMessage, if (activity == "encrypted") true else false)
            spamLine
            println("Your message has been " + activity + ": ")
        println(convertMessageToString(resultingMessage))
            spamLine
            exit(0)
    }

    def getRotorChoices(s1:String, p: String => Boolean):List[Int] = {
        val slot0 = getUserInput(s1 + "fast slot: ", "Entry Invalid.", p).mkString
        val slot1 = getUserInput(s1 + "medium slot: ", "Entry Invalid.", p).mkString
        val slot2 = getUserInput(s1 + "slow slot: ", "Entry Invalid.", p).mkString
        List(slot0.toInt,slot1.toInt,slot2.toInt)
    }
    
    def spamLine = println("##############################################")
    
    def askInput(statement: String) = { 
      val input = readLine(statement)
      if (input == "") "99" else input
    }
    
    def askAgain(statement: String, f: => String) = {
        println(statement)
        f
    } 
    
    def inputVals(f1: => String, f2: => String) = Stream.cons( f1, Stream.continually(f2))
    
    def getUserInput(s1: String, s2: String, p:String => Boolean) = inputVals(askInput(s1), askAgain(s2, askInput(s1))).find(p)
////////////////////   \\\\\\\\\\\\\\\\\\\\\\\\\
    
//////////////// MAIN \\\\\\\\\\\\\\\\\\\\\ 
        def main(args: Array[String]) {
            println("### WELCOME TO THE ENIGMA MACHINE! ###" )
            // GET INITIAL MACHINE SETTINGS FROM USER 
            // ROTOR PLACEMENT CONFIG
            val rotorChoices = getRotorChoices("Choose rotor (0 - 4) for ", x => (x.toInt >= 0 && x.toInt < 5)) 
            // ROTOR START VALS 
            val rotorStartVals =  getRotorChoices("Choose rotor START VAL (0 - 25) for ", x => (x.toInt >= 0 && x.toInt < 25))
            // PLUG MAP
            println("There is only 1 plug map option at present:") 
            println(plugMap1.toString.drop(4))
            //ENCRYPT OF DECRYPT - ONLY RELEVENT FOR GUI TEXT.. bleh not quite need to reverse circuit manually
            val userChoice = getUserInput("Press 1 to encrypt, 2 to decrypt: ", "Entry Invalid.", x => (x == "2" || x == "1"))
            userChoice.mkString match {
              case "1" => useMachine(rotorChoices, rotorStartVals, "encrypted") 
              case "2" => useMachine(rotorChoices, rotorStartVals, "decrypted")
              case default => exit(0)
            }
    }
////////////////////   \\\\\\\\\\\\\\\\\\\\\\\\\

//////////////// CONSTANTS \\\\\\\\\\\\\\\\\\\\\
    val rotorList = List(List(6, 17, 4, 3, 23, 22, 21, 5, 0, 19, 20, 11, 16, 9, 13, 24, 18, 1, 8, 2, 7, 10, 12, 14, 25, 15)
    ,List(2, 7, 23, 12, 13, 15, 18, 1, 8, 11, 0, 3, 10, 5, 24, 4, 17, 19, 6, 9, 20, 14, 25, 21, 16, 22)
    ,List(23, 1, 19, 18, 0, 10, 3, 8, 9, 21, 2, 12, 13, 14, 24, 6, 20, 16, 11, 17, 15, 7, 4, 25, 22, 5)
    ,List(21, 22, 16, 2, 18, 20, 15, 9, 5, 1, 17, 10, 3, 6, 0, 13, 11, 23, 4, 12, 19, 8, 7, 25, 24, 14)
    ,List(25, 3, 11, 7, 5, 15, 24, 10, 21, 6, 12, 17, 8, 23, 2, 18, 19, 14, 4, 0, 20, 9, 22, 1, 16, 13))
    
    val plugMap1 = List((3,7),(11,16),(18,24),(17,9),(21,19),(14,13),(0,12),(1,25),(22,20),(4,6))
////////////////////   \\\\\\\\\\\\\\\\\\\\\\\\\
}
Categories
Random

MapReduce Programming Model and Hadoop

After hearing the praises of Hadoop I had a brief look it and the Map/Reduce paradigm. Most info was read from Wikipedia and references in wiki articles. In particular, this paper from Jeffrey Dean and Sanjay Ghemawat.

Hadoop is an opensource software framework that implements MapReduce and Google File System. The aim is to enable easy and robust deployment of highly parallel data set processing programs. It is important to note that the MapReduce model is applicable to embarrassingly parallel problems. Processing can occur on data that is stored in a database or filesystem.

Map/Reduce refers to the steps in the model:

Map: A master node takes an input problem and divides it into sub-problems, passing them to worker nodes. Worker nodes can further divide sub-problems.  For example, in the problem of counting word occurrence in a document the Map function will output a key/value pair every time it sees a a specified work – ie: (“searchterm”, 1).

Reduce: The reduce function takes the list of word(key)/values and sums the occurrences:

(foo, (1)

(bar, 1)

(foo, 1)

(bar, 1)

(foo, 1)

The output of reduce in this case could be: (foo, 3).

The MapReduce model becomes powerful when considering giant datasets can be processed in large clusters very efficiently using this model.

Hadoop Run-time Takes care of:

  • Parallelalization
  • Partitioning of input data
  • Scheduling programs execution across machines
  • Handling machine failures
  • Managing inter-machine communication

Hadoop aims to enable developers with little distributed programming experience to utilize compute resources such as EC2.

With the emergence of ‘big data’ and the apparent value that can be extrapolated from massive databases/datastores, many organisations have found the limits of traditional relational databases. Hadoop has such a big buzz because it can pass the processing boundaries of relational database software and enable the extrapolation of value. The video below is a decent explanation of this point by data scientists at SalesForce.com