Transitioning from standard CA to LetEncrypt!

With the go-live of https://letsencrypt.org/ its time to transition from the pricy and manual standard SSL cert issuing model to a fully automated process using the ACME protocol. Most orgs have numerous usages of CA purchased certs, this post will cover hosts running apache/nginx and AWS ELBs, all of these usages are to be replaced with automated provisioning and renewal of letsencrypt signed certs.

Provisioning and auto-renewing Apache and nginx TLS/SSL certs

For externally accessible sites where Apache/Nginx handles TLS/SSL termination moving to letsencrypt is quick and simple:

1 – Install the letsencrypt client software (there are RHEL and Centos rpms – so thats as simple as adding the package to puppet policies or

2 – Provision the keys and certificates for each of the required virtual hosts. If a virtual host has aliases, specify multiple names with the -d arg.

This will provision a key and certificate + chain to the letsencrypt home directory (defaults /etc/letsencrypt). The /etc/letsencrypt/live directory contains symlinks to the current keys and certs.

3 – Update the apache/nginx virtualhost configs to use the symlinks maintained by the letsencrypt client, ie:

4 – Create a script for renewing these certs, something like:

5 – Run this script automatically everyday with cron or jenkins

6 – Monitoring the results of the script and externally monitor the expiry dates of your certificates (something will go wrong one day)

Provisioning and auto-renewing AWS Elastice Load Balancer TLS/SSL certs

This has been made very easy by Alex Gaynor with a handy python script: https://github.com/alex/letsencrypt-aws. This is a great use-case for docker and Alex has created a docker image for the script: https://hub.docker.com/r/alexgaynor/letsencrypt-aws/. To use this with ease I created a layer on top creating a new Dockerfile:

The explanation of these values can be found at https://hub.docker.com/r/alexgaynor/letsencrypt-aws/. Its quite important to create a specific IAM User to conduct the required Route53/S3 and ELB actions. This images need to be build on changes:

With this image built another cron or jenkins job can be run daily executing something like:

Again, the job must be monitored along with external monitoring of certificates. See a complete SSL checker at https://github.com/markz0r/tools/tree/master/ssl_check_complete.

Configuring Snort Rules

Some reading before starting:

Before setting out, getting some basic concepts about snort is important.

This deployment with be in Network Intrusion Detection System (NIDS) mode – which performs detection and analysis on traffic. See other options and nice and concise introduction:  http://manual.snort.org/node3.html.

Rule application order: activation->dynamic->pass->drop->sdrop->reject->alert->log

Again drawing from the snort manual some basic understanding of snort alerts can be found:

116 –  Generator ID, tells us what component of snort generated the alert

Eliminating false positives

After running pulled pork and using the default snort.conf there will likely be a lot of false positives. Most of these will come from the preprocessor rules. To eliminate false positives there are a few options, to retain maintainability of the rulesets and the ability to use pulled pork, do not edit rule files directly. I use the following steps:

  1. Create an alternate startup configuration for snort and barnyard2 without -D (daemon) and barnyard2 config that only writes to stdout, not the database. – Now we can stop and start snort and barnyard2 quickly to test our rule changes.
  2. Open up the relevant documentation, especially for preprocessor tuning – see the ‘doc’ directory in the snort source.
  3. Have some scripts/traffic replays ready with traffic/attacks you need to be alerting on
  4. Iterate through reading the doc, making changes to snort.conf(for preprocessor config), adding exceptions/suppressions to snort’s threshold.conf or PulledPork’s disablesid, dropsid, enablesid, modifysid confs for pulled pork and running the IDS to check for false positives.

If there are multiple operating systems in your environment, for best results define ipvars to isolate the different OSs. This will ensure you can eliminate false positives whilst maintaining a tight alerting policy.

HttpInspect

From doc: HttpInspect is a generic HTTP decoder for user applications. Given a data buffer, HttpInspect will decode the buffer,  find HTTP fields, and normalize the fields. HttpInspect works on both client requests and server responses.

Global config –

Custom rules

Writing custom rules using snorts lightweight rules description language enables snort to be used for tasks beyond intrusion detection. This example will look at writing a rule to detect Internet Explorer 6 user agents connecting to port 443.

Rule Headers -> [Rule Actions, Protocols, IP Addresses and ports, Direction Operator,

Rule Options -> [content: blah;msg: blah;nocase;HTTP_header;]

Rule Option categories:

  • general – informational only — msg:, reference:, gid:, sid:, rev:, classtype:, priority:, metadata:
  • payload – look for data inside the packet —
    • content: set rules that search for specific content in the packet payload and trigger a response based on that data (Boyer-Moore pattern match). If there is a match anywhere within the packets payload the remainder of the rule option tests are performed (case sensitive). Can contain mixed text and binary data. Binary data is represented as hexdecimal with pipe separators — (content:”|5c 00|P|00|I|00|P|00|E|00 5c|”;). Multiple content rules can be specified in one rule to reduce false positives. Content has a number of modifiers: [nocase, rawbytes, depth, offset, distance, within, http_client_body, http_cookie, http_raw_cookie, http_header, http_raw_header, http_method, http_uri, http_raw_uri, http_stat_code, http_stat_msg, fast_pattern.
  • non-payload – look for non-payload data
  • post-detection – rule specific triggers that are enacted after a rule has been matched

Validating certificate chains with openssl

Using openssl to verfiy certificate chains is pretty straight forward – see a full script below.

One thing that confused me for a bit was how to specify trust anchors without importing them to the pki config of the os (I also did not want to accept all of the trust anchors).

So.. here what to do for specif trust anchors

So here’s a simple script that will pull the cert chain from a [domain] [port] and let you know if it is invalid – note there will likely be come bugs from characters being encoded / return carriages missing:

SSL Review part 2

RSA in practice

Initializing SSL/TLS with https://youtube.com

In this example the youtube server is authenticated via it’s certificate and an encrypted communication session established. Taking a packet capture of the process enables simple identification of the TLSv1.1 handshake (as described: http://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_handshake):

Packet capture download: http://mchost/sourcecode/security_notes/youtube_TLSv1.1_handshake_filtered.pcap

The packet capture starts with the TCP three-way handshake – Frames 1-3

With a TCP connection established the TLS handshake begins, Negotiation phase:

  1. ClientHello – Frame 4 – A random number[90:fd:91:2e:d8:c5:e7:f7:85:3c:dd:f7:6d:f7:80:68:ae:2b:05:8e:03:44:f0:e8:15:22:69:b7], Cipher suites, compression methods and session ticket (if reconnecting session).
  2. ServerHello – Frame 6 – chosen protocol version [TLS 1.1], random number [1b:97:2e:f3:58:70:d1:70:d1:de:d9:b6:c3:30:94:e0:10:1a:48:1c:cc:d7:4d:a4:b5:f3:f8:78], CipherSuite [TLS_ECDHE_ECDSA_WITH_RC4_128_SHA], Compression method [null], SessionTicket [null]
  3. Server send certificate message (depending on cipher suite)
  4. Server sends ServerHelloDone
  5. Client responds with ClientKeyExchange containing PreMasterSecret, public key or nothing. (depending on cipher suite) – PreMasterSecret is encrypted using the server public key
  6. Client and server use the random numbers and PreMsterSecret to compute a common secret – master secret
  7. Client sends ChangeCipherSpec record
  8. Client sends authenticated and encrypted Finished – contains a hash and MAC of previous handshake message
  9. Server decrypts the hash and MAC to verify
  10. Server sends ChangeCipherSpec
  11. Server sends Finished – with hash and MAC for verification
  12. Application phase – the handshake is now complete, application protocol enable with content type 23

client random: 90:fd:91:2e:d8:c5:e7:f7:85:3c:dd:f7:6d:f7:80:68:ae:2b:05:8e:03:44:f0:e8:15:22:69:b7 = 10447666340000000000

server random: 1b:97:2e:f3:58:70:d1:70:d1:de:d9:b6:c3:30:94:e0:10:1a:48:1c:cc:d7:4d:a4:b5:f3:f8:78 = 1988109383203082608

Interestingly the negotiation with youtube.com and chromium browser resulted in Elliptic Curve Cryptography (ECC) Cipher Suitesfor Transport Layer Security (TLS) as the chosen cipher suite.

Note that there is no step mention here for the client to verify then certificate. In the past most browsers would query a certificate revocation list (CRL), though browsers such as chrome now maintain either ignore CRL functionality or use certificate pinning.

Chrome will instead rely on its automatic update mechanism to maintain a list of certificates that have been revoked for security reasons. Langley called on certificate authorities to provide a list of revoked certificates that Google bots can automatically fetch. The time frame for the Chrome changes to go into effect are “on the order of months,” a Google spokesman said. – source: http://arstechnica.com/business/2012/02/google-strips-chrome-of-ssl-revocation-checking/

nf_conntrack: table full, dropping packet on Nessus server

Issue caused by having iptables rule/s that track connection state. If the number of connections being tracked exceeds the default nf_conntrack table size [65536] then any additional connections will be dropped. Most likely to occur on machines used for NAT and scanning/discovery tools (such as Nessus and Nmap).

Symptoms: Once the connection table is full any additional connection attempts will be blackholed.

 

This issue can be detected using:

Current conntrack settings can be displayed using:

To check the current number of connections being tracked by conntrack:

Options for fixing the issue are:

  1. Stop using stateful connection rules in iptables (probably not an option in most cases)
  2. Increase the size of the connection tracking table (also requires increasing the conntrack hash table)
  3. Decreasing timeout values, reducing how long connection attempts are stored (this is particularly relevant for Nessus scanning machines that can be configured to attempt many simultaneous port scans across an IP range)

 

Making the changes in a persistent fashion RHEL 6 examples:

These changes will persist on reboot.

To apply changes without reboot run the following:

To review changes:

Reference and further reading: http://antmeetspenguin.blogspot.com.au/2011/01/high-performance-linux-router.html

Setting secure, httpOnly and cache control headers using ModSecurity

Many older web applications do not apply headers/tags that are now considered standard information security practices. For example:

  • Pragma: no-cache
  • Cache-Control: no-cache
  • httpOnly and secure flags

Adding these controls can be achieved using ModSecurity without any needs to modify the application code.

In the case where I needed to modify the cookie headers to include these now controls I added the following to core rule set file: modsecurity_crs_16_session_hijacking.conf.

 

This adds the cookie controls we were after – Depending on your web application you may need to change ‘JSESSIONID’ to the name of the relevant cookie.

You can find the cookie name simply using browser tools such as Chrome’s Developer Tools (hit F12 in chrome). Load the page you want to check cookies for, click on the Resources tab:

ChromeCookies

After setting the HTTPOnly and Secure flags you can check the effectiveness using the Console table and listing the document cookies… which should now return nothing.

document.cookie

Migrating to EJBCA from OpenSSL and TinyCA

Install and configure EJBCA

EJBCA 6.0.3 – http://www.ejbca.org/download.html

JBoss AS 7.1.1 Final – http://download.jboss.org/jbossas/7.1/jboss-as-7.1.1.Final/jboss-as-7.1.1.Final.zip

Prereqs:

Ref:

Detailed deployment guide: http://majic.rs/book/free-software-x509-cookbook/setting-up-ejbca-as-certification-authority

EJBCA doc: http://wiki.ejbca.org/

Architecture

Recommended architecture (source: http://ejbca.org/architecture.html)

Import existing OpenSSL CA

Step 1 – Export the OpenSSL priv key and cert to a PKCS#12 keystore:

Step 2 – Import the PKCS#12 keystore to EJBCA CA

Step 3 – Verify import

### IMPORTANT ###

Distinguished name order of openssl may be opposite of ejbca default configuration – http://www.csita.unige.it/software/free/ejbca/ … If so, this ordering must changed in ejbca configuration prior to deploying (can’t be set on a per CA basis)

Have not been able to replicate this issue in testing.

Import existing TinyCA CA

Basic Admin and User operations

Create and end entity profile for server/client entities

Step 1 – Create a Certificate Profile (http://wiki.ejbca.org/certificateprofiles)

Step 2 – Create and End Entity Profile (http://wiki.ejbca.org/endentityprofiles)

* EndEntities can be deleted using:

Issuing certificates from CSRs

End entities need to be created for clients/servers that require certificates signed by our CA.

Step 1 – Create and End Entity (http://ejbca.org/userguide.html#Issue a new server certificate from a CSR)

Step 2 – Sign CSR using the End Entity which is associated with a CA

Importing existing certificates

EJBCA can create endentities and import their existing certificate one-by-one or in bulk (http://www.ejbca.org/docs/adminguide.html#Importing Certificates). Bulk inserts import all certificates under a single user which may not be desirable. Below is a script to import all certs in a directory one by one under a new endentity which will take the name of the certificate CN.

Creating administrators

Create administrators that can sign CSR and revoke certificates: http://ejbca.org/userguide.html#Administrator%20roles

Revoking certificates

Checking certificate validity/revoke status via OSCP

Monitoring expiring certs

 

Getting started with ModSecurity

XSS, CSRF and similar types of web application attacks have overtaken SQL injections as the most commonly seen attacks on the internet (https://info.cenzic.com/2013-Application-Security-Trends-Report.html). A very large number of web application were written and deployed prior to the trend up in likelihood and awareness of XSS attacks. Thus, it is extremely important to have an effective method of testing for XSS vulnerabilities and mitigating them.

Changes to production code bases can be slow, costly and can miss unreported vulnerabilities quite easily. The use of application firewalls such as ModSecurity (https://github.com/SpiderLabs/ModSecurity/) become an increasingly attractive solution when faced with a decision on how to mitigate current and future XSS vulnerabilities.

Mod Security can be embedded with Apache, NGINX and IIS which is relativity straight forward. In cases where alternative web severs are being used ModSecurity can still be a viable option by creating a reverse proxy (using Apache of NGINX).

How can ModSecurity be used?

  • Alerting
  • Transforming
  • Blocking
  • and more

These functions can be enacted by rules.

A default action can be created for a group of rules using the configuration directive “SecDefaultAction

Using the following SecDefaultAction at the top of rule set that we want enable blocking and transforming on is a blunt method of protection. Redirection can also be used as a method of blocking.

A powerful web application firewall - free software!
A powerful web application firewall – free software!

Example of a default action to be applied by ruleset (note defaults cascade through the ruleset files):

SecDefaultAction “phase:2,log,auditlog,deny,status:403,tag:’Unspecified usage'”

Rulesets have been created by OWASP.

Using optional rulesets,  modsecurity_crs_16_session_hijacking.conf and modsecurity_crs_43_csrf_protection.conf ModSecurity can provide protection against Cross Site Request Forgeries [CSRF]. The @rsub operators can inject a token on every form (and/or other html elements). ModSecurity can store the expected token value as a variable which is compared to the value posted via forms or other html elements. ModSecurity rules can be based on request methods and URIs etc – alongside the ability to chain rules there are a huge number of options for mitigating XSS and CSRF without impacting normal applicatioin usage.

@rsub

Requirements:

  • SecRuleEngine On
  • SecRequestBodyAccess On
  • SecResponseBodyAccess On

## To enable @rsub

  • SecStreamOutBodyInspection On
  • SecStreamInBodyInspection On
  • SecContentInjection On

Injecting unique request id from mod_unique_id into forms:

Some simple rules:

Pros:

  • Wide capabilities for logging, alerts, blocking, redirecting, transforming
  • Parses everything coming into your web server over HTTP
  • Virtual patching – if a vulnerability is made public that affects your web application you can write and deploy a rule to mitigate the vulnerability much faster than re-release of application code patched
  • Extended uses – the capabilities of ModSecurity can be applied to applications outside the scope of application security

Cons:

  • Added complexity to your application delivery chain – another point for maintenance and failure
  • Performance costs? – Though I have not had the opportunity to test the performance costs holding session information in memory and inspecting every byte of HTTP traffic can’t be free from performance cost
  • Hardware costs – Particularly if using ModSecurity’s BodyAccess and BodyInspection features, memory usage will be significant

Improving deployments:

  • Starting off being aggressive on warnings and very light on action is a necessity to ensure no impact on normal application usage
  • From this point rules and actions need to be refined
  • Understanding how the applications works allows the use of ModSecuirtys header and body inspection in effective ways

Some other notes extracted from the ModSecurity Handbook – If you decide to use ModSecurity I strongly recommend buying the handbook. It is not expensive and saves a lot of time.

### RULE STRUCTURE ###
SecRule VARIABLES OPERATOR [TRANSFORMATION_FUNCTIONS, ACTIONS]

SSL Review part 1

Most of us use and rely on SSL everyday. The mathematical workings of the RSA [Rivest, Shamir, Adleman] algorithm are not overly complex but mapping everything back to what happens in reality requires detailed understanding. Skipping over the need for SSL (for confidential and authenticated exchange of a symmetric key over and insecure medium) I will review the mathematical workings then how they are applied in real world examples.

There are also details in previous posts – RSA1, RSA2

Mathematics 

Step
Components
1. public key – e
standard practice to choose: 65537 
2. random primes p,q
Let’s use:

579810099525248565010050509754571001027,

6989752565699505597485398979958574481969

3. key modulus – n
 n = pq:

4052729130775091849638047446256554071699019514021047339267026030072286291982163

4. φ(n) = (p – 1)(q – 1)
 φ(n):

4052729130775091849638047446256554071691449951355822585104530580582573146499168

5. find e that is co-prime with φ(n)
 already using a prime,which will be co- prime.. – e = 65537 (in binary 10000000000000001)
6. (ed) mod φ(n) = 1d = e–1 mod φ(n) –  Modular multiplicative inverseMore than one answer
using Extended Euclidean algorithm:

944402082567056818708092537028397604145319798848072425038015030084640082599681,

4997131213342148668346139983284951675836769750203895010142545610667213229098849,

..+ 2φ(n), +3φ(n))

7. private key – d
 de–1 mod φ(n):

944402082567056818708092537028397604145319798848072425038015030084640082599681

With a public key (e), a key modulus (n) and a private key (d) we can apply the RSA algorithm.

Message (mess) = 911

RSA encrypt -> mess ^ e mod n  = 911 ^65537 mod 4052729130775091849638047446256554071699019514021047339267026030072286291982163

RSA encrypted message (ciph) = 3095021178047041558314072884014000324030086129008597834642883051983162360819331

RSA decrypt -> ciph ^ d mod n = 3095021178047041558314072884014000324030086129008597834642883051983162360819331 ^ 944402082567056818708092537028397604145319798848072425038015030084640082599681 mod

4052729130775091849638047446256554071699019514021047339267026030072286291982163

= 911

How is that secure?

When Alice encrypts using Bob’s public key (e) along with the key modulus (n) the output is a protected cipher.

An eavesdropper does not know the private key so decryption is very difficult:

Attacker must solve:

(unknown val, x) ^ e mod n = ciph

x ^65537 mod 4052729130775091849638047446256554071699019514021047339267026030072286291982163 = 3095021178047041558314072884014000324030086129008597834642883051983162360819331

OR, easier – try to determine the private key:

The attacker knows e and n (which = pq). When we created the private key (step 6 above) we conducted:  e–1 mod φ(n) – Modular multiplicative inverse which is relatively fast for us to calculate.

The attacked does not know  e–1 mod φ(n) though. φ(n) = (p – 1)(q – 1). The attacker knows that n is a composite prime = pq (where p and q are both primes).

So… if the attacker can solve p * q = n (where they know n) then RSA is insecure.

Thankfully the process of Integer factorization is so much harder than the process of creating p,q,nφ(n), e and d that online business and confidentiality can be maintained to acceptable levels.

Threats to RSA

It would be extremely valuable to malicious individuals/groups and  (more importantly) intelligence organizations make large integer factorization efficient enough to break RSA.

However the theoretical aspects of RSA are not generally recognized as the main source of vulnerability