Thursday, October 23, 2014

If You Haven't Pen-Tested Now, Wait

Abstinence Or Whatever

This morning, my esteemed peer Shawn Moyer referred to a blog post he wrote in September 2013 on waiting for pen-testing until Q1, but buying in Q4. He's not wrong

Shawn makes strong points about the money and the actual work-load on the consultant side. The fact is, teams traditionally get so swamped in Q4 that they do indeed place junior resources where they shouldn't. But, Atredis and I have one thing in common: we don't have to care about that problem. None of us are juniors. We're all principal-level consultants. Nor do we have the overhead of managing or maintaining interns to try and eek out results. 

While Shawn discusses the pen-testing team's side of the fence, I'm discussing the client's issues with scheduling a penetration test in Q4. If you haven't purchased a pen-testing engagement already, you should schedule the test for Q1. 

There Goes E911

Q4 isn't just the most lucrative time of year for tech companies, it's the most profitable time of year for most companies. As a result, everyone is on high alert to manage their resources as effectively as possible. This means not focusing on organizational security in the event of an emergency. Case in point? 

Intrado, the company that manages the majority of the United States' E911 service had their system fail for over 11 million people across seven states, including the entire state of Washington. No one could make a 911 call during that time period. Intrado happens to be based here in Colorado, and I happen to have personal experience with them. The fact is, they're an exceptional company and I have been impressed by their above-average engineering expertise. 

Regardless, how do you think the organization - that is one of the country's only third-party 911 contractors - would have reacted if this event occurred during Q4? During Black Friday? During Christmas? God forbid, during Devil's Night? The security and engineering teams would be completely re-tasked toward assessing an event like this, why it occurred, how to remediate it, whether a bad actor was involved, etc. Any penetration test at this point would - and should - immediately stop. 

Point being, whether you're managing E911 infrastructure for the entire country, or simply building a web service that caters to hundreds of thousands of engineers world-wide, prioritization is key. Losing customers is never an acceptable choice. When resources are constrained during an already busy time of year, priorities must align with the business' key goals, and nothing else. 

Schedule Effectively 

So, if you're considering buying penetration services now, don't. Why? It's already going to be the last week of October. Buying services now generally means 
  • one to three weeks of sales process
  • a week or two of scheduling resources on both sides
  • actual engagement: anywhere from 2 days to 2 weeks (on average)
If you're buying services now, this puts your actual engagement starting date anywhere from November 10th to December 1st. If you're on the light side of a test and only need a couple days of effort, you're still looking at remediation through one of the busiest holiday seasons in the country: Thanksgiving. If you're on the heavy side of the test, you're going to be running into mid-December. That means the week of Dec. 15th your team will be scrambling to remediate security issues during *the* busiest two holidays in the world: Christmas and New Year's Day

Any critical event during this time means that the results of any penetration test must be put off until the critical event has passed. On average, any overlapping operations/engineering/security event takes between one week to a month to evaluate, remediate, and monitor. During this already swamped time of year, that means that the results of any penetration test will be ignored for up to two months if a critical event occurs adjacent to a common vacation or holiday period. This is a total waste of money! 

Don't Kill Money!

The only value of a penetration test has is when the results can be used within an effective period of time. This means weeks after the penetration test has occurred. Otherwise, because of the increased amount of security risks being identified in modern times, any penetration test performed today will have drastically different results than one performed a month from now. 

Remember ShellShock? Heartbleed? Cisco's ASA flaws? LZ4? SSLv3 POODLE? We're seeing more and more game changing flaws coming out, and security teams are already flooded with a To Do list longer than Santa's naughty list. 

Scheduling a pen-test during Q4 is basically asking to be put on Santa's naughty list for knowingly booking an engagement whose output can't be fully utilized in an effective time frame. Don't do it! Don't kill money!

Sure, spend the money today. Book the team. Get the most effective engagement for your budget and your organization's needs. But, schedule it for a time when the output of the engagement will bring the most value to your organization. That, typically, means Q1. 

So, if you want the most value out of a security review, check out Lab Mouse. Even check out Atredis. Book a team that is going to maximize your investment by providing you with only top-tier talent, but only schedule the engagement when the timing makes sense for your organization. 

Hey, Intrado, if you need someone that specializes in mobile/embedded/Erlang/telco to help you with a security or code review of your E911 system, give me a shout. I'll give you a good deal since you're local and I already love ya. 

Don A. Bailey
Lab Mouse Security

Tuesday, October 21, 2014

GoLang Debugging - Turning Pennies Into G's

GDB Ain't Great

Our favorite application debugger is awesome. Don't get me wrong, I use it often. Almost daily. But, the fact remains that GDB is dependent on a predefined legacy application architecture. GDB wasn't designed to predict new application architectures. As a result, it isn't elegant at supporting alternatively designed stacks, concurrency models, or execution flows. 

That's essentially why GDB has been enhanced with extension capabilities like Sequences, Guile, and yes, even Python. Unfortunately, even the extension system is a bit lacking. Case in point? The existing GoLang GDB script uses a completely outdated extension API that causes exceptions. It wasn't super useful to begin with, either. 

donb@evian-les-bains:~/home/library/golang/go/src/pkg/regexp$ GOMAXPROCS=9 /usr/bin/gdb -q ./regexp.test
Reading symbols from ./regexp.test...done.
Loading Go Runtime support.
(gdb) run
Starting program: /home/donb/home/library/golang/go/src/pkg/regexp/regexp.test 
Program received signal SIGINT, Interrupt.
runtime.usleep () at /home/donb/home/library/golang/go/src/pkg/runtime/sys_linux_amd64.s:77
77              RET
(gdb) info goroutines 
Python Exception <class 'gdb.error'> Attempt to extract a component of a value that is not a (null).: 
Error occurred in Python command: Attempt to extract a component of a value that is not a (null).
In the above example, we see a common debugging scenario that's been going on for over a year or two. The debugging process is even documented on the GoLang website under Debugging Go Code With GDB. Yet, when one tries to reproduce the steps outlined in the documentation, the above error occurs. 

Why does this happen? The script packaged with GoLang uses an outdated model for accessing gdb.Value objects, essentially treating them as dictionaries. Because the internals of the gdb.Value no longer support this model, or allow linking via Python generators in the fashion used in the script, simple commands will fail.

Go Routine Yourself

To solve these horrors, we really just need to generate a class that handles retrieval of pertinent values from the GoLang runtime environment. For those not in the know, each GoLang application acts like a kernel, scheduling the execution of each Go routine, monitoring memory allocations, and preparing for garbage collection.

Before you start to become concerned about potentially severe user-land bloat, take a moment to realize that this is actually the correct architecture. Any robust production quality application must handle resources elegantly by silently monitoring for operating system signals, scheduling and managing thread synchronization, handling per-thread intercommunication, and efficiently allocating and deallocating subsystem resources transparently. GoLang accomplishes all these things, but with surprising eloquence and a new level of light-weight that even Magdalena Frackowiak would be jealous of.

GoLang executes in tiny self-contained executable images called Go Routines, which are co-routines handled by GoLang's internal scheduler. GoRoutines are not threads, they are simple co-routines that execute under an Operating System thread. The benefit of this is that GoRoutines can move transparently across OS threads (pthread, libthread (Solaris), etc) with almost no cost to the application.

GoRoutines are managed by the C language structure 'struct G'. Operating system threads are managed with the C language structure 'struct M'. Therefore, for every OS thread M, one or more G may run within it. There are other abstractions within the GoLang scheduler, but since those aren't relevant to this discussion we'll leave those abstractions alone for now.

All The G

Internally, GoRoutines are managed under the runtime package. There is a variable called runtime.allg which points to a list of all GoRoutines in the system. A corresponding length variable, runtime.allglen, defines how large this array is. As GoRoutines die, they are marked by their status as Dead. But, unless it is overwritten at some point, the pointer in allg lives on. So, you can inspect what's left of a GoRoutine even after it has moved on to its next iteration.

To solve our problem with GDB, we have to inspect the allg variable. As can be seen in the existing code, this used to be as easy as calling gdb.parse_and_eval. Now that we can no longer act this easily, we have to use what resources are available to us to retrieve values from memory, even if it's a notorious pain in the ass.

Let's build a simple Python class around this idea. Because I solved my problem with this code last night in two hours while watching classic episodes of The Rockford Files, it isn't a super great solution. Regardless, it works and frankly because no one else has solved this problem for 2+ years, I don't care if you don't like it.

class Allg:
    __allglen = -1
    __position = 0
    __allg = 0

    __offsets = {
            'status': 152,
            'waitreason': 176,
            'goid': 160,
            'm': 200,
            'sched': 40,
            'sched.pc': 48,
            'sched.sp': 40,
            'stackguard': 120,
            'stackbase': 8,

    def __init__(self):
        # first, fetch the number of active goroutines
        self.__allglen = int(gdb.parse_and_eval("&{uint64}'runtime.allglen'"))
        print("found allglen = {0}".format(self.__allglen))

        # get the next address in the array
        s = "&*{uint64}(&'runtime.allg')"
        self.__allg = int(gdb.parse_and_eval(s))
        print("found allg = {0}".format(hex(self.__allg)))

    def fetch(self):
        if self.__position >= self.__allglen:
            return None

        s = "&*{uint64}(" + "{0}+{1})".format(self.__allg, self.__position*8)
        p = int(gdb.parse_and_eval(s))
        self.__position += 1
        return p

    def Status(self, a):
        s = "&*{int16}(" + "{0}+{1})".format(a, self.__offsets['status'])
        return int(gdb.parse_and_eval(s))

    def WaitReason(self, a):
        s = "&*{int64}(" + "{0}+{1})".format(a, self.__offsets['waitreason'])
        x = int(gdb.parse_and_eval(s))
        s = "&{int8}" + "{0}".format(x)
        return str(gdb.parse_and_eval(s))

    def Goid(self, a):
        s = "&*{int64}(" + "{0}+{1})".format(a, self.__offsets['goid'])
        return int(gdb.parse_and_eval(s))

    def M(self, a):
        s = "&*{uint64}(" + "{0}+{1})".format(a, self.__offsets['m'])
        return int(gdb.parse_and_eval(s))

    def Pc(self, a):
        s = "&*{uint64}(" + "{0}+{1})".format(a, self.__offsets['sched.pc'])
        return int(gdb.parse_and_eval(s))

    def Sp(self, a):
        s = "&*{uint64}(" + "{0}+{1})".format(a, self.__offsets['sched.sp'])
        return int(gdb.parse_and_eval(s))

    def Stackguard(self, a):
        s = "&*{uint64}(" + "{0}+{1})".format(a, self.__offsets['stackguard'])
        return int(gdb.parse_and_eval(s))

    def Stackbase(self, a):
        s = "&*{uint64}(" + "{0}+{1})".format(a, self.__offsets['stackbase'])
        return int(gdb.parse_and_eval(s))

Using the class Allg, I simply identify the address of the runtime.allg symbol in memory, and its corresponding size parameter, runtime.allglen. Once I store these parameters internally, I can just fetch every subsequent GoRoutine's address from the array. Since these routines are allocated sequentially in the array, I can fetch them using a simple iterator. Then, I just pass back the pointer of the actual G* structure. Any time the caller wants to learn more about a specific G*, they just pass back the address to any other function in the class, which will return the value for the corresponding G* field. 

This simple class makes data retrieval very easy. Let's look back at the class that gets invoked when we execute info goroutines on the GDB command line. 

class GoroutinesCmd(gdb.Command):
    "List all goroutines."
    __allg = None

    def __init__(self):
        gdb.Command.__init__(self, "info goroutines", gdb.COMMAND_STACK, gdb.COMPLETE_NONE)

    def invoke(self, _arg, _from_tty):
        self.__allg = Allg()

        # donb: we can retrieve the correctly size pointer with a cast
        # (gdb) python \
        # print("{0}".format(gdb.parse_and_eval("&*{uint64}&'runtime.allg'")))
        while True:
            ptr = self.__allg.fetch()
            # print("fetched ptr = {0}".format(hex(ptr)))
            if not ptr:

            st = self.__allg.Status(ptr)
            # print("status is {0}".format(st))
            w = self.__allg.WaitReason(ptr)
            # print("waitreason is {0}".format(w))
            #if st == 6:  # 'gdead'
                #print("skipping over dead goroutine")

            s = ' '
            m = self.__allg.M(ptr)
            if m:
                s = '*'

            # if the status isn't "waiting" then the waitreason doesn' tmatter
            if st != 4:
                w = ''
            w2 = w.split('"')
            if len(w2) > 1:
                w = """waitreason="{0}\"""".format(w2[len(w2) - 2])

            pc = self.__allg.Pc(ptr)
            blk = gdb.block_for_pc(pc)
            goid = self.__allg.Goid(ptr)
            a = "fname={0} faddr={1}".format(blk.function, hex(pc))
            print(s, goid, "{0:8s}".format(sts[st]), a, "&g={0}".format(hex(ptr)), w)

How simple is that? Now, the routine can fetch each G* from within the invoke function's while loop, and print information regarding the runtime. 

donb@evian-les-bains:~/home/library/golang/go/src/pkg/regexp$ GOMAXPROCS=9 /usr/bin/gdb -q ./regexp.test
Reading symbols from ./regexp.test...done.
Loading Go Runtime support.
(gdb) run
Starting program: /home/donb/home/library/golang/go/src/pkg/regexp/regexp.test 
Program received signal SIGINT, Interrupt.
runtime.usleep () at /home/donb/home/library/golang/go/src/pkg/runtime/sys_linux_amd64.s:77
77              RET
(gdb) info goroutines 
found allglen = 5
found allg = 0xc208018000
  16 waiting  fname=runtime.park faddr=0x4134d9 &g=0xc208002120 waitreason="chan receive"
* 17 syscall  fname=runtime.notetsleepg faddr=0x404a56 &g=0xc208002480 
  18 waiting  fname=runtime.park faddr=0x4134d9 &g=0xc208032240 waitreason="GC sweep wait"
  19 waiting  fname=runtime.park faddr=0x4134d9 &g=0xc2080325a0 waitreason="finalizer wait"
* 31 waiting  fname=runtime.gc faddr=0x40a0c6 &g=0xc2080326c0 waitreason="garbage collection"
(gdb) goroutine 31 bt
found allglen = 5
found allg = 0xc208018000
#0  0x000000000040a0c6 in runtime.gc () at /home/donb/home/library/golang/go/src/pkg/runtime/mgc0.c:2329
#1  0x000000000040a150 in runtime.gc () at /home/donb/home/library/golang/go/src/pkg/runtime/mgc0.c:2306
#2  0x00007fff00000000 in ?? ()
#3  0x000000c21531e000 in ?? ()
#4  0x000000000055fc00 in type.* ()
#5  0x0000000000000001 in ?? ()
#6  0x0000000000000000 in ?? ()

We can even use the goroutine command using the same Python class to retrieve information about a specific GoRoutine, and then execute a gdb command based on that routine. Excellent!


This isn't a great way to debug GoLang. There is a lot that is left desired here. For example, stack backtraces are still difficult because of the execution architecture. GoLang's toolchain uses an internal Base Pointer (BP) and doesn't emit one when the binary is generated. This is a legacy of the Plan 9 Operating System assembler, which is intelligent on CISC ASM architectures such as x86/64 becuase it enables %R/BP to be used as a general register. 

But, as a result, GDB doesn't know how the heck to rewind the stack. In fact, you have to inspect the current function's SP adjustment code to identify how far to rewind the stack before popping off the return value. I've accomplished this (the basics) in another change I've made to the GDB script. But, I'll share that another time once I finish dealing with some of the gotchas of this method. 

Regardless, for now, you have a simple way to print each GoRoutine during a GDB session. You also have an easy way to identify where in memory each G* exists, and can inspect them with ease, and that's a lot better than you've had it for the past couple of years! 

GoLang Security Auditing 

Are you worried about the real internal security surface of the GoLang application architecture? Are you worried about how the subtleties of the custom scheduler can affect data consistency across co-routines? Are you wondering if the split-stack architecture puts you at risk for memory segement collision under significant client-request pressure? Are you concerned that poorly-written third party libraries might subvert the otherwise mostly-sound GoLang security model?

Come check out Lab Mouse Security! I've been working with GoLang since the project was made public. I understand the internal runtime architecture, the compiler toolchain, and how the security model affects real-world applications. If you're interested in GoLang security, consider having Lab Mouse evaluate the security of your GoLang application today! 

Don A. Bailey
Lab Mouse Security

Tuesday, October 7, 2014

The Internet of Us

It'll all be OK, little guy. 

It's Not Me, It's You

I've been analyzing and building Internet of Things technology since 2009. At the time, my wife Jessica and I were living in a condo building in Denver's Capitol Hill neighborhood. Nick DePetrillo and I had just started working on The Carmen San Diego project, and I was just launching my career at iSEC Partners after leaving a failing security practice elsewhere. It was a stressful but exciting time. 

One of the ways I dealt with the stress was learning how to solve community problems with technology. Our condo building was tall, and the stairs were common "hotel style" stairwells, not optimal for travel in any scenario. Living on the seventh floor, we opted to take the elevators like everyone else. Normally this would not invite any kind of concern, except that we lived with Jessica's 80 pound golden doodle, Jasper. 

By a court of law!
Jasper is a great guard dog. He's as beautiful as he is vicious, and he is exceptional at protecting us from falling leaves, marauding squirrels, and even actual creepers lurking around Denver's parks. He's also quite disciplined. In elevators, he'd sit patiently and wait for the doors to open without a sound. Jessica did an excellent job of training him. 

The problem would come when the older residents of the condo would get into the elevator. Well, really, only two particular individuals. One was an older lady that enjoyed making a problem out of fading paint on the walls. She relayed to our door man that it was unacceptable for a dog to be in the elevator at the same time as she was. Ridiculous! The insolence of a couple with an approved dog traveling to and from their own condo! Unthinkable! 

I have a feeling that the woman was trying to somehow bully the condo board into allowing her exclusive access to an elevator. Seriously. Regardless, it presented an awkward problem as we didn't appreciate getting a complaint just for getting in the elevator with a dog that wasn't even behaving badly. 

No, Really. It's You.

My solution to this problem was tinkering. I attached a Zigbee module to an Atmel microcontroller, used an Electronic Assembly DOGS LCD screen, a few LEDs, and two LED-backlit push buttons. Result? I had a little mesh-based alerting system that would notify you if Jasper was in the elevator! 

The code was simple. As soon as I left my condo I could press the button on what I named the "Beagle Box" (I was unaware of the Beagle Bone at the time; don't hate). Pushing the button would send a message to all nodes in the mesh that Jasper would be in the elevator for the next 5 minutes. Either the "warning" would time out, or I could log in on another "Beagle Box" in the lobby. 

Class doesn't automatically come with age.
I never ended up deploying the box, because designing it made me realize that there was an easier solution to my problem. I could simply record a few videos of Jasper behaving perfectly in an elevator with other tenants. Then, I wouldn't have to bother maintaining equipment or debug RF signaling issues throughout the building. 

Point being, I found a social method for diffusing the problem. But, the technology itself made me realize a few key things about the Internet of Things. 

First, people that are eager to create problems can be exposed by technology that disputes their manipulative or sociopathic point of view. 

Second, real community problems can be isolated, evaluated, and potentially mitigated through cost-effective and practical technology. 

It Can Be Us.

I don't see the Internet of Things as just another trend in technology. I see IoT as the next generation of the Internet. But, the Internet is no longer about desktops and servers and intangible opaque applications. The Internet is about Us. We'll be living in the Internet of Us, and we need to think about how to build for Us, not I. Not the 1%. Not the Silicon Valley VC pool. Us. 

And Us isn't easy, is it? Us includes a population that isn't as technologically savvy as my peers. According to Pew Research, one in four teens solely uses a mobile phone to access the Internet. In fact, Pew goes on to state that teens in lower income socio-economic groups are even more likely to use their mobile phone as their primary Internet access point. 

Why is this important? Twenty-five percent (25%) of teens lack access to the Internet in their home, which is why they are focusing on their mobile phone as their pivot point to the Internet. Only half of teens have access to smart phones, which drastically changes the content that teens access, especially at-risk teens. This means they aren't coding. They aren't learning shell commands. They're not even learning about web technology. They're information sinks. 

A much more profound game of telephone is happening right now.
If IoT is going to solve community problems, IoT must be ready to solve the communications issue as well as the socio-academic gap. Communications technology will keep getting cheaper, and soon these numbers will be more representative of a largely connected population. But, we're just talking about USA. What happens when we expand to the Americas, Europe, Africa, and other evolving countries? How do we ensure technology is seamless, secure, and usable for all academic levels? 

That's quite the challenge, which is why technology usually focuses on a subset of the population. It's easier. And that's fine. But, focusing on easy to deliver platforms that enable 1% of the population doesn't solve community need and only widens the gap between the have's and have-not's. 

But First, I Need You. 

To solve these problems we need a cost-effective, secure, and agile platform for the Internet of Things. I am building one based on my research, and am in the middle stages of developing the proof of concept. However, the key metric for success, to me, isn't a snazzy new Thing that solves all our Facebook-connectivity-audio-enablement-bass-drop concerns. It's a thing that binds communities together and enables security, safety, and education. 

I *am* the music! Errr..
For example, have you heard of Shot Spotter? I've you've ever seen me give a talk on IoT, you've probably heard me bring it up. Shot Spotter detects gun shots in real time, and passes the information along to law enforcement. It can detect the caliber of a gun, direction of a shot, whether a shooter is moving (drive by), and more. That's excellent technology! But, it comes at a steep price. 

Unfortunately, a lot of communities that want to use Shot Spotter are struggling because they have much more immediate concerns to deal with. Detroit and Flint Michigan are having serious water treatment and infrastructure concerns, resulting in a massive increase in the price of water bills. Do you think the city wants to shell out hundreds of thousands of dollars for a gun shot detection system when basic services are at risk? 

Ominous raven agrees: expensive technology is scary!
What if a secure open source version of the same technology could be built with a simple to use and cost-effective platform? What if we could help secure cities for dollars instead of millions? We can. Internet of Things technology can enable this, but only with the right minds working together. 

I want to bring together lawyers, law enforcement, technologists, city planners, and members of communities to discuss tangible issues that can be addressed with a cross-section of open-source technology, IoT, and information security. The goal isn't to disrupt cities, but to help them restructure their foundation. 

Want to help? Contact me at Lab Mouse Security. As I build the IoT platform, we can find ways to use it for more than the next generation of the Bluetooth speaker. We can change lives for the better. 

Lab Mouse Security

Monday, October 6, 2014

Start-Ups, Information Security, and Budgets

Start Up, not Down. 

The 80's Were Ok, I Guess

As a child of the 80's, I was raised with a lot of mixed messages. These messages took a lot of bizarre forms. I distinctly remember Poison's "Open Up and Say Ahh" being re-released solely because parental groups were concerned that the devilish cover was somehow hypnotizing teens into a riotous hormonal rage. It surprised me, even at the tender age of nine, that somehow covering up the image except for the eyes would appease these groups that supposedly cared about decency. 

Hair Metal Hijab!
George Carlin was more controversial

Weren't the lyrics and hair-metal themes far more of a concern than ridiculous cover art? Either way, I certainly didn't care because it meant that I was finally able to purchase my own copy of the album. And boy, was I thrilled to tear open that cassette packaging - completely ignoring the cover art - to settle into less than an hour of sonic distress that I would soon toss into a pile of tapes and forget.  

Much Better Music
Regardless, this was one of the earliest instances I can recall where adults made insincere compromises in the name of safety (and, I guess, decency). Let's not forget that only three years earlier, Tipper Gore founded Parents Music Resource Center. The PMRC advocated against music that glamorized sex, drugs, and violence (by which I guess they mean Rock and Roll). What they didn't tell us is that Tipper was (at the time) a closet dead head that went on tour in her youth. I bet she and Bill had many a nostalgic night reminiscing about never inhaling

Tipper performing Sugar Magnolia with The Dead 4/20/09

But, I'm Checking Boxes

These behaviors and mixed messages bleed into everything humans do. We project our perception of social norms onto the things we create, whether it's a plane, car, or an Internet connected watch. I've engaged with a growing number of organizations this year that are thinking about security from a similar perspective. It's easy for executives to talk with each other and identify the methods they are using to secure their networks. Because security seems like such an intangible black box, it is difficult to quantify the actual return when a budget is assigned to decrease risk. Therefore, if executives see that their company's activities fall in line with the activities of other companies that have not been publicly compromised, all must be well! 

However, as is often the case with simply checking boxes and moving on, reality is quite different. Home Depot, Target, and other major organizations subject to recent computer-based attacks were all subject to PCI compliance. This means that they were indeed checking boxes, validating patches, and scanning networks to ensure a decreased threat surface. And yet, they were compromised! 

Hackers gonna hack, AMIRITE!?
Penetration tests can assist in providing a base line for infrastructure, but that baseline is simply a snapshot in time. Because of the constant change in the security ecosystem, today's scan will never be representative of tomorrow's network, even if the network components haven't changed. Security is a commodity whose value is constantly in flux, so evaluating risk on this action alone is not only misleading, it's devoid of value. 

Penetration tests cost tens of thousands of dollars and may be performed once a month, or even once per quarter. Yet, if the security landscape of the organization is constantly at flux, these tests provide absolutely no insight into the real state of the organization and its assets. As a result, the organization is essentially flushing away hundreds of thousands of dollars down the toilet because it hasn't used the output of a penetration test effectively. 

Realigning Organizational Strategy

This trend is even more of a concern when the Internet of Things trend becomes a part of every day business. As IoT systems become more ubiquitous in the workplace, new threat models, assessments, and controls must be put into place to identify how to monitor, manage, and deploy these assets. If you thought Bring Your Own Device was bad, consider Bring Your Own Anything. 

In several cases over the past year and a half, clients have had difficulty seeing the value in evaluating the architecture of their IoT product or service. The executives I've spoken with have had concerns about the cost of such a review, which is understandable. When you pay money for a service that has a seemingly intangible return it is difficult to pull the trigger. This is especially true if you misunderstand your team's security experience and believe that patching and penetration testing has anything to do with threat modeling and architectural strengthening. 

The flip side of the coin exposes the other executives I've spoken with. These executives understand the value of a security review and believe they need it. However, they are having trouble allocating budgetary resources toward security review because it is too early to allocate funds for consultants on an intangible piece of the technology puzzle. These executives know the value, but are having trouble selling the value upstream because start ups have limited resources and must work at a fast pace. 

Doozers solving architectural security concerns (clearly).

These are all understandable problems that I actually sympathize with. It's difficult to understand what a valuable security practice is without having gone through the process of incorporating one into a product or service. It's even more difficult to incorporate a valuable security practice when your entire seed budget or A round is focused on boot-strapping a proof-of-concept that will get you partnerships in key verticals, or access to a larger client pool. 

Bottom Line it For Me

Let's go over some numbers, shall we? Maybe this will help elucidate the actual return of a security program integrated into product or service, either being developed or being used. 

On average, the cost of a penetration test is Man Hours multiplied by the days needed to scale for the scoped infrastructure, plus overhead, reporting, and on-site requirements. Let's start out with a simple example of Average Start Up (ASU). ASU has an average start-up size of twenty people. They have a small in-office datacenter, cloud infrastructure, and hosted physical servers in two separate locations (east coast and west coast). This is Pretty Average. 

Let's say ASU is smart and knows that the penetration testers are skilled, estimating that it will take them 2-3 days to compromise the network. ASU wants data from this project, so they need a well written report and assistance interpreting the data correctly. Add another 2 days of work. So, let's give the project a full five business-day scope. Two penetration testers are engaged in the process to minimize the time required to scan and evaluate the entire infrastructure. Consultants work a flat eight hour day. The industry average is around 250 USD per hour. But, since ASU is a start up, let's presume they get the "friendly" introductory rate of 150 USD per hour. So, we're talking:

Days = 5
Hours per day = 8
Consultants = 2
Price Per Hour = 150 USD
Days * Hours * Consultants * Price = 12,000 USD

Keep in mind that the 12,000 USD is only to obtain the results of the penetration test. Once personnel at the organization implement the changes required to "pass" the test, a new test must be performed. Let's just presume for now that this second test is free, since Pen Test Company (PTC) is running a special for Start Ups like ASU. 

But wait! There's more! ASU gets the benefit of the re-test that gives them a passing grade. But, they keep hearing about all these new vulnerabilities coming out of the wood work: LZO, LZ4, HeartBleed, BashBug, etcetera. 

And what's this about some bizarre new RSA 1024bit key hack thingie?!? Is that even real!? How do we test for that?! 

Ah, yes. Now that penetration testers have to come back. If it's once per quarter, now we're talking about 48,000 USD per year. If it's once a month, which is a more realistic number to get a somewhat reasonable analysis, now we're talking about 144,000 USD per year. Unsustainable. 

At this point executives are rightfully pissed, and probably feel pretty shitty about where their money is going. The security process seems unmanageable, and the money feels burned. 

Stop Killing Money

There is an easy way to stop this ridiculous cycle. It starts with a simple threat model, and ends with processes and controls that integrate security into not only the daily engineering process, but the work place. Every technology can be hacked. My work is proof of that. I have exceptional colleagues that are proof of that even more than I am: Charlie Miller and Chris Valasek, Zach Lanier and Mark Stanislav, Stephen Ridley, Thaddeus Grugq, Ben Nagy, and countless others. 

Yes, everything can be hacked. But, there is a method for reducing the potential for risk and managing it in a cost-effective manner:
  • Identify key assets that affect the business, its partners, and its customers
  • Prioritize assets by potential effect 
  • Build security goals around these prioritized assets
  • Define policies and procedures that support security goals
  • Ensure infrastructure supports security goals
  • Monitor infrastructure constantly, assign responsibilities
  • Integrate security engineering into the product and/or service life cycle
This simple seven step process will take a company from zero to hero far faster than they ever could with penetration testing engagements or managed security services. Let's look at a simple engagement for a consultant to walk ASU through this process.

Asset review/Priority meeting/Initial Threat Model = 5 days - 10 days avg. 
Defining policy and procedure/Enhancing infrastructure/Building baseline = 5 - 10
Define monitoring system/Assign responsibility/Add OSS controls = 5 - 10
Integrate security into SDLC (only if org has engineering dept) = 10
Average consulting rate = 250 USD per hour
Total High Watermark = 80,000 USD
Total Low Watermark = 50,000 USD

A single threat modeling and architecture enhancement engagement will cost between 50,000 USD and 80,000 USD. It typically only needs to be performed once. A solid architecture not only enables the organization to diminish its risk, but it helps the organization understand how to manage its security in a way that provides longevity. When a threat arises, the organization will be better equipped to respond effectively, rather than relying on an external organization to swoop in and solve the problem for a six figure price. 

In addition, internal penetration testing capability and vulnerability assessment automation can be integrated into the process defined by this engagement, allowing the organization to not only audit themselves but to interpret the results of the audit effectively. 

For an organization the size of our example, ASU, they would come in at the low end of the price spectrum. Even at the high per-hour rate of 250 USD, the project would still come out only 2,000 USD above the quarterly penetration testing price, but with a far greater return on investment! If ASU negotiated the price down to the same rate as the penetration testing team, 150 USD per hour, the cost would end up at 30,000 USD. This is only two and a half times the price of an average penetration testing engagement. 

Failure Shouldn't Be Feared

Will organizations get hacked? Of course. Will an organization with a well defined security practice still get hacked? Unfortunately, it is likely. But, will an organization with a well defined security practice identify, isolate, and expunge the threat quickly and effectively with far less risk to the business and its clients? Yes! 

The security process is not perfect. But, that is no reason to allocate resources to the wrong activities, then argue that we did the same things everyone else did when a risk is abused. Instead of learning to do the wrong things, we must be brave enough to do the things that are harder, sooner. 

If we learn these lessons, we'll be able to decrease risk not only in our working environments, but within our products and services. Failure isn't something to be terrified of. We must integrate the lessons learned into our processes to ensure that we are less likely to fail, instead of shaking our head and presuming that isn't going to be me.

More importantly, when companies get ready to IPO or become acquired, lawyers are telling us that it is becoming increasingly more common for security audits to be a forced part of the process. There have been more than a few cases in the past several years where companies were being evaluated for acquisition and failed because the security architecture was so unmanageable it would require a complete architectural overhaul. This is not how you get a shiny new red 1993 Porsche 911. 

Success is a mobile phone. 

Everyone fails sooner or later, especially in information security. Pretending like the underlying problem doesn't exist is the same as putting black bands on a Poison cover album and saying you've saved the innocence of American teenagers. The real sign of success in an information security program is how quickly you recover, how effectively you can isolate risk to your business and your clients, and how much your customers trust your transparency. 

As it happens, Lab Mouse is running a special discount on threat modeling and architecture security engagements for start-ups and small businesses! If you're in need of security services, please reach out! I will be happy to provide you with a valuable engagement that scales according to your budget. 

Best wishes,
Lab Mouse Security