Could you describe at a high level what your role is and what you do?
I have worked in security control centers at different companies for the better part of ten years. I’ve been a supervisor level since 2010 or 2011. In that time, I’ve mostly worked at the global control center level, which means: monitoring travelling employees and events around them, monitoring world events to make sure they don’t impact our sites worldwide, monitoring the sites themselves to make sure they’re secure, and monitoring any other incidents that might be going on around the sites that might put them in jeopardy. My role is to protect people, assets, and property — basically in that order.
I’d always worked for security contractors up until recently when I was hired by a manufacturing company. My position is very rarely in-house; usually, it’s only more senior positions. My current company has much more in-house staff on the control center site. The rest, such as the guard staff, are for the most part contract.
How did you originally get into this line of work?
I’d say part nepotism, part strength. I started in 2003, and I got the job because my mom worked in the legal department, and at the time security was under legal. They had a split swing/grave shift position open that I started in. In my first position, I was in the role of an “operator.” I was the one on the computer monitoring the alarms, answering the phones, doing the basic point-of-contact stuff, and doing the radios for the security officers.
After a few years there, I saw the writing on the wall that they were closing that control center and moving it to Austin. I had a friend who was working at a tech company and reached out to him to get an introduction. That’s how I got into security operations work in the tech sector.
Were you specifically interested in the security industry when you took the role?
Not really. I did it because it seemed decent pay for not very stressful work. I mean, it’s a stressful job when shit’s going down, but 95 percent of the time it’s a whole lot of nothing. It was a relatively non-stressful job. I’d go in, I’d do my job, I’d come home, and I got a decent amount of money for it.
When I got the job, I was just out of school and didn’t really have a degree. I had been working in retail, getting paid maybe nine bucks an hour. It was a 50 percent raise to go into security for a whole lot less work, so I was like, sure, why not?
Could you talk a little more about control centers? What is a control center?
Control centers are the central communications hubs in most companies. We’re the intake point of calls to “security” — someone in a regional or global control center will be the one answering the phone if anyone calls into security.
We triage those calls depending on their content. We have people respond on the ground if it’s a security event, dispatch 911 if it’s a medical or police event, or, if it’s minor, just document what occurred. Say someone calls to say they lost their cell phone: we’ll document it so it’s in our records, so if it turns up we can give them a call.
The job spans from pretty mundane customer service — hey, we found your phone! — to things like the building next door blowing up, disabling all power and causing damage to our building. That would trigger us reaching out to upper management to let them know something’s happened so they can start a crisis management process and connect to the site to make sure everyone’s okay, evacuated, and accounted for.
What does your job look like at its most mundane?
At its most boring, which is the vast majority of the time, there is nothing going on and we’re getting run-of-the-mill phone calls. When you’re on the operator level, you’re basically just staring off into space between calls. As a supervisor, I’m a little bit more flexible than some other people I’ve worked with — if nothing’s going on, as long as the work is getting done, I don’t care about anything else you’re doing. If you’re on the internet and watching YouTube, I don’t care. So long as everything else gets done, that’s fine. Just don’t fall asleep; give me plausible deniability if someone higher than me walks in the room. Everything else? I don’t care.
Some companies are more strict about that. Thankfully, the control room I work in now doesn’t have cameras in it, but that hasn’t always been the case — I’ve worked at places with cameras that look at people’s monitors. I’ve had someone at the management level call me and tell me they saw someone on their phone, and that I should discipline them. And I’m thinking, “Why are you watching the camera? Don’t you have something better to do?”
It could be pretty dumb sometimes, but when you’re contract, you have to keep up the appearance of being 100 percent on your A-game at all times, or you might lose the contract. I understand it on a conceptual level, but that doesn’t mean I like it.
Since I’m in a supervisory role, I now get to spend downtime on things that are partially outside of the day-to-day scope. I use that time to get projects done that better both the SOC (security operations center) or myself or the security department as a whole.
What about on the opposite end of the spectrum — what is an example of “shit going down”?
The most intense is when national or international incidents happen, the sort of things that you see in the news.
For example, I was on duty when the Boston Marathon bombing happened. We had a site in Boston, and when the bombing occurred, it was three blocks from our office, and we had people on the ground in the race. The whole city basically went on lockdown. People were stuck at the office and at home, and we had to make sure that everyone was okay, everyone was accounted for, and that our management was aware of the situation going on, how close it was the site, how many people were impacted.
Then a few days later, all hell broke loose and the whole city closed down trying to find the guys. That was one of my busiest days.
I was also on duty when the Mumbai attacks happened in 2008. That was when the terrorists came in on boats and started shooting up a hotel. Our office was a mile or two away from that. That was an almost two-day affair and we were updating the CEO every six hours.
Another example is that I was also on the job when the coup happened in Turkey in 2016. Thankfully, our office was on the Asia side and most everything happened on the Europe side, but half of our employees were on-site at the time. We gave them direct instructions to stay on-site until the coup was over. Don’t leave, stay accounted for. We accounted for everyone who worked at the office, down to the last person. The last person ended up being on vacation in an island just off the coast of Turkey, and we were able to get a hold of him too.
So yeah, there’s been a few all-hands-on-deck, nothing-else-matters moments where it’s very intense until everyone is accounted for.
Shoestring War Room
Are there any generalizations you can make about the background of people who go into this area of work?
The easiest generalization I can make is ex-military and ex-police. A lot of companies will specifically try to hire people from those backgrounds. I know a few people who were actually pushed out of positions they were in because the company wanted someone with military or police experience.
I also see a lot of students because the hours are pretty flexible. And then I see a lot of people who might not fit a mold specifically in a company, but still need to work — that was more or less where I fit in when I first started. I don’t really have a relevant degree to work with and was not getting considered by interviewers in other realms.
Once you’re in, you also tend to stay in. Security has a stigma, so once you have security on your resume, it’s hard to change into other industries. Interviewers will look at your resume and say, “That’s interesting, buuuuut you were kind of a mall cop?” No, no I was not.
Unfortunately, pop culture has given security a very poor reputation. People either think of us in an ultra-high-tech Las Vegas casino war room, or they see us as Paul Blart: Mall Cop types. I’ve never seen it depicted the way it actually is, although I’ve seen it come close on occasion. Don’t get me wrong; some people who are hired on the rank-and-file frontline are not the sharpest tools, all things considered. But it’s unfortunate because it ends up being looked down upon as a career path.
It’s funny because I had a picture of that casino war room in my mind as well. I’m picturing something like a big room with thirty monitors on the wall with a map of the US and glowing lights on it. What is it actually like?
It depends on how much budget the company has. At my current company, we have no budget since the nature of manufacturing is that it’s penny-pinching. Our control room fits four desks. Each desk has maybe two to three screens and we have eight big screens on the wall showing cameras or news or some combination thereof.
[E-commerce Company] was probably my fanciest, but I’ve seen fancier. [Social Media Company] has one of those ultra-high-tech war rooms. They have more money to burn than sense. [Big Company] has a fancy, twenty thousand dollar window where you press a button and the glass leading into the room fogs up. It’s all about budget.
That’s the other thing about security: security is always a cost center. It does not make revenue for the company, so you’re the last in line for budget — unless something happens, and that’s when the budget explodes.
So you tend to get more budget after an incident happens?
At one company, just before I got hired, they had a bomb go off at one of their buildings after hours one night. After that, they staffed every exterior door 24/7 with a guard, installed a ton of super high-def cameras at all the driveways, and built up their SOC significantly. It was basically like, “We never want that to happen again, here’s more money so hopefully it doesn’t.”
Is there a general preference for how well you are funded or treated as a contract worker versus in-house employee?
It pays significantly more to be in-house. Not only because you’re typically getting a higher base pay, but because the benefits are usually better. When you’re a contractor, benefits packages and everything are all dictated by the specific contract. So if a company you’re working for is the lowest bidder on the contract side of things, you’re probably getting crappy coverage unless the company is very nice.
This is also the first time I’ve received stock options, too. I got hired on with a pretty significant stock grant — or at least I think it’s significant. Obviously I know people who get that in base pay raises all the time. I’m not high on the scale, all things considered, but I’m up there.
Just Following Procedures
I’m trying to imagine what your org chart looks like. Can you describe your reporting structure?
The organization is typically broken down by region. It just makes life easier, especially when you have 150 sites worldwide. You divide everyone up — the three typical divisions are Americas, EMEA (Europe, Middle East, and Africa), and APAC (Asia Pacific). Then you just divide down from there.
Most control centers are in North America, so they will typically fall under the NorAm or Americas regional manager. Each center will have a manager who runs the SOC itself. Below the manager, there’s a supervisor. The manager will do more of the strategic side of things, where the supervisors are still doing the day-to-day stuff. I’m a shift supervisor, so I cover the day shift, there’s someone who does swing, there’s someone who does grave. I have a bevy of operators who work for me on my shift, and same with the other shifts. People in these roles report either up to the Americas region or up to a global security manager of some variety.
That’s from top to bottom. Starting from the bottom moving up, typically you start at operator, move into a lead operator, and then into management of the security team or a security niche within another team. When you’re an operator, your job is to follow the path through the procedure; when you’re in the lead, you have the option to bend the rules a little bit more, or to have a say in making decisions on things outside of the scope of the existing procedures.
Developing the procedures is usually a collaboration between the control center’s management team and the physical security managers of the spaces they’re involved in. So if there’s a certain site that has certain requirements, like OSHA requirements, those have to be built into our procedures. For example, if there’s a medical event where someone lost a hand, has 911 been contacted, are we handling it, is the patrol team on-site handling it, is it some combination thereof? That all goes through a long committee process over time.
When you mentioned strategic work that is done at higher levels, what is an example of what that means?
Sometimes, it involves networking into other teams. We communicate what our wheelhouse is at the moment, and we think about how we can make other teams function better or have other teams help us improve. Other times, it involves reaching out to stakeholders, outlining the communications we send when incidents occur, and understanding: do they want to be involved in those communications, is there any information they need that we may not be including now, that sort of thing.
The long-term strategy is just to prove, and to increase, the value we provide to the company. Like every other department, if we’re not adapting and changing and building better, we’re just costing the company money.
What are the other sorts of niches or connections to other teams?
Myself, I’m trying to go into the technical side of things — the side that involves the cameras, the card readers, the systems and the design thereof. I’m currently working on a project with a tech team collaborating to improve systems for both their team and my own. The company I’m at was previously run cowboy-style where everyone was doing their own thing, so there’s seven different systems for cameras, there’s five different systems for alarms, and there’s no unity to it. Decisions were made site by site, or regional manager by regional manager, or whatever the budget allowed.
As they get to higher levels, a lot of people will go into the analysis side of things. Most security departments will have people who investigate larger incidents using some form of intelligence analysis. So security will look into some of the smaller, nickel-and-dime stuff, but it’s the investigation team that comes in when it’s big money losses, sabotage, or IP theft on incremental levels, where it’s someone who’s been taking out a little at a time for a long time.
Travel security can sometimes be a separate team. They make sure everything goes well when people are abroad. That can be looped into a control center team, though it’s not always. They’re also the ones who will look into best practices for higher risk countries like China, for example. Don’t bring your work phone, bring a burner phone; don’t bring your laptop, or bring a laptop you don’t care about. You need to have a VPN that works in that country; you need to have an anti-USB keylogger adapter system; you need to have one of those special power cords that only passes power through, so you don’t get any viruses when you plug your cell phone into a wall charger that was provided to you by the hotel.
Did the different companies you worked at have different levels of paranoia about that sort of thing?
When I worked at a non-customer-facing hardware company, they weren’t too worried about it. Obviously, they had IP theft concerns, but they licensed their stuff out anyways.
Manufacturing tends to be more super secret about it than the tech companies I was at. They are a lot more fearful of intellectual property disappearing and then being duplicated because it’s literally their livelihood.
When I worked at an e-commerce company, there wasn’t much IP to be stolen. The worry was more about user information getting out.
You mentioned that you’re interested in getting more into the technical side of things. How do you currently use technology in your role, and what kind of tools are at your disposal?
A lot of companies use a program called Lenel, which is made by a company called OnGuard. I’m well versed in Lenel because it’s something I’ve been using for almost twelve years now. But it’s getting a little dated and other companies are starting to branch out into other things. There are also about a half dozen different camera systems.
I’ve done surface-level programming in these sorts of systems: generating alarms, naming those alarms, timing them for certain things.
Alarms can tell you many kinds of things; one door can generate a dozen different types of alarms. They’re usually either technical or security-related. If a wire is not connected properly, it’ll generate an incorrect voltage alarm or an intermittent disconnection alarm. Those are technical alarms, and only the technical side of the organization really needs to know about them.
But you also have security-related alarms, like if a door is forced open, if it’s held open for more than thirty seconds, or if it’s an emergency exit that has a screamer on it. Or maybe it’s a glass break alarm, a duress alarm, or someone is pressing the button that says, “Oh my God, there’s a guy with a gun in here and I’m pressing this button, so hopefully someone looks at the camera and sees the incident going on without triggering the guy with the gun.” Obviously, my team needs to see those.
For the most part, no company uses any sort of in-house technology because they don’t want to spend the money on it — because it would have to be developed, it would have to be serviceable, it would have to have more than one person who knows how to fix it. So it’s always farmed out to other software systems or contract companies.
At Your Service
How does the work that you do relate to security guards? Is that in the same organization? Or do security guards work for different companies and you coordinate boots on the ground?
If the SOC is contract, security guards are typically part of the same contract company. It depends on the site and company. Some companies will cherry-pick contract companies based on their strengths — one company for on-campus security guards, one for travel security, etc. — and other companies will pick one contract company to cover everything. In general, if it’s a small enough company, the SOC will also control the biggest site, or direct the local site’s guard force.
The guard force is partially autonomous — unless otherwise given instructions, they do their own thing for the most part. Then when something happens, we act as an override on top of their typical procedures. That can be as small as needing them to go unlock the conference room for someone, to having this belligerent person in the lobby and making sure everyone responds to make sure he’s escorted off the property. If there is a medical incident, it’s making sure they respond to that — once they are there, they have their own process and procedure they follow. Or when it’s a security event, or heaven forbid an active shooter — which thank God I’ve never dealt with — we direct them and they take over.
So your job is to deal with everything from active shooters to people locked out of conference rooms.
This job is in part an over-glorified add-on to an insurance policy. In essence, your being there makes their insurance cheaper. But it’s also in large part a concierge service. Security is customer service before it’s actually security. We’re there to say yes more often than we are to say no. With manufacturing it’s a bit of a different story, but with tech it’s 100 percent, we’re there to help people get what they need, more than tell them they can’t do things.
Huh. When I think of infosec at least, that emphasis on customer service seems like it could be a vulnerability since social engineering is a big threat — so being overly-friendly to people could lead to malicious actors trying to take advantage of you. Have you ever had to be rude to people?
Not to sound terrible, but having worked at the control center — which is all just customer service over the phone — those times when I could say no was a glorious thing. It felt great for once to be able to tell someone, “No you can’t do that, fuck off.” But it was a very rare occasion, and it had to be like a pure breach of protocol. We could bend things every once in a while, but not break them.
We are customer service to a point where we’ll help someone get to the information they need, though we won’t necessarily give them that information. And obviously understanding social engineering from a basic level — to not just volunteer stuff — is important.
Tech is a lot more customer service because, even though it’s all on a server somewhere and it’s easier to sneak stuff out because it’s a lot smaller, there’s very little we could do to prevent things that aren’t already dealt with by infosec, or with all the stuff that’s built into your laptops. Manufacturing is a lot more keeping track of material stuff, because every ounce that goes out that isn’t in the product is basically money lost.
When I was working at an e-commerce company, whenever sellers called in because they were experiencing a technical problem with the site, it was typically their livelihood on the line. It was like, “Oh my God, my store isn’t working, and if it doesn’t work in the next forty-eight hours I’m going to go kill myself.” I had to call police departments in more countries than I care to count, since if someone says they are going to kill themselves, it’s a liability and we need to make sure they’re okay. Or at least do the due diligence of reporting it, so that it’s on the police if they don’t actually go and do anything about it.
Beyond answering phones I assume you’re also monitoring security cameras. How much of the job is that?
People always assume someone is watching a camera at all times. I can 100 percent say no one is watching a camera at any time. We might have some up for peripherally keeping an eye on trouble spots. For the most part, though, the cameras are recording at all times, but no one is ever watching them.
Once you get past a certain number of cameras, you are past the point where it makes any sense for someone to be watching all of them; it’s 100 percent looking back retroactively and seeing what exactly happened.
On average, at my current manufacturing company, I do maybe three to four of those investigations a day. And the previous tech companies I worked at, it was maybe three to four a week depending on the scenario. It was kind of feast or famine with tech. With manufacturing, they care a lot more, because they’re keeping an eye on when people screw up, when things break, or when damage is done — because every penny counts.
So in manufacturing, beyond security and managing the infrastructure or assets of the tech company, it sounds like there is also a level of surveillance over the work that people are doing?
In a way. They have cameras on the production itself. It’s more a matter of making sure that if there is damage done, the person who does it is held accountable, and it’s dealt with in a timely fashion. Because they need to investigate — was it just an accident? Was it malicious? That sort of thing. If it is malicious, they deal with it rapidly; if it’s just an accident, the safety team is there to determine how it happened and how to prevent it in the future.
The degree of surveillance differs a lot when you’re dealing with the US versus Europe. In the US, we have a lot more liberty in terms of what we can watch and how we can document things. In most of our US sites, we have high-def cameras at all corners of the building, at all entrances, and cameras that even look out into streets in the area.
For a lot of the European sites nowadays, with GDPR, those cameras are required to be blocked off or blacked out. They cannot have any visibility into public space or anyone’s workspace. We have limitations on how much can be recorded and how much an investigative body can share with us when they’re requesting access to our footage. We’ve had cases where the police will reach out to us and say, “We had an incident here, we want the video from X time to Y time, and we can’t tell you anything else.” And we say, “You realize three days’ worth of footage is like ten gigs of video file, you sure you want this? ‘Cause we could narrow it down really quickly if you told us…” And they’re like, “Nope, we can’t tell you.”
After applying all those rules, what can you record? Where are the cameras pointed in Europe?
The cameras are typically on entrance doors. If it’s an exterior camera, it’ll be looking at just the door; if it’s an interior camera, it’ll typically looking at just the door or the hallway leading up to the door. They’ll be looking at lobbies, elevators, stairwells, and that’s pretty much it. Maybe server rooms or networking closets on occasion.
So any places where people are entering and exiting, or enclosed spaces that are meant to be sensitive.
Yeah, high-risk zones basically. That’s actually one of the weird things having come to manufacturing from typical Silicon Valley companies. Most companies will put cameras in places where the highest volume of OSHA violations or OSHA-like issues could occur. As I said, stairwells, hallways, elevators, and entry doors are the big places accidents can occur. Manufacturing companies are looking at the process on the floor; they could care less about stairwells or elevators or anything else. Where there are machines, there are cameras.
No Can Do
What kind of interactions do you have with other people in the companies you work at, if any? You sometimes hear about the different badge colors at some organizations and there being a kind of caste system, I’m curious if you have experienced that.
There is some kind of caste system, obviously. Most companies see their own employees as the top line, and then everyone else is down the chain. Most of the service-level stuff gets contracted out — janitorial, cafeteria, security. Facilities is a big one that’s typically contracted out. We work directly with facilities fairly regularly. We’re the ones who will find all the issues with the building and let them know when glass gets broken, or ceiling tiles are falling out, or there’s a water leak somewhere, that sort of thing. We work pretty heavily with janitorial as well.
But we treat everyone pretty equally, unless they have a C in their title or a V in their title. For the most part, everyone else is stuck following the same exact policy and procedure. There are no exceptions. There are some policies that are employee-specific versus contract-specific — who has physical access to what areas of the building, some contractors can only work eight to five Monday to Friday whereas employees have 24/7 access, that sort of thing. But for the most part, everyone else is treated relatively equally.
Do you see any difference in how you as someone on that team are treated?
At best, security tends to get ignored, and, at worst, we get sneered at or yelled at. Because we’re the ones saying, “No, you can’t do that — it’s against the policies and procedures that you signed up for when you joined the company and you have to follow the rules.” I’ve had many occasions where someone’s like, “Well I’m important in this company and it shouldn’t apply to me.” And I tell them, “Sorry, you’re like everyone else.”
My running joke for engineers is that you could paint a door red and write “You will die if you walk through this door,” and certain engineers and still walk through the door. Many times, people will walk through doors that literally say in big giant letters “Do not walk through this door. Audible alarm will occur.” Then we get angry calls from people who sit at the desk near that door. That’s pretty common.
When those cases happen, is it that people are clueless, or they think rules don’t apply to them?
A little of column A, a little of column B.
How these incidents get treated depends on the company. In manufacturing, when someone breaks a rule, our job is to reach out to the person and their manager and have the manager enforce that they never break the rule again. In tech when someone breaks a rule, our response is usually limited to reaching out to the person and saying, ”It would be nice if you did not do that again.” The previous two tech companies I worked at told us to be very, very nice when we told people these things, versus in manufacturing, where we are encouraged to lay down the hammer.
I would say 20 percent of the time, we can get people who are understanding of it and will acknowledge it. But the other 80 percent are like, “But I need that, I need to do this thing now, it needs to be done now, let’s do this now, why can’t we do this now.” And we’re like, “We can’t because it’s against policy, sorry.”
Are there escalation paths for things like that?
We can bend rules on certain occasions, but it’s context-specific. Is this person an intern or are they a vice president? It literally is dependent on: are they in the upper echelon of the company, or do we not care about them at all? They can escalate it all they want, and if it escalates far enough we might make changes to the policy. But most of the time, even if it escalates up to our senior management, the senior management will go, “Well, they followed the procedures to the letter. Everything’s good.”
Caught on Tape
What are some of the things about the job that you think would be most surprising to people outside the industry?
The biggest thing I’ve come across over the years working security is how much people think they can get away with things, or operate under the assumption that there aren’t cameras in an area, or the cameras aren’t recording. Which, to be fair, that’s sometimes the case. But 99 percent of the time, if they’ve done something stupid, it’s probably on camera. A control center operator has probably seen it or has been asked to look into it and found it.
To give another example of how much info we have: any time there’s a card reader and you tag on it, it generates a “granted access” or an “invalid access” code depending on if you have access or not. So if there is ever a time where we need to find out where someone is, we can trace their badge and see, oh, at 8 a.m., they badged here, at 8:20, they badged into this conference room. That’s the last badge-in they have and it’s 8:30 now, so they’re probably in that conference room.
This typically comes up during investigations, anytime we’re trying to track someone. For instance, if we have a high-risk termination, HR will reach out and let us know that someone is high-risk because he’s made threats to his manager or something. We’ll then be asked to track him and let them know when he comes on-site, so we can make contact immediately and have security on standby in case something goes wrong. In cases like that, we can trace his badge, set it up so we get a ping when he comes on-site, and bring up cameras to track his movements.
But back on the topic of getting away with things: I’ve had times where we’ll see an incident occur in a parking lot. Someone will ding someone else’s car and then we’ll see them park, get out of their vehicle, and walk into a building. From our perspective, we know which building entrance they used and what time they walked in. We can run a trace on that door, look at that time, see the person, look at their badge picture, and call them up. This is a pretty common occurrence. We can also reach out to the victim because we can go back, watch them park, get out of their car and badge into the door, and see who they are too.
There are always occasions of petty theft, where people who make six figures will walk up to a desk, see a phone charger or a laptop on the desk, and just walk away with it. It’s just like: why? You have a perfectly good job, making really good money, and you’re going to steal someone’s phone charger or their laptop or their cell phone or their wallet? Why?
And then on the flip side, one of the first things that happened at one job was someone tailgated into the parking lot — a non-employee who was wearing a lanyard with a convincing-enough badge. They went upstairs, walked up to someone’s purse, took the credit cards out of it, and within an hour had charged five or six thousand dollars on their cards. They did that two or three times, at multiple companies, until they finally got caught.
I’m always surprised by the cleverness of the people outside the company who try to steal things, and the stupidity of the people inside the company who feel they can get away with things.