When did you first start learning to program?
I’ve been programming since I can remember. I grew up in Israel so I couldn’t actually speak or read English, but I could write elementary BASIC code.
What was it like not knowing English when you were learning BASIC?
I memorized the commands in Hebrew, which were overlaid on top of the standard QWERTY English keyboard. So to draw a circle I spelled the word ’circle’ in the Hebrew letters. It’s not an actual word, obviously. I didn’t recognize or understand English letters.
I’ve been reading fluently in English since I was eleven or twelve, and learning English definitely helped me learn to code because I could recognize the shape of the letters and something about their meaning earlier than the smarter kids did.
Do you feel that the prevalence of English in coding languages is a barrier for non-native English speakers?
I stopped using Hebrew when working on computers when I was fourteen or fifteen, as soon I had access to control my own computers. Growing up, I used the Hebrew version of Windows, which is built to support right-to-left languages and tries to translate. But, supporting both Hebrew and English languages is really hard and is never done well, and so I gave up on Hebrew as a working language at a very young age.
So, the short answer is yes, absolutely it’s a barrier. And the solution is that most of us just give up. My phone and computers have been in English for about as long as I can remember, because the experience is horrible otherwise.
When you were young, what did you think that you wanted to do with your life?
I was very, very certain until my late teens that I was going to be a doctor, like a good Jewish son of a good Jewish mother. I have some medical challenges though, and when I joined the Israeli Army for my mandatory service I was assigned to be a truck driver, which was less cool.
At that point, I had already developed some level of programming knowledge. I remember an interesting discussion with my HR non-commissioned officer, where I expressed discontent with being a truck driver. She tried to explain that it’s a fantastic gateway to a career in life, and I—maybe slightly rudely—expressed that as an eighteen-year-old, I had slightly higher aspirations than being a truck driver.
And so I failed the truck driving class. I failed it horribly.
Was that intentional?
Yes, very much so. Somewhere in my files, I still have that rejection letter, which more or less says that I am too stupid to drive a truck. It’s one of the things that I’m most proud of.
That got me out of that black hole and into the general assignment. Then I mentioned that I could code and they made me a small computer operator in the Salary Analysis unit of the HR division of the Israeli army, which is now called the Manpower Directorate.
This job started on an actual terminal, which is basically a keyboard attached to a printer. You typed blind, hit enter, and got a response, which is usually something like “error on line three.” Then you have to guess what line three is, because the end of line submits a query so everything you’re typing is on one single line. So, to debug, you just type it over and over again.
The language used on the terminals was a Hebrew query language called אמת3, or EMET 3. In Israel, speaking Hebrew has a nationalistic value. It’s a revived language that was part of the creation of the country as a country. So a lot of the motivation behind the creation of אמת3 was that it had to be in Hebrew.
The query language was written in Hebrew but compiled into English—because everything is compiled in English. Brackets had to be reversed because figuring out how to write brackets in reverse does not work in that compiler installed in pre-Unix VMS. They used the ASCII code that corresponded with the letter that was on the keyboard when you typed in Hebrew, and while compiling reversed the Hebrew letters into those ASCII codes. It was essentially gibberish English, basically what I did when I was learning BASIC. The language was written in 1973, and I started working on it in 1999.
But, that was my life. You’d get a question like, "How many people served between 1998 and 2000 as nurses in this unit? And how much money did they get paid?" And then they would run an analysis to get a sense of what kind of modification to their salary they should have because they were practicing emergency medicine in a medical combat unit. My job was to run those reports for them, because translating from the logic of the query to that stupid language was hard.
Like every other piece of legacy software, when you work with it long enough it shapes the organization around it. Workflows are created to handle the limitations and people like me start building bits and pieces on top of it to make it better so that the workflow can become better. And what you end up with is a whole bunch of legacy code held together with duct tape.
How did you feel about your job at the time?
Initially, I was very grateful because I wasn’t driving a truck. Being the clever guy who can do these reports when everybody else gets stuck was exciting for a while. But then I got tired of doing the same thing over and over again. You just have to copy and paste, change the dates. It gets very boring.
Outside of the desire to do interesting things, did you feel any responsibility to your job or a sense of civic duty? I’m curious what motivated you.
That’s a discussion that’s really interesting for my current job. But, for that job? You need to remember that I was eighteen and serving in the Israeli army is a complicated thing. Everybody has to do it. Caring about the people that I was affecting by supporting the system clearly isn’t something that eighteen-year-olds do. Or at least, I didn’t at that age. So, no. I was there because it was fun and interesting and it didn’t involve driving a truck.
Now, it’s very different. My current work affects five million federal employees and two and a half million military units. What I’m doing directly improves their lives and improves the life of the people who serve them, and that is awesome. That’s why I’m doing my job now. I don’t think that any of those considerations existed back then.
What made you leave that job? Where did you go next?
I moved from the Salary Analysis unit, to the HR division IT. At the end of my service, I was given the opportunity to work on a tool that was supposed to remove the need for anybody accessing or using אמת3 ever again. I was so full of resentment of that system that it sounded like an awesome idea.
The first RFP for that project came out in 1981. I was born in 1981. In 2001, I was essentially the tech lead on it. The intention was to wrap this system in a modern—2001 modern—UI system where people could go into a graphical interface, type in the ID number of the soldier or the query, and get the results they needed to make their decisions. I decided to volunteer for an extra six months of army service to see this project through.
The workflow that we're trying to solve is the following: we have something called lonely soldiers in the Israeli army. They are soldiers who usually are American volunteers. Their family does not live in Israel. They get housing, food, and extra money. For the record, when you join the Israeli army, as a soldier, you basically make twenty-four bucks a month. You don't get paid for your service for the first three years. But, lonely soldiers get a bunch of services and more money because they need to eat.
So, let's pretend that a lonely soldier is in a combat unit on the border somewhere. When they submit their application for benefits, there’s a couple-months-long process of passing paper up the chain of command, at the end of which, someone runs a bunch of queries and updates the record in the mainframe system. Now, let's pretend that there’s a typo in the form being passed. That form would then travel back another two and a half months to get back to the soldier to be corrected.
What we wanted to do was to shorten that process by offering a graphical interface accessible to the local HR officer so they could submit the applications form electronically and have it approved by the central command HR staff. That was our goal. Take a six month process, turn it into two weeks. To make that possible, we have to build that UI system, wrap it around the actual mainframe system, and make the connection.
Now, that is a fairly common modernization approach to mainframes, but it’s really not modernization. Today, I am generally angry at this approach.
How come?
Because what I’m doing now is cleaning up after people who made that decision twenty years ago. I’m also twenty years older and I understand technical debt a lot better now. Wrapping a UI around a legacy system doesn’t pay the technical debt. It adds more technical debt. My job now is literally cleaning up after that situation.
There’s a bunch of parts of the industry where people basically make their living out of building wrapper systems around mainframes, because it means that, despite whatever is currently running, you appear to have a shiny new modern system six months later. But you don’t, not really, because you can’t actually change the underlying workflow in any way—which works for a while, until you have to change something and you can’t.
Making Moves
Where did you go after the army?
I got to New York and as I was about to head out on a month-long cross-country roadtrip, a relative of an army buddy of mine said he needed a website to write quotes for his moving company. I said okay, I’ll work for a couple weeks and make a little bit of money and I can do a two month long trip... And that was nineteen years ago.
Eventually, around 2010-ish, I got pulled into fintech by somebody that I was working with on the moving side, introducing me to the idea of taking programming algorithms, hosting them on a stable system, and running it.
Essentially, the foreign exchange (forex) market is ginormous, and has many, many, many, many traders, with each of them trading tiny amounts with insane leverage. It’s a very, very bad industry, morally speaking. There’s a tool called MetaTrader, which was the primary go-to tool for retail trading in the forex industry—this is a Windows application running thirty-two-bit code in 2016. You wrote your algorithm and you ran it on this application on your desktop. If you had a power outage or one of your kids came over to play a game and closed your application, you would lose money as a result of it.
Fairly early on, we came up with the concept of taking a machine and running it on a Linux server using Wine, to make things more stable. I ended up doing a lot of work with Wine to make it run this trading algorithm properly online. Today I would just do this with virtualization or containerization, but since we’re talking about the late 2000s, none of these were options. I spent the next four years getting that up and running, having up to 11-12,000 people using the system at some point, and then selling that. That was the first and only software that sold successfully. It was my primary success in fintech.
Then, the company I worked for got bought by a Slovak conglomerate of foreign exchange brokerages. After buying our company, they just kept buying more and more brokerages. Each brokerage would hand us—hand me—their technology team and their technology stack. We would work to bring their team into our environment, taking their technology and combining everything to create a unified service. We became a technology service company.
As we bought new companies, we became really good at wrapping around other people’s projects—regardless of which stack they were using—and getting them to talk to each other. We’d take complete projects, wrap them, and turn them into services supporting our unified frontend and unified APIs. Over time, we got into refactoring stuff and breaking them into different pieces and we eventually started building lean microservices, running on AWS, things like that.
Now that I’m talking to you about this, it’s very, very sad. The story of my career is that I absorb other people’s legacy software and then kick it until it becomes useful again. Which is, to an extent, what I’m doing now for the federal government.
What was your experience transitioning to the public sector?
Well, at first I went looking for jobs at nonprofits as opposed to government because government is—in my head, at least—morally ambiguous. But non-profits don’t like people who don’t have degrees. Government doesn’t like people who don’t have degrees. Anywhere else in tech, nobody cares about what certificates you have; people care about what you’ve done. You only need certificates or degrees if you haven’t done enough yet.
I spent two and a half years trying to find a way to transition my career over to something meaningful. At the end of the day, the problem is that the not-for-profit jobs are few and far between. I applied for dozens of jobs. All of that failed miserably. My search for more morally fulfilling work ended up taking two years.
Wow, two years is a long time.
Eventually I discovered the United States Digital Service (USDS) on Twitter. I applied to USDS and I went through their very tedious six-interview process. Apparently I passed that. But, because I’m a former officer in a foreign army and they do most of their work with the Department of Veterans Affairs and the Department of Defense, they felt I would never pass their security checks and decided not to hire me.
But there’s a circle of government technology organizations where they all know each other and work together—USDS, the General Services Administration (GSA), Technology Transformation Services (TTS) which includes 18F, etc. They forwarded my resumé to them.
I interviewed for GSA and then waited three months to hear they wanted to hire me. And then, three months after that, they told me I’d need to start the background check process. It took something like eight months altogether from the time I interviewed for GSA to the time I started to work for them.
I’m interested in how you feel about that government hiring process. My understanding is that one of the reasons why there’s an insistence on pedigrees or a specific number of years of experience is to try to level the playing field. For example, testing someone on specific topics in an interview can privilege certain backgrounds and it’s easier to let bias slip in. But, if you look just at years of experience, in theory that will make it a bit more equal. That doesn’t account for your experience with needing a specific academic pedigree, though.
Yeah. I understand the need to try to remove bias from hiring. But access to higher education is a biased measurement. And in tech, it’s also kind of irrelevant to skill.
One of the things that I have done many, many, many times is hire people who have just come out of their bachelors or masters programs and then spent six months paying them salaries while I’m undoing what academia did to them. People come out of academia with an interesting theoretical background. But honestly, let me ask you, have you ever done Big O analysis during your professional work as an engineer?
I’ve had to think about it, but not deeply or often.
Exactly. You think about complexity and you think about efficiency and you measure it, but Big O analysis as an interview tool is only useful to measure how much attention you paid in your second year of college. It’s one of my pet peeves. It’s a symptom of a disease.
Nobody Puts COBOL in the Corner
As I understand it, you’re not currently a permanent employee with the federal government. How would you describe your position?
My position right now is a termed position. Technically, in eight months, I am supposed to go back to the private sector and move on with my life. I probably won’t because this is what I want to do with my life. So I am going to fight to find a way to stay beyond my term. For the government, every change requires years.
As an example, take my current effort to replace a specific mainframe overseeing the benefits and livelihoods of two and a half million people. Getting away from that system will take more than five years, because of the amount of technical debt that has to be paid.
We have led a successful acquisition of a mainframe that took three months and we’re very proud of that. Usually it takes more like seven months to finish an acquisition process in government. We bought a new mainframe, which is a horrible thing to say, but that was the right thing to do because the old system was really about six, seven years past the end of life, and we needed five years to pay the technical debt to fix the system. So, we had to buy a new one and we did it in three months.
That’s what I think makes the GSA’s Centers of Excellence (CoE) so special; because the CoE is a part of the federal acquisition service, we can cut that time down to get a new system. But there is value in people like me, people who are angry at the way government works, sticking around for five years, for ten years to see these things through.
Right now, government is shortcutting the hiring process with termed employees like me who have short two-year stints to try to make change. I don’t think that’s effective because you need someone to stay and see it through. Especially for an organization like the federal government where, every four years, regardless of what happens from the mission perspective, leadership goes away. Targets change. To create sustainable change, we need angry people in the same chair for more than two years.
What is a termed position? And what are the other positions available in government, whether salary or contract?
The federal government has four positions that I’m aware of. You have general service (GS) employees, who are salaried employees who work for the federal government. They’re somewhere between GS-1 and GS-15, which is a salary scale, and they are competitive service positions. Essentially, that means they have the ability to apply to any position within government, compete for it, and get it as a long-term position.
Then there are people like me, termed service. To make my position valuable, they let me slide into GS-15 step 8, which is two steps below the maximum salary of GS-15 step 10. They try to match our salaries from the private sector. They couldn’t—they could only match two thirds of what you make in fintech in New York. But they tried, which I appreciate. I was completely aware that I was going to take a salary hit by switching.
In my case, the termed position is two plus two, meaning I have a two-year term and would be up for renewal for another two years. They can fire me within the first three months and they can decide whether or not to renew my term at the end of the first two years. Terms can be anywhere from a year to eight years.
Termed employees are essentially civilians—I can’t apply for other competitive positions outside of my current role. When I’m done with this role, I have to apply again to a GS position and all of my government experience doesn’t count.
When you say it doesn’t count, do you mean toward the qualifications for other positions or for benefits?
All of the above. That experience essentially doesn’t exist. If I apply and get into a permanent position that is open to the public, which is some small percentage of the perm job positions, it could contribute to the pension calculation, if and when I retire. But that’s about it.
It also doesn’t count for experience for other positions. My position right now is nonsupervisory. For accessing the third kind of position, you have to have a year of supervisory experience. I can manage as many people as I want right now, it doesn’t count. I have to start from scratch from that perspective.
The third kind of position is Senior Executive Service, or SES. Most people with senior titles—CIO, CEO, directors—tend to be SES. There is a subset of management criteria that you have to reach to become SES, and you also have to have the supervisory experience as a competitive employee that I mentioned earlier.
My goal in life now if I’m going to state government is to become SES. But I can’t even start working towards that until I am rehired. Practically speaking, from a career perspective I may as well be a civilian right now.
Why is your goal to be SES?
Because whichever system I’m in, I require myself to accomplish the highest level possible. Right? Also, because that’s where change is. I’m trying to effect change.
The more you spend time in technology, the more you understand that technology is not the problem. I can build the shiniest system in the world, but it doesn’t mean that anyone will use it. Change is always about people, and people are controlled by management.
For example, if I want to get an agency to start working in a different way, I have to be near enough to the top to be able to make decisions that affect how people function. Being at the top doesn’t make it easy, but it gives you the option of trying to accomplish it. This is kind of the sad thing about technology—the more you look into it, the more you understand how little the technology means.
We haven’t yet talked about contractors. From your perspective, what is the role of contractors within these teams and systems? And why does government often procure or contract technology versus building in-house?
That’s a really complicated question. I mentioned there are four classifications, right? Termed, GS, SES—and contractors are the fourth one.
It takes me somewhere between two to three months to get a contractor in the door, whereas a new hire takes about eight months to a year. So as a product manager, if I’m trying to get something done in government, contractors are usually my go-to solution.
The other piece here is that for political reasons or other reasons, hiring freezes are a thing in government. A bunch of agencies have had eight to ten years of hiring freezes and, as a result, have basically gotten to the point where only 20 percent of the workforce are federal employees. Because they can’t hire, they just buy more contracting because there’s no monitoring of that. That is actually fairly common.
As an example of one of these freezes, say a lot of the mainframe support people across government started to work as federal employees, but then the federal government can’t hire any new ones. The federal government will go out and buy contractor support to help maintain this legacy system, and the contractor support will turn to the salaried employee and say, “we’ll pay you 150 percent of your salary and give you better benefits if you come work for us.” Federal employees in those positions get to essentially keep the same job, but have better quality of life as a contractor.
The law says that federal contractors can’t manage contractors, so a federal employee has to be the contracting officer representative who manages that contract. So you have people whose job used to be technologist or system administrators, but now they are federal employees who are middle management and as a result all they do is manage those acquisition contracts. We’ve seen this pattern a lot.
How does that impact the maintainability of software?
Theoretically, it should improve it because every single person is replaceable. Practically, it presents interesting challenges.
For example, VB6 was decommissioned in 2012, I think, but there are many VB6 systems across government because VB6 presents a fairly low barrier to entry for development. There’s a whole bunch of these lying around.
Say I have a contract with the vendor and that contract says I need VB6 experts to manage these VB6 systems. Awesome. The contractor gets to move those experts around, and they’re supposed to be interchangeable. But what you learn from managing technology as opposed to doing technology is that knowledge management is really hard to do. Organizational knowledge dissipates even if we get a new VB6 expert, because that expert has never seen this system and the person who was an expert in this system gets moved to another project.
Documentation is hard—and if it doesn’t exist, I end up paying the same contracting company over and over to learn the business logic and reasoning behind a specific system. That’s the flaw of this approach.
To your point about people, COBOL and mainframe systems have been in the news lately as state unemployment applications struggle under the historic load of applicants from the pandemic. A lot of folks are demonizing the languages, but my opinion is that it’s not just about the language or the technical systems—there’s an important social component, too. What is your take on that?
I actually got into a very angry discussion with a few of the protectors of COBOL. I described what happens to COBOL in its lifecycle. Every language has a lifecycle for large systems. COBOL has a very specific one.
Mainframe systems have one primary benefit, which is a horrible trap: you can set the system up, leave it alone, and it will keep running, forever. What you get is forty-year-old codebases that keep doing their job and management doesn’t bother paying for it. Why would you? I can get away with paying less money and the system will keep producing, so I don’t need to bother maintaining it. But then you have code that was written forty years ago by people who have since retired.
When the time comes to make a change, you see one of the following. I’m looking at a system that had 900,000 lines of code. It was obviously not man-made. Who made it? Multiple times in the past, IBM introduced new hardware. To make the migration to the new hardware easier, they built tools that would automatically rewrite code. So I have a piece of code that was a hundred thousand lines long to begin with that has now been rewritten and rewritten by other pieces of code. It works, but now there’s no documentation to speak of. There is no understanding of which piece of the code does what. And what I have is people—and they’re really good people—who’ve been maintaining it from the edges, writing around pieces of code that are too big to change like I did back in 2001. But the system is still there. And so you have massive systems that are completely monolithic. There is no understanding which piece of the code is what, or what I can do to kill it or make it better. I have no way to maintain it.
And then something happens like COVID-19. Suddenly my system that can handle x amount of calls per day needs to handle 400x, and I have no idea how to approach it because that is part of the underlying system itself. It’s not an edge problem.
I’m not saying that COBOL is a bad language. I’m saying that it’s so good that it gets abandoned over and over again.
May the Pipeline Be Unbroken
One of the things we’ve struggled with in this series is finding a diverse set of people who have held technical roles with mainframe systems for a long time—and are willing to speak with us. So, given the breadth of your career and where you are now, I’m curious if that lack of diversity matches your experience or if it’s just a problem with our sourcing?
I think it’s an instance of the fake “pipeline problem” for people who are not white men into technology roles. I call the pipeline problem “fake” because I don’t think that’s the actual problem. General technology in GSA has the largest number of queer people, the largest number of women I have ever worked with. For people who shoulder the joys of unpaid work as part of their lives, government does what is actually necessary to make it feasible for those people to stay at their jobs.
For people still maintaining these systems, though, you’re looking at the pipeline from thirty to forty years ago. There are new people who are pouring into the larger pool of workers, but the mainframe teams haven’t changed. That’s the way it was forty years ago and that’s the way it is now, because these are the same people.
What we’ve heard from people maintaining these systems on under-resourced teams is that they’ve been basically expected to be on call for their entire career. One of the things we’ve been trying to understand is: What impact does it have on their personal life? What you are describing gives some color to that, as well.
Ah, that question brings me back to the joys of working in fintech. Foreign exchange is open 24/7. The market opens Sunday at 4:00 p.m. and closes on Friday at 4:00 p.m. and is open 24/5 in the middle. This means that if you’re in charge of the system, you’re on call during the week because you have to be there if things break, and the weekend is when you do all your maintenance. So, you just work constantly.
In terms of the impact on people's personal lives: you really shouldn’t talk to me, you should talk to my partner and ask him. The fact that I can’t make plans, the fact that I have to walk out of dinner with his parents and go deal with work calls, I’m sure that has an impact. And yeah, people working maintaining government systems have been doing that sort of on-call work forever. That is what they do.
By the way, these people that are on call all the time, the same people who weren’t willing to talk to you when you asked for an interview: part of the reason they refused is that they are still GS-12s and 13s. They’re low in the hierarchy, and yet so essential. Let’s say you publish something and it somehow comes out as negative about GSA. In my position, I would get a slap on the wrist. If they said anything that they shouldn’t say about their agency however, then they’re done. That is their career. So, I have a kind of privilege; I’m coming at this from a position where I can talk to you and I’m not risking that much. These people who’ve been here for forty years—and they’re completely essential—know for a fact that they’ll be railroaded if anything happens.
I'm sorry, I'm not trying to guilt you. I'm trying to expose the dynamic.