Time and again during the COVID-19 pandemic, Federal IT has proven its value as a vital service-delivery lifeline between government and citizens. MeriTalk is chronicling the untold stories and lessons of Federal IT success as the nation takes its first steps toward recovery. In this chapter of CIO Crossroads, we explore the State Department’s accelerated IT modernization journey in the midst of the storm.
State Turns to Cloud, Browsers to Jumpstart IT Modernization – CIO Q&A
Federal agencies don’t get any older than the venerable Department of State – created in 1789 as the first Executive branch department and charged with advising the president on policy and conducting diplomacy on the nation’s behalf. With a presence in more than 160 countries, few agencies are responsible for maintaining such a far-reaching network around the world.
With the COVID-19 storm gathering on the horizon in January, long-standing traditions at State regarding how to conduct business – face-to-face meetings in offices – were put to the test. In an exclusive interview with MeriTalk, Principal Deputy CIO Michael Mestrovich tells the remarkable story of how the agency quickly shifted from telework “in the hundreds” to remote capabilities for State’s entire 107,000-person workforce.
By deploying a combination of browser and cloud technologies that gave the workforce access to Microsoft 365 and a great majority of the agency’s internal applications, State scaled robust remote capabilities to the workforce almost overnight. More than 40,000 existing mobile deployments and quick moves to upgrade firewalls and internet circuits were also crucial components of the agency’s successful strategy.
New and existing capabilities were instrumental in accomplishing State’s most important mission over the past three months – rapidly repatriating more than 100,000 Americans around the world and maintaining worldwide financial and logistical systems that enable such feats. The investments will continue to pay tremendous dividends down the line, as Mestrovich estimates that the three-month whirlwind of tech improvements have advanced agency IT modernization by four to five years.
MeriTalk: Can you provide some metrics to illustrate the success of your work during the pandemic?
Mestrovich: The Department of State has 107,000 technology users, including foreign service, civil service, contractors, and local staff – both domestic and overseas. When we started on a pandemic footing in February, our telework number was probably in the hundreds.
The Department of State is a very long-lived institution, where people came into the office and held meetings in office spaces. Telework was seen as kind of an accommodation in cases of a long illness or something similar. It wasn’t considered a way that the vast majority of the workforce could actually effectively perform their functions. We all had PCs at our desks. It wasn’t an organization where people had laptops.
We had been deploying mobile devices for several years, and so people had mobile phones and tablets, and were familiar with working remotely through those. But there’s a big difference between consuming content through a mobile device, and the active art of creating content, where you need a more robust platform. We had deployed 40,000 mobile devices, but it probably wasn’t effective for the masses to create content and continue the promotion of diplomacy and foreign policy.
Knowing we weren’t going to get 100,000 laptops out to people in a couple of weeks, we enabled two things. We had remote access via virtual desktop, and we ramped up that capacity. We had capacity for 5,000 and we doubled that in two or three weeks to get to 10,000. Then, we put in equipment orders to scale that up to the 15,000 range. In the process, we upgraded firewalls immediately, and we upgraded Internet circuits anywhere from five to seven gigs, and 10 gigs at our data centers.
The lifesaver was when we turned on GO Browser, which is effectively using a web browser on your local computer to log into Office 365. You then have all the published applications – so now you can do full-blown Word, Excel, and PowerPoint, as well as access Outlook.
Through that same mechanism, we published internal applications so people could use a web browser to access on-prem corporate applications, such as HR and travel. And, we published all our training materials. That was key because we’re still bringing people into the organization, and we had to onboard them.
That was our initial pivot, and we were able to effectively get approximately 107,000 people remote access capability through their mobile device, a web browser on their home computer, a virtual desktop platform, or a laptop. We put in some orders for additional laptops, so today we have about 3,000 laptops out there.
We released a telework survey at the beginning of May, and 80 percent of the 18,000 people who responded rated their telework experience as positive. That’s huge for us.
MeriTalk: Tell us about some of your largest priorities and successes during the last few months. What are you proudest of either for the IT team or the agency?
Mestrovich: I think everybody in the department is most proud of the fact that we’ve repatriated 100,000 Americans who had been stranded overseas since the beginning of this crisis. Airlines cancelled flights or enacted restrictions, so the department had to charter aircraft, make deals with airlines, and get in touch with embassies locally to get assistance to ex-pats.
In addition, we still had to process visas and passports for people, including Americans trying to come home, while we were having people telework.
The other piece that we’re really proud of is we have a huge financial system that pays for foreign aid, and we had to pay airlines to get these people on flights. The financial group is very dispersed across the globe. They have major processing centers, and we were able to keep them online. We were able to give them the tools and equipment they needed to process all the paperwork necessary to keep the whole system afloat from a financial perspective. Nobody went without a paycheck, all the invoices came in and all the bills got paid.
MeriTalk: Are there specific IT needs and systems that required adjustments because the State Department operates in so many countries?
Mestrovich: I don’t know if there are any unique requirements, but our medical group needed a surge in laptops. We had been dealing internally with our security folks about enabling the cameras and microphones that are embedded in those laptops. The importance of that was people were using the cameras and the microphones to conduct telemedicine. They were interviewing American citizens determining whether or not they had symptoms of COVID-19, and were making assessments on the ground if they needed to be quarantined or needed medication. It’s not a unique IT requirement, but that’s one of the cases where IT was really thrust to the forefront in enabling a medical assessment as to the health of the individual before we put them on an aircraft.
MeriTalk: Did the pandemic change anything from an IT perspective in countries that have more robust communications infrastructure versus those that don’t?
Mestrovich: We had two enterprise collaboration platforms – Microsoft Teams and Cisco Webex. The thing that made Webex useful in many instances was there was a dial-in number unique to the individual country. So, if you were in country X, you could dial the local number and still get into Webex, as opposed to having to dial a U.S. toll-free number and incur international charges. One of the things that certainly helped out from the Webex perspective is that people who didn’t have the app on their phone or didn’t have good bandwidth and wanted to be just an audio participant could do that. Some folks did use Zoom, and it worked for them, but it’s not an enterprise platform.
The infrastructure of the country that you were in certainly may have impacted user experience for some of the tools and the capabilities. That is something that we have to consider when we roll these out because not everybody has the bandwidth of North America, Western Europe, Singapore, Japan, or South Korea.
MeriTalk: Can you tell us what systems have worked best during the pandemic? Are there lessons learned from that for IT modernization, cloud, cyber, and authentication?
Mestrovich: In this instance, we let cloud do what cloud does best, which is just scale out as demand increases. As I mentioned, we had a couple of different remote access systems. The virtual desktop system, that’s all our infrastructure, and so there was a long lead time to buy more equipment and get it in and racked. That was a 90-day lead time from the time we said we wanted to increase the VDI capability from 10,000 to 15,000.
When we use the browser-based mechanism through Office 365, that was instantaneous. We could scale that to 100,000 people overnight. Likewise, our identity management systems – which were in the cloud – scaled immediately to accommodate 100,000 people. Webex and Teams – both cloud-based collaboration platforms – scaled immediately to handle the load. We didn’t have to lift a finger from an infrastructure perspective.
Now, you pay for that. You’re paying for consumption costs and paying extra licensing costs. But, if you want to talk about flexibility and ease of expansion, we let the cloud platforms do exactly what they did best. Overnight we were able to scale to the 100,000-user level because we moved to Office 365. We had already implemented the Cisco Webex cloud-based collaboration platform. We had implemented ServiceNow. So, we were able to iterate on developing applications for ServiceNow.
We delivered 10 or 12 ServiceNow-based apps in three or four weeks, and these were big apps that tracked every country on the planet and what their COVID-19 requirements were. If you came from North America to Great Britain, did you have to quarantine? If you went from Great Britain to Germany, did you have to quarantine? If you did, what were the quarantine requirements? So, there’s a huge tracking mechanism that shows what phase these countries are in, what phase our posts are in, and the COVID requirements for each. All that was done through the ServiceNow platform.
Our teams were able to pop those applications out in a matter of weeks, and then iterate on them as new requirements came up. That’s a huge success story because before it would have taken us months to figure out the application requirements and then go back to development. But, with these platform-based services, we’re able to iterate on those almost instantaneously.
MeriTalk: What keeps you up at night when you think about magnified cyber vulnerabilities in the pandemic, or any other emerging threats that may come along in the post-pandemic world?
Mestrovich: We don’t have people in buildings that are protected by guards, gates, and guns. They are in their homes, and if it’s a government-issued device, we really don’t know who’s around it – whether it’s left on and unattended, or there are cleaning crews coming through on any given day. It’s out there in the wild. So, there is an enhanced cyber threat.
If people are using their own personal computers, we really don’t want them downloading government information that may have Personally Identifiable Information (PII) or proprietary contract material, because we don’t know the security posture of that person’s individual computer. We do have new concerns about cybersecurity because now devices are either in an uncontrolled space, or we’re allowing people to use cloud-based resources for business, but it’s through a personal computer that we don’t really have visibility into.
The question is, “How do we implement technologies that understand that environment and still work with users to get done what they need to get done, but take into account the implications of how they’re doing that work?” We get into things like cloud application security brokers, and we’re pressing ahead to implement that. For identity management, we had just enforced Multi-Factor Authentication (MFA) across the enterprise in December 2019. That was a lifesaver for us because now everybody had enrolled in MFA, so we could enforce it across the board, no matter where a user logged in from. That was a great help for us to ensure at least the integrity of the user coming in to access our systems.
MeriTalk: What about the benefits of the CDM program? Has that been helping during the pandemic?
Mestrovich: I’m certainly not as close to that as our cybersecurity team or our CISO has been, so I can’t really comment. What I can talk about is a data analytics program we began at the end of last summer. This was data analytics on IT log data. We are bringing in log data now, from not only the government devices we have provided, but from the browsers people are using to access the government cloud infrastructure.
I want to make it clear that we’re not monitoring what they do on their personal computers. But, when they access one of our cloud infrastructures, we can bring in the log data that shows what browser type they use, the operating system underneath that browser, and where they’re logging in from geographically speaking based on the IP address. That helps in our security reviews because we understand the browsers people are using.
MeriTalk: What do you take away as the greatest lessons learned since the pandemic began?
Mestrovich: It’s not the technology; it’s the organizational process flow. It’s the business side of the house that is the long pole in the tent. We’ve been able to push out technology easily, and to our users’ credit, they’ve been able to absorb and adapt to it.
Where we struggle is with age-old applications and business processes on the back end. Here’s a classic case. We need a wet signature to show that an Authority to Operate (ATO) was granted. But we’re not mailing these documents around. So, how do you establish a process to get digital signatures and incorporate that into a workflow to allow the ATO to move forward?
Help desk was another area that presented challenges. They were flooded with calls because, in a matter of weeks, you have 100,000 people operating in a completely new way. As much as you try to make it seamless, there are going to be issues you didn’t factor in – like somebody using a version of Chrome that hasn’t been updated in three years. I don’t know that it was a surprise, but it certainly was an area where we could build in some additional contingencies.
MeriTalk: What have you seen from your perspective in terms of collaboration? What’s worked well, and where is there room for some improvement?
Mestrovich: Webex has worked well. Anybody can participate. You send them a link, and if they have a browser, they can click on the link and call in through their computer. If they don’t have a computer, they can dial in. It gives people a lot of flexibility.
Identity management, and the ability to do federated identity management, is a key enabler in collaborating within and beyond government organizations. We’ve been able to use our identity management systems to extend trust to other organizations. That’s important because, if I can bring people in and expose to them more proprietary data or bring them in behind the virtual firewall, we can collaborate more fully and effectively. Having those identity management systems and being able to trust one another’s certificates helps enable intergovernmental and interagency collaboration.
I don’t necessarily think it’s about everybody having the same platform. The internet wasn’t built that way. But, you do need to present an identity in a way that says you’re trusted.
MeriTalk: What’s worked well when it comes to best-practice sharing between IT teams and leaders across government? How has your agency worked with others?
Mestrovich: It’s been really good. OMB has continued to have CIO Council sessions, even though they’re virtual. I know there have been a lot of conversations about best practices for distributing mobile devices and mobile device management.
In our case, there’s a large percentage of our workforce that needs access to classified information on a routine basis. Maybe some folks discovered they don’t need it every day; maybe they need it multiple times a week. But still, you’re going to have to go into a controlled facility to have access to that information. Understanding your population demographics and how you need to service them has been helpful.
MeriTalk: Let’s return to the early days of the pandemic. What were your first few weeks like, and what is your new normal?
Mestrovich: In the first few weeks, it was all about IT. And I don’t mean that to be self-serving, but we had seven calls a week with different groups across the department. It was on the pandemic, but IT was a participant in that.
For the first two-to-three weeks, the vast majority of those calls were about two specific things. First, was keeping people safe. And second, “How are we going to use our IT systems to do what we normally do?” After that, things settled down, and IT was not at the forefront of those conversations. I was happy about that because people got used to using the toolsets. That’s on the IT provisioning side. Then, there were people just getting used to telework, which coincided with the first three weeks.
It was a hectic IT environment, but people showed up at their desks at six o’clock in the morning and didn’t leave until five at night. They were in their telework space all day long. They were there because Teams and Webex allowed them to go from one meeting to another without any breaks. Whereas, before you would naturally walk from one building to another or have a 30-minute break.
Once people got used to the tools for telework, they found that they were really productive. I think, in some cases, they might have overused them. I think productivity increased a lot. But people were really exhausted because they didn’t take a break. We had to tell people, “You’ve got to manage your time a little bit better. This is a marathon.” I think people have adapted and are building in breaks.
MeriTalk: Are there other stories and anecdotes you’d like to share in terms of things that have been important or inspiring?
Mestrovich: In the span of three months, we probably advanced the state of IT modernization to a degree that normally would have taken up to four or five years. There was a clear necessity to do business in a different way.
I’m not going to say that there was a different risk appetite, but we got things done. We haven’t deviated from the plan, but we were able to implement the plan significantly faster because this gave us a forcing factor.
Telework is a classic example. There were managers who said, “If you’re teleworking five days a week, you’re obviously not doing work.” That was a real conversation. Now people are saying, “People can be more effective.” In our telework survey, people have said, “Please, please keep us teleworking.” That’s the overwhelming sentiment – don’t revert.
People have realized that telework is acceptable, and they’re productive doing it. That cultural shift would have taken years had it been left to the normal course of events.
MeriTalk: Any specific shout outs to folks, either on your team or across the government?
Mestrovich: When [State Department CIO] Stuart McGuigan and I got to IRM (Information Resource Management) we made it very clear that we intended to make this a family. We are a team and are all in this together. We’re only going to be successful if we all pull in the same direction.
During this crisis, our operations group did amazing work getting the tools out. We have a foreign operations group that coordinated all of those activities with the 276 posts overseas. They did an enormous amount of work. Our Information Assurance (IA) team has been incredibly responsive to all the requests that we put forward for risk acceptance or risk mitigation. Our budgeting and finance team was great, processing orders in just a day or two. And, the contracting team awarded contracts immediately and helped with emergency procurement for equipment.
Across the board, this has been 100 percent a team effort. Like anything, if any one part of the machine breaks down, it all comes to a screeching halt. I couldn’t be more pleased with how much everybody has pulled together.
MeriTalk: What things might we do differently in the future as a result of this experience?
Mestrovich: The one thing that someone else said is, “Work is what you do, not where you are.” If we take one lesson from this, it’s that we should be able to do the work from wherever we happen to be.
My fear is that people will start going back to the office and will want to revert to the traditional mentality. Going forward, every IRM meeting that I schedule is going to be a collaborative meeting, hosted either on Teams or Webex. If nobody’s outside of the office and nobody wants to call in, that’s fine, but it’s always going to be remotely accessible. I think we just have to get in the habit of doing that, so that we can continue to afford people the opportunity to telework.
Then there’s the investment side. A lot of the places where people meet on the inside don’t have video cameras or microphones. So, we need to enable our workplace to have the multimedia capabilities that allow people to collaborate remotely.
As time goes by, more people will certainly go back to the office. We’re social animals; we need that engagement. There’s always going to be that instance where you have a conversation with somebody in the hallway and that moves something forward. But I want to preserve the telework capability because it’s a quality of life issue, not only for current staff, but for recruiting. For the young workforce, time and time management is very valuable. Affording them the opportunity to set their own schedules and work at their own pace, will be very, very valuable from a recruiting perspective.
MeriTalk: One of the big changes in the IT world is no more in-person conferences. How do you envision interacting with industry?
Mestrovich: We’re social animals, so we’re going to want to get together. We’re going to want to fly to the company headquarters and visit the research lab. And, I think we’re going to resume that once we have a vaccine and effective treatments. However, I think we have been afforded a new opportunity. We now have these other alternatives. I don’t know that it’s going to replace business travel or conferences, but it may get people who were otherwise disadvantaged in some way, an opportunity to participate more broadly.
At the end of the day, it doesn’t change our behavior but, instead, makes opportunities much more equitable and accessible across the board for all potential participants.
Read other Federal success stories from the COVID-19 pandemic.