Your Military

SOCOM wants to start data mining the open web

U.S. Special Operations Command (SOCOM) is building an open-source data-mining program that will run automatic keyword searches across a variety of websites and databases, allowing its operators to build a better picture of their operating environment in as close to real time as possible.

The command held a series of meetings with a group of defense industry representatives in late June to discover what commercial tech might be available in the near future, according to a SOCOM official.

Dubbed AVATAR — "Automated Visualization for Tailored Analytical Reporting" — the program would be run by existing SOCOM staffers at the "tactical, strategic and operational levels" of action, according to command spokesman Navy Capt. Kevin Aandahl.

The objective, Aandahl said, is to "filter and display open-source information in a way that is specific and timely to the needs and requirements of the SOF [special operations forces]."

A request for information released in May stated that the program would comprise functional areas: "data acquisition, data mining and analysis, visualization and reporting, and alerts and monitoring."

SOCOM is also looking for contractors to provide the ability to "perform high-volume queries quickly and conduct searches on pre-determined websites."

The data-mining software, the May solicitation said, would "automatically extract information of interest from all types of structured, unstructured, and multimedia data," then perform link analysis and correlate that information with intelligence that has already been provided by the big US intelligence agencies.

The national security strategy outlined by the White House in 2012 places a premium on the use of special operations forces to operate — quietly — with allies on train and assist missions while continuing their counterterror mission wherever Washington deems fit.

But that doesn't always mean that specific missions will be given a high priority by the big intelligence agencies.

When operators go to perform a small mission, they have to tailor their intelligence packages for the geographic area, and "that's something that's hard to get from one of the big intel agencies on short notice, because those agencies are worried about those top-level national priorities," said Jim Penrose, a former National Security Agency intelligence officer.

"So when SOCOM needs to go into a denied area that's way down on the priority list, the big agencies are going to have a hard time moving and shifting resources because they've already got so many other priorities," he added.

Penrose, executive vice president for cyber intelligence at Dark Trace, a UK-based cybersecurity firm, said domestic privacy issues are not as much of a concern, since much of what he expects SOCOM operators to query are existing government databases.

If and when they do scan the web, SOCOM will have to operate under DoD rules that prohibit operations within the United States and place limits on what kinds of data they can access overseas.

While there is some seed money for the program in the fiscal 2015 budget, no decision as to when and how to fully launch the program has been made. Aandahl declined to say how much was requested.

SOCOM claims not to expect to have to add staffing to the 69,000 person agency in order to maintain the program, though Penrose isn't so sure. "It's a noble goal but the reality is that there's going to be a lot of technologists behind the scenes" to maintain the system, he said. "So I think there will be a human capital cost to it even if it's not an intelligence analyst doing it."

In many respects, the program sounds like a study that SOCOM scuttled in August 2012 called Quantum Leap.

That effort involved a group of civilian contract and government employees gathering in Crystal City, Virginia, for a nine-day experiment to ascertain if the command could successfully mine social media and other open-source information databases to gather intelligence.

An after-action report leaked by watchdog organization Secrecy News in August 2013 reported that the experiment "was successful in identifying strategies and techniques for exploiting open sources of information, particularly social media."

That effort was short-lived, however. When contacted for comment, a SOCOM spokesman insisted that Quantum Leap was "a very small, little-known, inconsequential experiment that was defunded" after that first experiment in Virginia.

When asked about the similarities between the previous study and AVATAR,Aandahl said personnel working on the AVATAR effort "do not have any knowledge of what you referenced as the 'Quantum Leap study.' "

Recommended for you
Around The Web